python urllib2 - wait for page to finish loading/redirecting before scraping? -


i'm learning make web scrapers , want scrape tripadvisor personal project, grabbing html using urllib2. however, i'm running problem where, using code below, html not correct page seems take second redirect (you can verify visiting url) - instead code page briefly appears.

is there behavior or parameter set make sure page has finished loading/redirecting before getting website content?

import urllib2 bs4 import beautifulsoup  bostonpage = urllib2.urlopen("http://www.tripadvisor.com/hacsearch?geo=34438#02,1342106684473,rad:s0,sponsors:abest_western,style:szff_6") soup = beautifulsoup(bostonpage) print soup.prettify()

edit: answer thorough, however, in end solved problem this: https://stackoverflow.com/a/3210737/1157283

inreresting problem isn't redirect page modifies content using javascript, urllib2 doesn't have js engine gets data, if disabled javascript on browser note loads same content urllib2 returns 

import urllib2 beautifulsoup import beautifulsoup  bostonpage = urllib2.urlopen("http://www.tripadvisor.com/hacsearch?geo=34438#02,1342106684473,rad:s0,sponsors:abest_western,style:szff_6") soup = beautifulsoup(bostonpage) open('test.html', 'w').write(soup.read())

test.html , disabling js in browser, easiest in firefox content -> uncheck enable javascript, generates identical result sets.

so can well, first should check if site offers api, scrapping tends frown http://www.tripadvisor.com/help/what_type_of_tripadvisor_content_is_available

travel/hotel api's? looks might, though restrictions.

but if still need scrape it, js, can use selenium http://seleniumhq.org/ used testing, easy , has docs.

i found scraping websites javascript enabled? , http://grep.codeconsult.ch/2007/02/24/crowbar-scrape-javascript-generated-pages-via-gecko-and-rest/

hope helps.

as side note:

>>> import urllib2 >>> bs4 import beautifulsoup >>>  >>> bostonpage = urllib2.urlopen("http://www.tripadvisor.com/hacsearch?geo=34438#02,1342106684473,rad:s0,sponsors:abest_western,style:szff_6") >>> value = bostonpage.read() >>> soup = beautifulsoup(value) >>> open('test.html', 'w').write(value)

Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

How to get the ip address of VM and use it to configure SSH connection dynamically in Ansible -

i trying ip address of vm , trying use establish ssh vm. following script: --- - hosts: localhost become: yes connection: local gather_facts: false serial: 1 vars_files: - createvmvars.yml tasks: - name: gathering vm info. vsphere_guest: vcenter_hostname: "{{vcenter_hostname}}" username: "{{vcenter_username}}" password: "{{vcenter_password}}" guest: "{{guest_name}}" vmware_guest_facts: yes register: var - debug: msg="{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" - name: establishing ssh connection. script: /home/shasha/devops/scripts/ssh_configure.sh "{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" following content of ssh_configure.sh : #!/bin/sh ssh-keygen -t rsa ssh-copy-id $1 but when running playbook, getting following error while executing third task: error! 'var' undefined but debug module(second task) printin
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more