How To Call Ajax In A Webpage From Python Script Without Browser Emulation Or Headless Brawser?

I am new to ajax and javascript. I am crawling a website in which I am able to fetch relevant piece of details with the help of XPath after downloading webpage using Python(urllib2

Solution 1:

Well first of all install firebug (https://getfirebug.com/)

Then go to your page, launch the firebug and go to the Net tab in firebug panel. Now in this tab you can see all of the get/posts calls your firefox sends to the website.

Now you can click around, refresh the page and see what calls are being made. In your case click the button and you'll see new calls are being made, you will probably find it at html tab.

There you can find a call once you click on it you'll see the request and other details.

make a dictionary of the parameters and attach it to "data=" in your post. you can also make headers by making a dictionary of it and attaching it to the "headers=" in your post.

Take note: a lot of websites use cookies to identify if the calls are made by a legit browser so it might require quite a bit of fiddling with cookies and urls. !

it's hard to give examples if you don't give us the website.