I am navigating a site using python's mechanize module and having trouble clicking on a javascript link for next page. I did a bit of reading and people suggested I need python-spidermonkey and DOMforms. I managed to get them installed by I am not sure of the syntax to actually click on the link.

I can identify the code on the page as:

<a href="javascript:__doPostBack('ctl00$MainContent$gvSearchResults','Page$2')">2</a>

Does anyone know how to click on it? or if perhaps there's another tool.

Thanks

I am navigating a site using python's mechanize module and having trouble clicking on a javascript link for next page. I did a bit of reading and people suggested I need python-spidermonkey and DOMforms. I managed to get them installed by I am not sure of the syntax to actually click on the link.

I can identify the code on the page as:

<a href="javascript:__doPostBack('ctl00$MainContent$gvSearchResults','Page$2')">2</a>

Does anyone know how to click on it? or if perhaps there's another tool.

Thanks

• javascript
• python
• screen-scraping
• mechanize
• spidermonkey

• Wouldn't you just click on it normally? If python-spidermonkey and DOMForms are any good it would just work. – Peter C Commented Mar 6, 2011 at 1:14
• I was trying to but its really hard to find examples that work. I am actually not sure how to do it. Most of the commands I found in the example didn't work either. I have read of alot of people suggesting these tools for their ability to handle javascript but using the packages is not very straight forward. – Lostsoul Commented Mar 6, 2011 at 1:19
• If I need to deal with JavaScript, I avoid mechanize (or twill, which I prefer) and instead use something like Selenium or Splinter (which is my favorite between the two). – brandizzi Commented Dec 3, 2011 at 4:20

3 Answers 3

I mainly use HtmlUnit under jython for these use cases. Also I published a simple article on the subject: Web Scraping Ajax and Javascript sites.

instead of struggling with python-spidermonkey try webkit's qt python bindings.

Here is a full example to execute JavaScript and extract the final HTML.

how about calling __doPostBack('ctl00$MainContent$gvSearchResults','Page$'+pageid); (javascript method, via python-spidermonkey)