Is there any programming libraries available that will parse an HTML document, execute JavaScript and then allow me to navigate the DOM?  This needs to be performed server side, not client side. Any language will do, but Java, PHP, or Ruby are preferred.

Is there any programming libraries available that will parse an HTML document, execute JavaScript and then allow me to navigate the DOM?  This needs to be performed server side, not client side. Any language will do, but Java, PHP, or Ruby are preferred.

• java
• php
• javascript
• ruby
• browser

9 Answers 9

Have you tried Bringing the Browser to the Server?

in java: http://lobobrowser.org/cobra/java-html-parser.jsp
this is a a Javascript-aware, CSS-aware HTML parser
the most important feature in relation to your question: It is Javascript-aware. DOM modifications that occur during parsing will be reflected in the resulting DOM.

Java has support for javascript with Rhino, also look at this page for server side javascript solutions: http://en.wikipedia.org/wiki/Server-side_JavaScript

For Java, be sure to check out HtmlUnit and HttpUnit.

PhantomJS does this and can be used with any server side language. See some integration modules below for NodeJS and PHP

NodeJS

https://npmjs.org/package/node-phantom

https://github.com/sgentle/phantomjs-node

PHP

https://github.com/diggin/php-PhantomjsRunner

PHP has DOMDocument for navigating the DOM. I haven't heard of anything for executing JavaScript.

Start from this post and follow a links. Or just search for Rhino.

There are now several projects that do a really good job of this:

• PhantomJS is a headless version of WebKit, and there are some helpful wrappers such as CasperJS.

• Zombie.js which is a wrapper over jsdom written in Javascript (Node.js).

You need to write JavaScript code to interact with both of these projects. I like Zombie.js better so far, since it is easier to set up, and you can use any Node.js/npm modules in your code.

node.js ?

Node can run any javascript file in its console. I would try node first & see if it can do what you want as it likely has the largest user base & documentation.