Playwright is not working as expected when i try to use xpath functions.

This is the code that i wrote to scrape the text inside the <h1> tag of .

const pw = require('playwright');

async function fetch(url) {
    var browser = await pw.chromium.launch();
    var context = await browser.newContext();
    var page = await context.newPage();
    await page.goto(url);
    const h1 = await page.$('//h1')
    console.log(await h1.evaluate(h1 => h1.innerHTML, h1));
    await browser.close();
}

fetch('')

When executed this code works perfectly and displays,

Example Domain

But if i try to get the text inside the h1 tag using the xpath function text() like below,

const h1 = await page.$('//h1/text()'); // also tried await page.$('xpath=//h1/text()');
console.log(await h1.evaluate(h1 => h1.textContent, h1));

It is throwing,

UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'evaluate' of null

Am i doing something wrong or doesn't it work with xpath functions.

Playwright is not working as expected when i try to use xpath functions.

This is the code that i wrote to scrape the text inside the <h1> tag of https://example.

const pw = require('playwright');

async function fetch(url) {
    var browser = await pw.chromium.launch();
    var context = await browser.newContext();
    var page = await context.newPage();
    await page.goto(url);
    const h1 = await page.$('//h1')
    console.log(await h1.evaluate(h1 => h1.innerHTML, h1));
    await browser.close();
}

fetch('https://example.')

When executed this code works perfectly and displays,

Example Domain

But if i try to get the text inside the h1 tag using the xpath function text() like below,

const h1 = await page.$('//h1/text()'); // also tried await page.$('xpath=//h1/text()');
console.log(await h1.evaluate(h1 => h1.textContent, h1));

It is throwing,

UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'evaluate' of null

Am i doing something wrong or doesn't it work with xpath functions.

• javascript
• node.js
• xpath
• web-scraping
• playwright

• I suspect it's because h1(in your 2nd example) is already a text node so it doesn't have the property textContent. So I would try to evaluate just h1 and see what happens. – Jack Fleeting Commented Jan 26, 2020 at 16:20
• @JackFleeting I tried and it returned null. – CodeIt Commented Jan 27, 2020 at 2:50

2 Answers 2

As Michael said, the purpose of the $ function is to return a DOM element. If you want to evaluate an XPath expression you could use document.evaluate inside an evaluate function.

async function fetch(url) {
    var browser = await playwright.chromium.launch();
    var context = await browser.newContext();
    var page = await context.newPage();
    await page.goto(url);
    console.log(await page.evaluate(() => 
        document.evaluate('//h1/text()', document, null, XPathResult.STRING_TYPE).stringValue));
    await browser.close();
}

fetch('https://example.')

Firstly, text() isn't actually a function. It's an abbreviation for an axis step child::text() that selects the text nodes of the containing element.

The XPath expression is working perfectly; it's the invoking application code that's wrong. If your XPath expression returns text nodes, then the application can't access textContent, because text nodes don't have a textContent property. That's DOM for you...