Is there a way to wait for network requests to resolve after performing an action on a page, before performing a new action in Puppeteer?

I need to interact with a select menu on the page using page.select() which causes dynamic images and fonts to load into the page. I need to wait for these requests to plete before executing the next action.

--

Caveats:

1. I cannot reload the page or go to a new url.
2. I do not know what the request types might be, or how many

--

// launch puppeteer
const browser = await puppeteer.launch({});

// load new page
const page = await browser.newPage();

// go to URL and wait for initial requests to resolve
await page.goto(pageUrl, {
  waitUntil: "networkidle0"
});

// START LOOP
for (let value of lotsOfValues) {

  // interact with select menu
  await page.select('select', value);

  // wait for network requests to plete (images, fonts)
  ??

  // screenshot page with new content
  await pageElement.screenshot({
    type: "jpeg",
    quality: 100
  });

} // END LOOP

// close
await browser.close();

Is there a way to wait for network requests to resolve after performing an action on a page, before performing a new action in Puppeteer?

I need to interact with a select menu on the page using page.select() which causes dynamic images and fonts to load into the page. I need to wait for these requests to plete before executing the next action.

--

Caveats:

1. I cannot reload the page or go to a new url.
2. I do not know what the request types might be, or how many

--

// launch puppeteer
const browser = await puppeteer.launch({});

// load new page
const page = await browser.newPage();

// go to URL and wait for initial requests to resolve
await page.goto(pageUrl, {
  waitUntil: "networkidle0"
});

// START LOOP
for (let value of lotsOfValues) {

  // interact with select menu
  await page.select('select', value);

  // wait for network requests to plete (images, fonts)
  ??

  // screenshot page with new content
  await pageElement.screenshot({
    type: "jpeg",
    quality: 100
  });

} // END LOOP

// close
await browser.close();

• javascript
• node.js
• puppeteer

• 1 you can set request interception on and check if every request to (image, fonts) after page.select has corresponding response. – Gaurav Dhiman Commented Jan 16, 2020 at 14:02
• Thanks for the point in the right direction. I re-wrote this module to do what I wanted - npmjs./package/pending-xhr-puppeteer – danlong Commented Jan 17, 2020 at 9:07

2 Answers 2

The answer to this lies in using page.setRequestInterception(true); and monitoring subsequent requests, waiting for them to resvolve before moving on to the next task (thanks @Guarev for the point in the right direction).

This module (https://github./jtassin/pending-xhr-puppeteer) does exactly that, but for XHR requests. I modified it to look for 'image' and 'font' types.

Final code looks something like this:

// launch puppeteer
const browser = await puppeteer.launch({});

// load new page
const page = await browser.newPage();

// go to URL and wait for initial requests to resolve
await page.goto(pageUrl, {
  waitUntil: "networkidle0"
});

// enable this here because we don't want to watch the initial page asset requests (which page.goto above triggers) 
await page.setRequestInterception(true);

// custom version of pending-xhr-puppeteer module
let monitorRequests = new PuppeteerNetworkMonitor(page);

// START LOOP
for (let value of lotsOfValues) {

  // interact with select menu
  await page.select('select', value);

  // wait for network requests to plete (images, fonts)
  await monitorRequests.waitForAllRequests();

  // screenshot page with new content
  await pageElement.screenshot({
    type: "jpeg",
    quality: 100
  });

} // END LOOP

// close
await browser.close();

NPM Module

class PuppeteerNetworkMonitor {

    constructor(page) {
        this.promisees = [];
        this.page = page;
        this.resourceType = ['image'];
        this.pendingRequests = new Set();
        this.finishedRequestsWithSuccess = new Set();
        this.finishedRequestsWithErrors = new Set();
        page.on('request', (request) => {
            request.continue();
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.add(request);
                this.promisees.push(
                    new Promise(resolve => {
                        request.resolver = resolve;
                    }),
                );
            }
        });
        page.on('requestfailed', (request) => {
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.delete(request);
                this.finishedRequestsWithErrors.add(request);
                if (request.resolver) {
                    request.resolver();
                    delete request.resolver;
                }
            }
        });
        page.on('requestfinished', (request) => {
            if (this.resourceType.includes(request.resourceType())) {
                this.pendingRequests.delete(request);
                this.finishedRequestsWithSuccess.add(request);
                if (request.resolver) {
                    request.resolver();
                    delete request.resolver;
                }
            }
        });
    }

    async waitForAllRequests() {
        if (this.pendingRequestCount() === 0) {
            return;
        }
        await Promise.all(this.promisees);
    }

    pendingRequestCount() {
        return this.pendingRequests.size;
    }
}

module.exports = PuppeteerNetworkMonitor;

For anyone still interested in the solution @danlong posted above but wants it in a more modern way, here is the TypeScript version for it:

import { HTTPRequest, Page, ResourceType } from "puppeteer";

export class PuppeteerNetworkMonitor {
    page: Page;
    resourceType: ResourceType[] = [];
    promises: Promise<unknown>[] = [];
    pendingRequests = new Set();
    finishedRequestsWithSuccess = new Set();
    finishedRequestsWithErrors = new Set();
    constructor(page: Page, resourceType: ResourceType[]) {
        this.page = page;
        this.resourceType = resourceType;
        this.finishedRequestsWithSuccess = new Set();
        this.finishedRequestsWithErrors = new Set();
        page.on(
            "request",
            async (
                request: HTTPRequest & { resolver?: (value?: unknown) => void },
            ) => {
                await request.continue();
                if (this.resourceType.includes(request.resourceType())) {
                    this.pendingRequests.add(request);
                    this.promises.push(
                        new Promise((resolve) => {
                            request.resolver = resolve;
                        }),
                    );
                }
            },
        );
        page.on(
            "requestfailed",
            (request: HTTPRequest & { resolver?: (value?: unknown) => void }) => {
                if (this.resourceType.includes(request.resourceType())) {
                    this.pendingRequests.delete(request);
                    this.finishedRequestsWithErrors.add(request);
                    if (request.resolver) {
                        request.resolver();
                        delete request.resolver;
                    }
                }
            },
        );
        page.on(
            "requestfinished",
            (request: HTTPRequest & { resolver?: (value?: unknown) => void }) => {
                if (this.resourceType.includes(request.resourceType())) {
                    this.pendingRequests.delete(request);
                    this.finishedRequestsWithSuccess.add(request);
                    if (request.resolver) {
                        request.resolver();
                        delete request.resolver;
                    }
                }
            },
        );
    }

    async waitForAllRequests() {
        if (this.pendingRequestCount() === 0) {
            return;
        }
        await Promise.all(this.promises);
    }

    pendingRequestCount() {
        return this.pendingRequests.size;
    }
}

I did change one thing, where instead of hard-coding what resource type to look for in the network requests, I am passing the resource types to look for as one of the constructor arguments. That should make this class more generic.

I've tested this code with my API that uses Puppeteer, and it works great. For the usage of this class, it would be similar to what @danlong posted above like this:

// other necessary puppeteer code here...
const monitorNetworkRequests = new PuppeteerNetworkMonitor(page, ["image"]);
await monitorNetworkRequests.waitForAllRequests();