I want to scrape a list of posts while the page scroll is infinite loading. I want scrollIntoView() for each element in the loop. my code is temporarily like this and the result when the page is loaded, it bounces out but no error.

for (let i = 0; i < 20; i++) {
    const selector = 'div[role="feed"]:nth-child(2) div.sjgh65i0'

    await page.evaluate((selector, i) => {
      setTimeout(() => {
        const element = document.querySelectorAll(selector)[i]
        if(element) {
          element.scrollIntoView();
        }
      }, 2000);
    }, selector, i)
  }

For example, I want to take 20 posts and then the scraper will take the posts one by one.

I want to scrape a list of posts while the page scroll is infinite loading. I want scrollIntoView() for each element in the loop. my code is temporarily like this and the result when the page is loaded, it bounces out but no error.

for (let i = 0; i < 20; i++) {
    const selector = 'div[role="feed"]:nth-child(2) div.sjgh65i0'

    await page.evaluate((selector, i) => {
      setTimeout(() => {
        const element = document.querySelectorAll(selector)[i]
        if(element) {
          element.scrollIntoView();
        }
      }, 2000);
    }, selector, i)
  }

For example, I want to take 20 posts and then the scraper will take the posts one by one.

• javascript
• node.js
• dom
• puppeteer

1 Answer 1

With your current flow, all timeouts are set almost at once and then all fire after the same 2 sec.

Try something like this:

const selector = 'div[role="feed"]:nth-child(2) div.sjgh65i0';

for (let i = 0; i < 20; i++) {
  await page.waitForTimeout(2000);

  await page.evaluate((selector, i) => {
    const element = document.querySelectorAll(selector)[i];
    if(element) {
      element.scrollIntoView();
    }
  }, selector, i);
}