I am using request and cheerio node modules to create get some data out of a website. I'd like to get a list of item and once this list is complete, call an asynchronous function:

request('http://myurl', function(req,res,data){
    var $ = cheerio.load(data);
    var List = [];

    $('.myItems').each(function(i, element){
        console.log( typeof $(this).text() )
        List.push($(this).text());
    });

   for (var i=0; i <  List.length; i++){
      // make an asynchronous call to a API
   }
});

My question is how do I wait for the list to be completed, ie, how can I know than the .each function has looped through all the items?

Can I do this with async?

Thanks

I am using request and cheerio node modules to create get some data out of a website. I'd like to get a list of item and once this list is complete, call an asynchronous function:

request('http://myurl', function(req,res,data){
    var $ = cheerio.load(data);
    var List = [];

    $('.myItems').each(function(i, element){
        console.log( typeof $(this).text() )
        List.push($(this).text());
    });

   for (var i=0; i <  List.length; i++){
      // make an asynchronous call to a API
   }
});

My question is how do I wait for the list to be completed, ie, how can I know than the .each function has looped through all the items?

Can I do this with async?

Thanks

• javascript
• node.js
• asynchronous
• cheerio

4 Answers 4

The .each function of cheerio is synchronous (see source). So as long as you do nothing asynchronous in the callback (which is the case in the question), you have nothing to do : at the following line the loop will be complete.

If you do call asynchronous functions in the loop, the cleanest and simplest solution is to use a Promise library.

In a sequential way (exemple with Bluebird) :

var p = Promise.resolve();
$('.myItems').each(function(i, element){
    p = p.then(function(){ 
         // do something with $(this).text() and return a promise
    });
});
p.then(function(){
   // all asynchronous tasks are finished
});

If sequential requests aren't needed (here an example with Q) :

Q.allSettled($('.myItems').map(function(){
   // return a promise
})).then(function(){
   // all asynchronous tasks are finished
});

Well, for anyone else having this problem, just know that you don't really need to use the each method, you can transform them in an array with the toArray method and iterate over them in a for loop.

That way, there is no need to use an external lib like Bluebird.

request('http://myurl', function(req,res,data){
    var $ = cheerio.load(data);
    var List = [];

    const items = $('.myItems').toArray();
    for(let i = 0; i < items.length; i++){
        const el = items[i];
        console.log( typeof $(el).text() )
        List.push($(el).text());
        // make an asynchronous call to a API
    }
});

You could use the async module to easily handle those kinds of async tasks.

Specifically async.each or async.eachLimit if you need concurrency > 1 or async.eachSeries if you want to go through the items in the array, one at a time.

I have to admit I haven't managed to get Denys Séguret's solution working (for the async calls within .each() loop) - the "after" operation still happens before the .each() loop is completed, however I found a different way, and I hope it helps someone:

var Promises = require('bluebird');

request('http://myurl', function(req,res,data){
    var $ = cheerio.load(data);
    var List = [];

    Promises
    // Use this to iterate serially
    .each($('.myItems').get(), function(el){
        console.log( typeof $(el).text() )
        List.push($(el).text());
    })
    // Use this if order of items is not important
    .map($('.myItems').get(), function(el){
        console.log( typeof $(el).text() )
        List.push($(el).text());
    }, {concurrency:1}) // Control how many items are processed at a time
    // When all of the above are done, following code will be executed
    .then(function(){
        for (var i=0; i <  List.length; i++){
            // make an asynchronous call to a API
        }
    });
});

In this particular code example it looks like you could be doing your async calls within then mapping functions, but you get the gist...

Map: https://github.com/petkaantonov/bluebird/blob/master/API.md#mapfunction-mapper--object-options---promise

Each: https://github.com/petkaantonov/bluebird/blob/master/API.md#eachfunction-iterator---promise