Currently I'm using Fireproof to make it promise-based and here's what I have:

fbase.child("questions").once("value").then(function(questionsSnapshot) {
  return questionsSnapshot.forEach(function(questionSnapshot) {
    return console.log(questionSnapshot.val());
  });
});

But questions is huge. 20,000+ records. Not sure - is there a better way?

Currently I'm using Fireproof to make it promise-based and here's what I have:

fbase.child("questions").once("value").then(function(questionsSnapshot) {
  return questionsSnapshot.forEach(function(questionSnapshot) {
    return console.log(questionSnapshot.val());
  });
});

But questions is huge. 20,000+ records. Not sure - is there a better way?

• javascript
• firebase

• 2 The Firebase Javascript client supports promises nowadays: firebase./blog/2016-01-21-keeping-our-promises.html. Aside from that: the best way to deal with long lists is to avoid them. How to do that depends on the use-case, which I doubt is to log them. – Frank van Puffelen Commented Feb 20, 2016 at 9:20
• @FrankvanPuffelen I get that but how would you send an email marketing campaign to all your users? I plan to keep /notices/{auth.uid} and need to iterate in a scheduled job. child_added would efficiently handle it but "value" would be inefficient[1] as it stores items in memory cache. [1] entire contents has been synchronized firebase.google./docs/reference/js/… – Leblanc Meneses Commented Dec 28, 2016 at 14:55
• @FrankvanPuffelen I'm guessing the only real answer is to use a storage engine that provides a nextPageToken or some type of continuation token. Then the problem is how to sync firebase data with said storage engine. – Leblanc Meneses Commented Dec 28, 2016 at 15:20
• @LeblancMeneses Firebase supports anchor items with orderByKey().startAt(key) or orderByChild("prop").startAt("value", key). – Frank van Puffelen Commented Dec 28, 2016 at 15:40

1 Answer 1

This question is really about iteration performance in Node rather than anything to do with Firebase, if I understand correctly. The question is, once you have this 20000+ array of objects, what is the best way to iterate through them and perform an operation on each one?

There are a few possible performance optimizations you could make. First, you could use a for loop rather than Array.prototype.forEach(), which tends to be faster for very large arrays.

...
var len = questionsSnapshot.length;
for (var i = 0; i < len; i++) {
  console.log(questionsSnapshot[i].val());
}

Or, using ES6 for...of syntax:

for (let question of questionsSnapshot) {
  console.log(question.val());
}

However, in a quick test using console.time() in Node on an array of 20000 objects, I noticed no consistent performance gain. In fact Array.prototype.forEach() was often faster than using a for loop.

The other optimization you might make is to move the nested callback passed to forEach() into the module scope. That way, it will only be created once, rather than being recreated each time its parent function is called.

fbase.child("questions").once("value").then(function(questionsSnapshot) {
  return questionsSnapshot.forEach(logValue);
});

function logValue(questionSnapshot) {
  return console.log(questionSnapshot.val());
}

However, this will only result in a significant performance gain if you make many calls to the parent function, i.e., if perform your Firebase query very often. It sounds to me like the question is more about iterating over a large array once you already have it, not about performing the same query many times over.

So, if you want to perform an operation for each item in your array, I think what you are doing is fine. 20000 objects is not all that huge; it takes about 5ms to iterate through such an array on my unimpressive puter. In fact, whatever it is you are doing to each item is likely to take longer than iterating through the array.

If you notice a particular lag in performance, you might want to elaborate by editing your post. Otherwise, I would suggest not worrying about it.