I have a nodejs codebase deployed on google cloud, using an ubuntu image. This code will open a headless browser using puppeteer, render some HTML and save an image. This is currently taking about 4-6 seconds, which is relatively quick.
However I feel it could be quicker if I were to leave the headless browsers (uses google chrome) open somehow, and just have the code utilize these. I also have BullMQ implemented, where I can setup queues and workers to utilize those pools.
Is there some way I can smartly utilize nodejs (and optionally bullmq) to manage this pool of browsers? I'm also not sure what the memory constraints would be, for leaving so many google chrome browsers "open" in a headless manner
I have a nodejs codebase deployed on google cloud, using an ubuntu image. This code will open a headless browser using puppeteer, render some HTML and save an image. This is currently taking about 4-6 seconds, which is relatively quick.
However I feel it could be quicker if I were to leave the headless browsers (uses google chrome) open somehow, and just have the code utilize these. I also have BullMQ implemented, where I can setup queues and workers to utilize those pools.
Is there some way I can smartly utilize nodejs (and optionally bullmq) to manage this pool of browsers? I'm also not sure what the memory constraints would be, for leaving so many google chrome browsers "open" in a headless manner
Share Improve this question asked Feb 7 at 13:04 Mike K.Mike K. 6083 gold badges23 silver badges51 bronze badges 1- @ Mike K. Did you have time to check my answer? It helped you to solve your problem? If yes,could you please have a look at this doc and accordingly proceed. – Sai Chandini Routhu Commented yesterday
1 Answer
Reset to default 0For your case instead of creating a new puppeteer browser instance every time, you should use BullMQ and Puppeteer Cluster.
BullMQ helps you to queue and distribute jobs efficiently. So, multiple workers can process tasks in parallel. Puppeteer Cluster helps to maintain a pool of browsers and you can reuse them across jobs. Because cluster keeps a limited number of browsers open and reuses them.
If you use both BullMQ and Puppeteer Cluster which helps you to improve speed because the browser is not opened every time. It helps to reduce memory usage also.
Refer to this Puppeteer Cluster Article by Webshare Academy and How To Handle Asynchronous Tasks with Node.js and BullMQ Digital Ocean tutorial by Stanley Ulili and Caitlin Postal.