I'm currently building a cloud function to scrape some data from emerce websites. I'm running them on the Node.js v8 environment, so I'm able to use async/await without transpiling.

I need a way to save a file with the html response from the website so I can inspect it and decide how to best scrape it. And I haven't been able to make it work.

As I found resarching in many places, I need a way to promisify the fs.writeFile() function so that I can await for it. I've tried with util.promisify and also building the custom function below (both produce the same result, which is my cloud function crashing when I await for the result):

index.js (main file for the cloud functions - simplified version)

const functions = require('firebase-functions');
const request = require('request');
const cheerio = require('cheerio');
const fs = require('fs');

// PROMISIFY WRITEFILE TO SAVE URL RESPONSE
const promisifiedWriteFile = function(text) {
  return new Promise((resolve, reject) => {
    fs.writeFile('./response.html', text, err => {
      if (err) {
        return reject(err);
      }
      return resolve();
    });
  });
};

exports.getUrlProductDetails = functions.https.onCall(
  async (data) => {

    try {
      // SEND REQUEST TO URL PRODUCT PAGE, PARSE WITH CHEERIO
      response = await promisifiedRequest(productURL);
      $ = cheerio.load(response.body);

      // TRYING TO SAVE FILE
      await promisifiedWriteFile('12345'); // <----- THIS IS CRASHING
    } // TRY BLOCK - OUTER - END

    catch (err) {
      console.log(err);
      throw new functions.https.HttpsError('unknown', 'Some internal error', err);
    }

  console.log('End of async function...');

  return {
    productDetails: productDetails,
    body: response.body // My client code is getting the body response just fine
  };

Note1: This is a simplified version of the code, but the single line that is crashing the function is: await promisifiedWriteFile('12345'); and when I ment it out, everything else works just fine.

Note2: The file response.html gets created (with empty content), even with the function crashing at that point.

Note3: I should be trying to write the response.body. The '12345' is just for testing. But it crashes either way.

What am I doing wrong?

EDIT

// THIS IS WHAT LOGS OUT OF MY FUNCTION
info: User function triggered, starting execution
info: Execution took 2034 ms, finished with status: 'crash'

And this below is what my client gets: the error message is simply internal, because cloud functions avoid passing internal error details to the client.

Note4: I haven't deployed the function yet. This is all on local development.

I'm currently building a cloud function to scrape some data from emerce websites. I'm running them on the Node.js v8 environment, so I'm able to use async/await without transpiling.

I need a way to save a file with the html response from the website so I can inspect it and decide how to best scrape it. And I haven't been able to make it work.

As I found resarching in many places, I need a way to promisify the fs.writeFile() function so that I can await for it. I've tried with util.promisify and also building the custom function below (both produce the same result, which is my cloud function crashing when I await for the result):

index.js (main file for the cloud functions - simplified version)

const functions = require('firebase-functions');
const request = require('request');
const cheerio = require('cheerio');
const fs = require('fs');

// PROMISIFY WRITEFILE TO SAVE URL RESPONSE
const promisifiedWriteFile = function(text) {
  return new Promise((resolve, reject) => {
    fs.writeFile('./response.html', text, err => {
      if (err) {
        return reject(err);
      }
      return resolve();
    });
  });
};

exports.getUrlProductDetails = functions.https.onCall(
  async (data) => {

    try {
      // SEND REQUEST TO URL PRODUCT PAGE, PARSE WITH CHEERIO
      response = await promisifiedRequest(productURL);
      $ = cheerio.load(response.body);

      // TRYING TO SAVE FILE
      await promisifiedWriteFile('12345'); // <----- THIS IS CRASHING
    } // TRY BLOCK - OUTER - END

    catch (err) {
      console.log(err);
      throw new functions.https.HttpsError('unknown', 'Some internal error', err);
    }

  console.log('End of async function...');

  return {
    productDetails: productDetails,
    body: response.body // My client code is getting the body response just fine
  };

Note1: This is a simplified version of the code, but the single line that is crashing the function is: await promisifiedWriteFile('12345'); and when I ment it out, everything else works just fine.

Note2: The file response.html gets created (with empty content), even with the function crashing at that point.

Note3: I should be trying to write the response.body. The '12345' is just for testing. But it crashes either way.

What am I doing wrong?

EDIT

// THIS IS WHAT LOGS OUT OF MY FUNCTION
info: User function triggered, starting execution
info: Execution took 2034 ms, finished with status: 'crash'

And this below is what my client gets: the error message is simply internal, because cloud functions avoid passing internal error details to the client.

Note4: I haven't deployed the function yet. This is all on local development.

• javascript
• node.js
• firebase
• async-await
• google-cloud-functions

• 2 Can you please share the error msg ? – Stanimirovv Commented Mar 14, 2019 at 11:08
• share err you are getting and also use promisified fs method rather manually wrapping it nodejs/api/fs.html#fs_fs_promises_api or use Promisifly from bluebird or use npm fs-extra – arizafar Commented Mar 14, 2019 at 11:12
• fs_promises_api is not available in Node 8 – user835611 Commented Mar 14, 2019 at 11:21
• @Stanimirovv just added the error details. – cbdeveloper Commented Mar 14, 2019 at 11:47
• @AZ_ just addedd the error details. – cbdeveloper Commented Mar 14, 2019 at 11:48

2 Answers 2

I have managed to do it.

According to this Firebase youtube video, the os.tmpdir() only one directory that is writable by your function's code.

So did the following and now it works:

index.js

const fs = require('fs');
const os = require('os');
const path = require('path');
const tmpdir = os.tmpdir();
const filePath = path.join(tmpdir,'response.html');

// PROMISIFY WRITEFILE TO SAVE URL RESPONSE
const promisifiedWriteFile = function(text) {
  return new Promise((resolve, reject) => {
    fs.writeFile(filePath, text, err => {
      if (err) {
        return reject(err);
      }
      return resolve();
    });
  });
};

And from inside the function I'm calling:

await promisifiedWriteFile(response.body);

Found some related doc also on this link:

https://cloud.google./functions/docs/concepts/exec

I believe you're not allowed to write to the filesystem on Firebase functions. Try writing to /tmp, see Does Cloud Functions for Firebase support file operation?.

But I think you shouldn't be writing response.html, instead you should set the text and status code in the response that the Firebase function returns.