I'm making a web scraper, most of the data on the web page is in JavaScript object literal form, e.g.:

// Silly example
var user = {
    name: 'John', 
    surname: 'Doe',
    age: 21,
    family: [
        {
            name: 'Jane',
            surname: 'Doe',
            age: 37
        },
        // ...
    ]
};

So when I search for the contents in my JavaScript app the Object above would be:

"{name: 'John', surname: 'Doe', age: 21, family: [{name: 'Jane', surname: 'Doe', age: 37}]}"

Is it possible to parse those to regular JavaScript Objects without using 'eval' or making my own parser? I saw other similar questions about this but the answers are not applicable: they all suggest JSON.parse() (not applicable) and eval (I can't use it for security reasons). In this question, for example, all the answers suggest eval or new Function() which are basically the same thing.

If there are no other ways would it be a viable option to convert the literal to proper JSON and then parse it to JavaScript object?

This is what I tried right now, it worked on a simple object but I'm not sure it will work everywhere:

const literal = script.innerText.slice(script.innerText.indexOf('{'), script.innerText.lastIndexOf('}') + 1);
const json = literal.replace(/.*:.*(\".*\"|\'.*\'|\[.*\]|\{.*\}|true|false|[0-9]+).*,/g, (prev) => {
  let parts = prev.split(':');
  let key = '"' + parts.shift().trim() + '"';
  let value = parts.join(':').replace(/'.*'/, (a) => {
    return '"' + a.slice(1, a.length - 1) + '"';
  }).trim();
  return key + ':' + value;
});
const obj = JSON.parse(json);

I'm making a web scraper, most of the data on the web page is in JavaScript object literal form, e.g.:

// Silly example
var user = {
    name: 'John', 
    surname: 'Doe',
    age: 21,
    family: [
        {
            name: 'Jane',
            surname: 'Doe',
            age: 37
        },
        // ...
    ]
};

So when I search for the contents in my JavaScript app the Object above would be:

"{name: 'John', surname: 'Doe', age: 21, family: [{name: 'Jane', surname: 'Doe', age: 37}]}"

Is it possible to parse those to regular JavaScript Objects without using 'eval' or making my own parser? I saw other similar questions about this but the answers are not applicable: they all suggest JSON.parse() (not applicable) and eval (I can't use it for security reasons). In this question, for example, all the answers suggest eval or new Function() which are basically the same thing.

If there are no other ways would it be a viable option to convert the literal to proper JSON and then parse it to JavaScript object?

This is what I tried right now, it worked on a simple object but I'm not sure it will work everywhere:

const literal = script.innerText.slice(script.innerText.indexOf('{'), script.innerText.lastIndexOf('}') + 1);
const json = literal.replace(/.*:.*(\".*\"|\'.*\'|\[.*\]|\{.*\}|true|false|[0-9]+).*,/g, (prev) => {
  let parts = prev.split(':');
  let key = '"' + parts.shift().trim() + '"';
  let value = parts.join(':').replace(/'.*'/, (a) => {
    return '"' + a.slice(1, a.length - 1) + '"';
  }).trim();
  return key + ':' + value;
});
const obj = JSON.parse(json);

• javascript

• 4 An object literal is a JS object. If you need a JSON string, then see developer.mozilla/en-US/docs/Web/JavaScript/Reference/… – Teemu Commented Jun 18, 2019 at 12:24
• 2 @Teemu but based on the statement that OP is using a web scraper, I'd assume that he has it in string format. I'd be interested in why it isn't possible to use JSON.parse – Tom M Commented Jun 18, 2019 at 12:25
• 1 You could use JavaScript parser like esprima or acorn – ponury-kostek Commented Jun 18, 2019 at 12:28
• 2 @TomM It looks like I'm not reading enough the question nowadays. Yes, the provided example is most likely a string, but it is not a JSON string, that's why JSON.parse can't parse it. – Teemu Commented Jun 18, 2019 at 12:28
• 1 @TomM I'd be interested in why it isn't possible to use JSON.parse Because even if you take away var user = and the trailing ;, its not valid JSON. eg.. {hello: 'there'} is not valid JSON. – Keith Commented Jun 18, 2019 at 12:29

4 Answers 4

It's simple demo how you can use esprima to get globally declared variables

"use strict";

const src = `
var user = {
	name: 'John',
	surname: 'Doe',
	age: 21,
	family: [
		{
			name: 'Jane',
			surname: 'Doe',
			age: 37
		},
		// ...
	]
};`;
const src2 = `
var a = [1,2,3], b = true;
var s = "some string";
var o = {a:1}, n = null;
var some = {'realy strange' : {"object":"'literal'"}}
`;

function get_globals(src) {
  return esprima.parse(src).body
    .filter(({type}) => type === "VariableDeclaration") // keep only variables declarations
    .map(({declarations}) => declarations)
    .flat()
    .filter(({type}) => type === "VariableDeclarator")
    .reduce((vars, {id: {name}, init}) => {
      vars[name] = parse(init);
      return vars;
    }, {});
}

console.log(get_globals(src));
console.log(get_globals(src2));

/**
 * Parse expression
 * @param expression
 * @returns {*}
 */
function parse(expression) {
  switch (expression.type) {
    case "ObjectExpression":
      return ObjectExpression(expression);
    case "Identifier":
      return expression.name;
    case "Literal":
      return expression.value;
    case "ArrayExpression":
      return ArrayExpression(expression);
  }
}

/**
 * Parse object expresion
 * @param expression
 * @returns {object}
 */
function ObjectExpression(expression) {
  return expression.properties.reduce((obj, {key, value}) => ({
    ...obj,
    [parse(key)]: parse(value)
  }), {});
}

/**
 * Parse array expression
 * @param expression
 * @returns {*[]}
 */
function ArrayExpression(expression) {
  return expression.elements.map((exp) => parse(exp));
}

<script src="https://unpkg./esprima@~4.0/dist/esprima.js"></script>

For data like this you could might be able to use a couple of regex's to convert into a valid JSON object.

Below is an example..

ps. It might not be 100% foolproof for all object literals.

var str = "{name: 'John', surname: 'Doe', age: 21, family: [{name: 'Jane', surname: 'Doe', age: 37}]}";

var jstr = str
  .replace(/\'(.*?)\'/g, '"$1"')
  .replace(/([\{ ]*?)([a-z]*?)(\:)/gi, '$1"$2"$3');

var obj = JSON.parse(jstr);

console.log(obj);

As pointed out by @ponury-kostek, and by myself using regEx can be limited. Using some sort of AST parsing like Esprima is certainly a good idea, especially if your already using an AST parser.

But if an AST parser is overkill, a more robust version below using Javascript might be better. Ps. again it might not be 100% correct, but it should cope with the majority of Object literals.

var str = `{
  name: 'John:', surname: 'Doe', age: 21,
  family: [
    {name: '