HTML 5 Web workers are very slow when using worker.postMessage on a large JSON object. I'm trying to figure out how to transfer a JSON Object to a web worker - using the 'Transferable Objects' types in Chrome, in order to increase the speed of this.

Here is what I'm referring to and appears it should speed this up quite a bit:

I'm having trouble finding a good example of this (and I don't believe I want to use an ArrayBuffer). Any help would be appreciated.

I'm imagining something like this:

worker = new Worker('workers.js');

var large_json = {};
for(var i = 0; i < 20000; ++i){
   large_json[i] = i;
   large_json["test" + i] = "string";
};

//How to make this call to use Transfer Objects? Takes approx 2 seconds to serialize this for me currently.
worker.webkitPostMessage(large_json);

HTML 5 Web workers are very slow when using worker.postMessage on a large JSON object. I'm trying to figure out how to transfer a JSON Object to a web worker - using the 'Transferable Objects' types in Chrome, in order to increase the speed of this.

Here is what I'm referring to and appears it should speed this up quite a bit: http://updates.html5rocks.com/2011/12/Transferable-Objects-Lightning-Fast

I'm having trouble finding a good example of this (and I don't believe I want to use an ArrayBuffer). Any help would be appreciated.

I'm imagining something like this:

worker = new Worker('workers.js');

var large_json = {};
for(var i = 0; i < 20000; ++i){
   large_json[i] = i;
   large_json["test" + i] = "string";
};

//How to make this call to use Transfer Objects? Takes approx 2 seconds to serialize this for me currently.
worker.webkitPostMessage(large_json);

• javascript
• json
• html
• web-worker

• 5 This is where workers, in their current guise, fall down, in that they have no connection to the parent script, thus anything you want to pass to them involves a tiresome copy operation. Webkit's "pass by reference" approach is definitely the way to go. Other than that, two points strike me: if you're lumbered with a 2-second delay, that probably defeats any saving you might otherwise have gained by using web workers, so you might as well avoid them and, thus, also avoid the serialisation. 2) How about web SQL for data this size - any quicker? – Mitya Commented Jul 6, 2012 at 16:19
• I basically need to manipulate the JSON and transfer it back, so I don't believe web SQL would work. In my normal cases, I wouldn't be transfering this large of a JSON object, but I used it for testing/demonstration purposes. I believe from what I've been reading that the Transferable Objects would make it a very fast pass-by-reference operation, resulting in an extremely timely transfer, far under the current 2 seconds. However, I'm not able to find any example of how to actually do that. – kclem06 Commented Jul 6, 2012 at 16:28
• 1 "complex types like File, Blob, ArrayBuffer, and JSON objects. " -- Appears it should be supported: developer.mozilla.org/en/Using_web_workers – kclem06 Commented Jul 6, 2012 at 20:53
• 3 I am facing the same exact problem. The solution I was toying with was using the ArrayBuffer and stringifying the json then converting that string to a UInt16Array (updates.html5rocks.com/2012/06/…) but that process is also quite time-consuming. Still haven't found any method of posting large JSON objects to and from a worker in such a way that makes using web workers worth it :-/ – stinkycheeseman Commented Jul 12, 2012 at 20:41
• 1 @stinkycheeseman - Thanks for the input, I was wanting to try this but I wasn't sure exactly how to do the conversion. In regards to the timing for just transferring the ArrayBuffer (not converting it to an ArrayBuffer), approximately how quick is that? I'd imagine its a few ms at most, but I haven't actually tried it. – kclem06 Commented Jul 13, 2012 at 19:55

3 Answers 3

Using a transferable object won't help if you have to build it from scratch from an existing Json array (That's very close to cloning...)

Where does the Json data comes from? One possible way to keep all the hard work on the worker thread is to have it fetch the data using XmlHttpRequest, transform it and send it to the UI thread. This way, the high cost of cloning occurs on the worker thread and while it will take the same time as it would in the UI thread, it won't block your app.

Ok i did this i dont know if it's good or bad, ideal or worst way to do it. I just did it. in worker file

var data = e.data;
var string = String.fromCharCode.apply(null, new Uint16Array(data));
var objnow = JSON.parse(string);

in html file

function str2ab(str) {
   var buf = new ArrayBuffer(str.length*2); // 2 bytes for each char
   var bufView = new Uint16Array(buf);
   for (var i=0, strLen=str.length; i<strLen; i++) {
     bufView[i] = str.charCodeAt(i);
   }
   return buf;
 }
function stop() {
 var obj = {'cmd': 'stop', 'msg': 'Bye'};
 var str= JSON.stringify(obj);
 var arbfr = str2ab(obj);
 worker.postMessage(arbfr,[arbfr]);
}

And now it works i am able to send json object, transfer.

While not using the 'Transferable Objects', but this may solve your problem.

You may try to optimize your data representation as well. E.g. your example takes ~1350ms to pack/unpack for me (Google Chrome 19), but the following code is executed ~25 times faster (50ms):

console.time('json');
var a = [], test = [];
for(var i = 0; i < 20000; ++i){
   a.push(i);
   test.push("string");
};
var large_json = {
  a: a.join(','),
  test: test.join(',')
};
large_json = JSON.parse(JSON.stringify(large_json));
large_json.a = large_json.a.split(",");
large_json.test = large_json.test.split(",");
console.timeEnd('json');