I am trying to execute a sample performance benchmark using Benchmark.js. Here is what I have wrote:

var Benchmark = require('benchmark');
var arr = []
benchmark = new Benchmark('testPerf',function(){
    arr.push(1000);
},
{
    delay: 0,
    initCount: 1,
    minSamples: 1000,
    onComplete : function(){ console.log(this);},
    onCycle: function(){}
});
benchmark.run();

Now like we do in JUnitBenchmarks:

@BenchmarkOptions(clock = Clock.NANO_TIME, callgc = true, benchmarkRounds = 10, warmupRounds = 1)

Here also I want to declare benchmarkRounds and warmupRounds count in benchmarkjs. I think warmupRounds maps to initCount? And how to set exact number of cycles/benchmark iteration?

Or if we have some other good JavaScript library that can handle it would work too.

I am trying to execute a sample performance benchmark using Benchmark.js. Here is what I have wrote:

var Benchmark = require('benchmark');
var arr = []
benchmark = new Benchmark('testPerf',function(){
    arr.push(1000);
},
{
    delay: 0,
    initCount: 1,
    minSamples: 1000,
    onComplete : function(){ console.log(this);},
    onCycle: function(){}
});
benchmark.run();

Now like we do in JUnitBenchmarks:

@BenchmarkOptions(clock = Clock.NANO_TIME, callgc = true, benchmarkRounds = 10, warmupRounds = 1)

Here also I want to declare benchmarkRounds and warmupRounds count in benchmarkjs. I think warmupRounds maps to initCount? And how to set exact number of cycles/benchmark iteration?

Or if we have some other good JavaScript library that can handle it would work too.

• javascript
• benchmark.js

• Benchmarking in JavaScript using a fixed iteration count is problematic, so Benchmark.js does not allow that option. Instead, it runs tests over and over again until results can be considered reasonably accurate. – approxiblue Commented Sep 30, 2015 at 1:47
• @approxiblue Please post this comment as answer. – Tariq Commented Oct 1, 2015 at 6:19

3 Answers 3

Using fixed iteration counts in JavaScript benchmarks is risky: we might get zero-time results eventually, as browsers become faster.

Benchmark.js does not allow setting the number of rounds/iterations in advance. Instead it runs tests over and over again until results can be considered reasonably accurate. You should check out the code reading by Monsur Hossain. Some highlights from the article:

• A cycle in Benchmark.js consists of set-up, tear-down, and multiple iterations of the actual test.
• Benchmark.js starts with the analysis phase: run a few cycles to find the optimal number of iterations (finish the test as fast as possible while collecting enough samples to generate accurate results).
• The number of cycles run during analysis is saved in Benchmark.prototype.cycles.
• Knowing the optimal number of iterations, Benchmark.js begins the sampling phase: runs the tests and actually stores the results.
• Benchmark.prototype.stats.sample is an array of results from each cycle during sampling.
• Benchmark.prototype.count is the number of iterations during sampling.

Looking over the docs at:

http://benchmarkjs.com/docs

Sounds like you are correct

1. warmupRounds => initCount (http://benchmarkjs.com/docs#options_initCount)
2. cycles => http://benchmarkjs.com/docs#prototype_cycles

Regardless if it's a good idea, if you set minTime and maxTime to some negative values then minSamples and initCount will be left as the only criteria and they will correspond to #cycles and warmup iterations run at each cycle. So the test function will be executed (initCount+1) * minSamples times. At least that's what my experiments show.

var Benchmark = require('benchmark');
var counter = 0;
Benchmark('counting', {
  'fn': function() { ++counter; },
  minSamples: 3,
  initCount: 1,
  minTime: -Infinity,
  maxTime: -Infinity,
  onCycle: function () { console.log('[onCycle] counter: ' + counter); },
  onComplete : function(){ console.log('mean: ' + this.stats.mean);},
}).run();

gives me with benchmark.js 2.1.0:

$ node count.js
[onCycle] counter: 2
[onCycle] counter: 4
[onCycle] counter: 6
mean: 0.0000034683333333333333