Our software uses ffmpeg to do various video/audio encoding (e.g. changing volume, re-encoding to a different video format (e.g. yuv420p), scaling, etc.)

Every few months we upgrade ffmpeg versions, and we've found that sometimes parameters change and regressions are introduced.

What ffmpeg tools could we use to create unit tests to validate certain commands for regressions?

I believe checking hashes won't be enough; doing visual inspections manually is also something we'd like to avoid to automate.

Here's an example pseudocode:

var ffmpegV1 = "C:\ffmpegv1.exe";
var ffmpegV2 = "C:\ffmpegv2.exe";

var inputVideo = "C:\video.mp4";

var outputVideoV1 = "C:\OutputV1.mp4";
var outputVideoV2 = "C:\OutputV2.mp4";


ffmpegV1.Encode($"-y -i {inputVideo} -ar 22050 -ab 64k -ac 1 -s 120x40 -crf 30 -g 150 OutputV1.mp4");


ffmpegV1.Encode($"-y -i {inputVideo} -ar 22050 -ab 64k -ac 1 -s 120x40 -crf 30 -g 150 OutputV2.mp4");



// This is where I want to figure out how I can programmatically determine that the videos are the same, or within some tolerance of difference, e.g.

bool isAudioTheSame = FfmpegTest.CompareAudio(outputVideoV1, outputVideoV2, Epsilon.Tolerance);

bool isVideoTheSame = FfmpegTest.CompareVideo(outputVideoV1, outputVideoV2, Epsilon.Tolerance);

I'm not a video/audio expert, so I'm not really sure how to flesh these out. Do I just compare the bytes? Or is there another ffmpeg tool/command to use?

Our software uses ffmpeg to do various video/audio encoding (e.g. changing volume, re-encoding to a different video format (e.g. yuv420p), scaling, etc.)

Every few months we upgrade ffmpeg versions, and we've found that sometimes parameters change and regressions are introduced.

What ffmpeg tools could we use to create unit tests to validate certain commands for regressions?

I believe checking hashes won't be enough; doing visual inspections manually is also something we'd like to avoid to automate.

Here's an example pseudocode:

var ffmpegV1 = "C:\ffmpegv1.exe";
var ffmpegV2 = "C:\ffmpegv2.exe";

var inputVideo = "C:\video.mp4";

var outputVideoV1 = "C:\OutputV1.mp4";
var outputVideoV2 = "C:\OutputV2.mp4";


ffmpegV1.Encode($"-y -i {inputVideo} -ar 22050 -ab 64k -ac 1 -s 120x40 -crf 30 -g 150 OutputV1.mp4");


ffmpegV1.Encode($"-y -i {inputVideo} -ar 22050 -ab 64k -ac 1 -s 120x40 -crf 30 -g 150 OutputV2.mp4");



// This is where I want to figure out how I can programmatically determine that the videos are the same, or within some tolerance of difference, e.g.

bool isAudioTheSame = FfmpegTest.CompareAudio(outputVideoV1, outputVideoV2, Epsilon.Tolerance);

bool isVideoTheSame = FfmpegTest.CompareVideo(outputVideoV1, outputVideoV2, Epsilon.Tolerance);

I'm not a video/audio expert, so I'm not really sure how to flesh these out. Do I just compare the bytes? Or is there another ffmpeg tool/command to use?

• c#
• ffmpeg

• Can you give an example of one of the tests you would like to do. Possibly with pseudocode – shotor Commented Feb 2 at 2:03
• 1 just did, thanks – lavantgarde Commented Feb 3 at 19:43

1 Answer 1

This was a fun exercise, here's a starting point. I used ffmpeg for all of it. But you should be able to use a nuget package to get structured output.

Given 2 mp4 files called 6.1.mp4 and 7.1.mp4 encoded from the same input file using different versions of ffmpeg.

We first split the audio and video channel from both files (I'm comparing ffmpeg 6.1 and 7.1 in this case)

./bin/6.1/ffmpeg -y -i ./output/6.1.mp4 -an -c:v copy ./output/6.1-video.mp4 -vn -c:a pcm_s16le -ar 44100 -ac 2 ./output/6.1-audio.wav
./bin/7.1/ffmpeg -y -i ./output/7.1.mp4 -an -c:v copy ./output/7.1-video.mp4 -vn -c:a pcm_s16le -ar 44100 -ac 2 ./output/7.1-audio.wav

Audio

Then we invert the audio files, we're going to subtract the inverted audio file from the other versions original later (as per: https://forum.audacityteam./t/comparing-two-supposedly-identical-tracks/36424/2)

./bin/6.1/ffmpeg -y -i ./output/6.1-audio.wav -af "volume=-1" ./output/6.1-audio-inverted.wav
./bin/7.1/ffmpeg -y -i ./output/7.1-audio.wav -af "volume=-1" ./output/7.1-audio-inverted.wav

Next we mix the audio tracks. 6.1-inverted.wav with 7.1.wav. Per: How to overlay/downmix two audio files using ffmpeg.

This subtracts the waveform of 6.1 from the waveform of 7.1. If they're equal the result will be total silence. If they're not, there will be some audio remaining.

./bin/7.1/ffmpeg -y -i ./output/6.1-audio-inverted.wav -i ./output/7.1-audio.wav -filter_complex "[0:a][1:a]amix=inputs=2:duration=first:normalize=0" ./output/6.1-audio-inverted-7.1-audio-mixed.wav

(Optional) Generate a spectogram of the mixed audio file for visual inspection. If the audio files are identical it should be blank.

./bin/7.1/ffmpeg -y -i ./output/6.1-audio-inverted-7.1-audio-mixed.wav -lavfi showspectrumpic=s=800x600:legend=disabled -frames:v 1 -update 1 -vsync vfr ./output/6.1-audio-inverted-7.1-audio-mixed-spectrogram.png

Which it is for my sample and versions of ffmpeg:

To understand programmatically whether the files are identical we use the volumedetect filter on the mixed audio file. If it's below -80db the files are identical. This part you'll want to use a dotnet library to get structured output. Or just grep it.

./bin/7.1/ffmpeg -y -i ./output/6.1-audio-inverted-7.1-audio-mixed.wav -af volumedetect -f null /dev/null

[Parsed_volumedetect_0 @ 0x7b5ac0002cc0] n_samples: 4112384
[Parsed_volumedetect_0 @ 0x7b5ac0002cc0] mean_volume: -91.0 dB
[Parsed_volumedetect_0 @ 0x7b5ac0002cc0] max_volume: -91.0 dB
[Parsed_volumedetect_0 @ 0x7b5ac0002cc0] histogram_91db: 4112384

A more complex way to compare the audio files is to export the raw values of the spectrogram histogram and use a similarity algorithm to compare them.

Video

Video was a bit easier, there's an SSIM filter. I don't know what it means, but if you give it 2 files it'll tell you how identical they are.

The closer the output value is to 1, the more identical the video files are.

./bin/7.1/ffmpeg -i ./output/6.1-video.mp4 -i ./output/7.1-video.mp4 -filter_complex "ssim" -f null -

[Parsed_ssim_0 @ 0x7a5480003f40] SSIM Y:1.000000 (inf) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (inf)

Here's a git repo to play around with it: https://github/shotor/ffmpeg-test

Hope this helps.