I have a binary document (mp4 video file) in a database (MarkLogic). I am using the database's Node.js API to stream the document in chunks. The setup looks like this:
html file
<video controls="controls" width="600">
<source src="/video/myvideo.mp4" type="video/mp4">
</video>
In express I have then setup a route which handles the /video/:param route (in the database the video has the unique identifier which is the string '/video/myvideo.mp4')
node.js
// I'm only showing the relevant things in here
const serveVideo = (req, res) => {
var stream = db.documents.read('/gopro/malta.mp4').stream('chunked');
var chunks = [];
var chunkBytes = 0;
var start = 0;
stream.on('data', (chunk) => {
var headers;
var range = req.headers.range;
var total = 214335483; //total length of vid in bytes
if (range) {
var chunkSize = chunk.length;
// (start === 0) ? start = 0 : start += chunkBytes;
if (chunkBytes === 0) {
start = 0
} else {
start = chunkBytes + 1
}
chunkBytes += chunkSize;
headers = {
'Content-Range': 'bytes ' + start + '-' + chunkBytes + '/' + total,
'Accept-Ranges': 'bytes',
'Content-Length': chunkSize,
'Content-Type': 'video/mp4'
};
res.writeHead(206, headers);
chunks.push(chunk);
}
});
stream.on('end', () => {
var allChunks = Buffer.concat(chunks);
res.end(allChunks);
});
});
router.route('/video/:uri').get(serveVideo);
Now of course the above fails with 'Error: Can't set headers after they are sent.' which is all fair and square. But I can't get my head around this - the .stream('chunked') call forces the database to retrieve the document in chunks and I do see those chunks just fine, however how can I return a 206 for the browser? I can't do it in the .on('data') as data is being streamed so the header would be sent multiple times. I guess which database I'm using is not really relevant - I would like to understand the concept, or at least see what I'm doing wrong.
Any help is appreciated. All the examples and other discussions that I have seen that stream video using Node.js are reading the video file from disk.
update
Making a change to the code now allows FF to play the video but not Chrome:
let stream = db.documents.read({uris:'/gopro/malta.mp4'}).stream('chunked');
stream.pipe(res);
There are no errors in Chrome's console. Here are the header details - note that there are two requests for the mp4 file:
1st
Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:30 GMT
Transfer-Encoding:chunked
X-Powered-By:Express
Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=0-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
2nd
Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:31 GMT
Transfer-Encoding:chunked
X-Powered-By:Express
Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=28-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
I have a binary document (mp4 video file) in a database (MarkLogic). I am using the database's Node.js API to stream the document in chunks. The setup looks like this:
html file
<video controls="controls" width="600">
<source src="/video/myvideo.mp4" type="video/mp4">
</video>
In express I have then setup a route which handles the /video/:param route (in the database the video has the unique identifier which is the string '/video/myvideo.mp4')
node.js
// I'm only showing the relevant things in here
const serveVideo = (req, res) => {
var stream = db.documents.read('/gopro/malta.mp4').stream('chunked');
var chunks = [];
var chunkBytes = 0;
var start = 0;
stream.on('data', (chunk) => {
var headers;
var range = req.headers.range;
var total = 214335483; //total length of vid in bytes
if (range) {
var chunkSize = chunk.length;
// (start === 0) ? start = 0 : start += chunkBytes;
if (chunkBytes === 0) {
start = 0
} else {
start = chunkBytes + 1
}
chunkBytes += chunkSize;
headers = {
'Content-Range': 'bytes ' + start + '-' + chunkBytes + '/' + total,
'Accept-Ranges': 'bytes',
'Content-Length': chunkSize,
'Content-Type': 'video/mp4'
};
res.writeHead(206, headers);
chunks.push(chunk);
}
});
stream.on('end', () => {
var allChunks = Buffer.concat(chunks);
res.end(allChunks);
});
});
router.route('/video/:uri').get(serveVideo);
Now of course the above fails with 'Error: Can't set headers after they are sent.' which is all fair and square. But I can't get my head around this - the .stream('chunked') call forces the database to retrieve the document in chunks and I do see those chunks just fine, however how can I return a 206 for the browser? I can't do it in the .on('data') as data is being streamed so the header would be sent multiple times. I guess which database I'm using is not really relevant - I would like to understand the concept, or at least see what I'm doing wrong.
Any help is appreciated. All the examples and other discussions that I have seen that stream video using Node.js are reading the video file from disk.
update
Making a change to the code now allows FF to play the video but not Chrome:
let stream = db.documents.read({uris:'/gopro/malta.mp4'}).stream('chunked');
stream.pipe(res);
There are no errors in Chrome's console. Here are the header details - note that there are two requests for the mp4 file:
1st
Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:30 GMT
Transfer-Encoding:chunked
X-Powered-By:Express
Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=0-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
2nd
Response Headers
Connection:keep-alive
Date:Sat, 21 May 2016 17:05:31 GMT
Transfer-Encoding:chunked
X-Powered-By:Express
Request Headers
view source
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,it;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Cookie:__distillery=v20150227_a8e22306-65b3-4c2e-9a8a-159e308156ad; __smToken=7nYU8NYQY15mPowjjCZsS5D3
DNT:1
Host:localhost:8080
Pragma:no-cache
Range:bytes=28-
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
- Have you proven that your code above works by reading from the disk? I suggest that as the first step of troubleshooting. Once you break the back on that the general code is sound, then add the DB steps back in. – David Ennis -CleverLlamas. Commented May 20, 2016 at 6:43
- Good question! I d say you should not call for writeHead, but that you ll need to dig into express to write header directly. You can also take a look to this code which i know can stream a video with 206 responses, it is used in peerflix. see also gist.github./paolorossi/1993068 – user4466350 Commented May 20, 2016 at 6:50
-
You must most likely don't need all the chunk stuff. Set headers manually:
res.status(206);. Then simply pipe the response:let stream = db.yourChunkStuff(); stream.pipe(res);. On a phone here so can't test, so only putting it in a ment now. – Zlatko Commented May 21, 2016 at 8:35 - Also a side ment: you use const and arrows and then a lot of vars and not lets, as is mon in Es2015 codebases, I wonder why? – Zlatko Commented May 21, 2016 at 8:39
-
@Zlatko - thanks. So the following:
let stream = db.documents.read({uris:'/gopro/malta.mp4'}).stream('chunked'); stream.pipe(res);works in FF but not Chrome/Safari. If I addres.status(206)as well it doesn't work in either of those browsers. – Tamas Commented May 21, 2016 at 16:19
2 Answers
Reset to default 4The on('data') callback could always have a closure over an isFirstChunk variable that is initialized to true and have a test that, if isFirstChunk is true, emits the headers and sets isFirstChunk to false.
It would be preferable to pipe the stream if possible. npm might offer a stream library (maybe even through2()?) that has an event when the first data arrives.
For the longer term, you could reasonably file an RFE for an event at the start of a stream.
Hoping that helps,
You most likely don't need all the chunk stuff. Set headers manually:
res.status(206);
Then simply pipe the response:
let stream = db.yourChunkStuff();
stream.pipe(res);
The simplest way to is pipe streams.