I'm doing the following request (using request/request) against a web service:

return request.postAsync({
    url,
    charset: 'Cp1252', // I also tried utf-8
    encoding: null, //
    // I also tried Cp1252 -> unknown encoding,
    // I also tried utf-8 and nothing at all
    headers: {
         "Accept": "application/octet-stream, text, text/plain, text/xml",
         "Accept-Encoding": "UTF-8",
         'Content-Type': "text/plain; charset=Cp1252;", // also tried utf-8, ISO-8859-1
         "User-Agent": "me"
    }
}).spread((res, body) => {
    body = body.toString();  // I also tried without toString();
    let ws = fs.createWriteStream('hhh.csv');
    ws.write(body);
    ws.end();

Whatever I do, umlauts are turned into �.

Those are the headers the web service sends back:

'content-type': 'text; charset=Cp1252',
'content-length': '1895980',
vary: 'Accept-Encoding,User-Agent'

I'm trying this for days with no luck at all. What am I doing wrong?

Here's a list of question/answers that didn't solve my problem so far:

• Umlauts broken when doing get request
• Problems making http-post requests with German umlauts (ä ö ü) or special chars (ß) with Node.js
• Send umlauts from Node.js to iOS
• Automatic UTF-8 encoding in Node.js HTTP client
• Custom HTTP header value - trying to pass umlaut characters
• How to get UTF-8 in Node.js?
• How to get i18next-node to display umlauts the right way?
• How do I URl encode something in Node.js?

Can it be that one of the following causes my input string to not be UTF-8?

let hash = crypto.createHmac("sha256", this.options.Signer);
this.query['Signature'] = hash.update(stringToSign).digest("base64");

signer is a string containing 0-9, a-z, A-Z, +, and /.

this.query['Signature'] is part of the URL.

I'm doing the following request (using request/request) against a web service:

return request.postAsync({
    url,
    charset: 'Cp1252', // I also tried utf-8
    encoding: null, //
    // I also tried Cp1252 -> unknown encoding,
    // I also tried utf-8 and nothing at all
    headers: {
         "Accept": "application/octet-stream, text, text/plain, text/xml",
         "Accept-Encoding": "UTF-8",
         'Content-Type': "text/plain; charset=Cp1252;", // also tried utf-8, ISO-8859-1
         "User-Agent": "me"
    }
}).spread((res, body) => {
    body = body.toString();  // I also tried without toString();
    let ws = fs.createWriteStream('hhh.csv');
    ws.write(body);
    ws.end();

Whatever I do, umlauts are turned into �.

Those are the headers the web service sends back:

'content-type': 'text; charset=Cp1252',
'content-length': '1895980',
vary: 'Accept-Encoding,User-Agent'

I'm trying this for days with no luck at all. What am I doing wrong?

Here's a list of question/answers that didn't solve my problem so far:

• Umlauts broken when doing get request
• Problems making http-post requests with German umlauts (ä ö ü) or special chars (ß) with Node.js
• Send umlauts from Node.js to iOS
• Automatic UTF-8 encoding in Node.js HTTP client
• Custom HTTP header value - trying to pass umlaut characters
• How to get UTF-8 in Node.js?
• How to get i18next-node to display umlauts the right way?
• How do I URl encode something in Node.js?

Can it be that one of the following causes my input string to not be UTF-8?

let hash = crypto.createHmac("sha256", this.options.Signer);
this.query['Signature'] = hash.update(stringToSign).digest("base64");

signer is a string containing 0-9, a-z, A-Z, +, and /.

this.query['Signature'] is part of the URL.

• javascript
• node.js
• encoding

• What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? – zerkms Commented Oct 5, 2015 at 21:53
• In the place where the umlaut is "still" umlaut. – zerkms Commented Oct 5, 2015 at 21:56
• "node js request turns umlauts to �" "Whatever I do, umlauts are turned into �." --- what these phrases mean then? – zerkms Commented Oct 5, 2015 at 21:58
• What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? – zerkms Commented Oct 5, 2015 at 22:01
• 1 Have you tried changing the defaultEncoding of your writeStream? i.e. fs.createWriteStream('hhh.csv', {defaultEncoding: 'utf8'}); – Yan Foto Commented Oct 5, 2015 at 22:03

1 Answer 1

I finally solved it, using iconv-lite and setting request's encoding¹ to null, making it return the body as a Buffer. Here is my now working configuration:

return request.getAsync({
        url,
        encoding: null,
        headers: {
            "Accept": "text, text/plain, text/xml",
            "Accept-Encoding": "UTF-8",
            'Content-Type': "text/plain; charset=utf-8;",
            "User-Agent": "me"
        }
    }).spread((res, body) => {
        let a = iconv.decode(new Buffer(body), 'cp1252');
        // now a is holding a string with correct Umlauts and ß