Recently I've e across some very strange behavior related to character encoding for AJAX calls made using the POST method. To make a long story short, I have an HTML form with text fields that can accept diacritics (e.g. "ä"). When the form is submitted, the form data is wrapped in an XML block and sent to a server, which stores that information in a MySQL database. Subsequently, that information is retrieved from the database and displayed to regular users, as is.

If the request is sent from Chrome or IE, everything is fine. This means that the data, including the diacritics, is sent, stored, then retrieved and displayed correctly. However, when I use Firefox for this, the XML appears to submit the form data right, but when I reload the web page, the previously sent diacritics don't appear. In other words, they seem to get lost somewhere along the way. For example, if the XML contains the word "tästä", when I load the page I see "tst".

Why is this happening? Is Firefox encoding the post messages differently from IE and Chrome?

In case it helps, I've attached the request and response headers from Chrome and Firefox, for exactly the same form content - only one example:

By the way, I'm not encoding the data before sending it to the server, just simply retrieving the value of the form fields, as is.

CHROME:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:562
Content-Type:application/x-www-form-urlencoded
Cookie:PHPSESSID=rlne2d787j0np52ec5rtn04dm1
Host:83.150.87.220
Origin:
Referer:http:///?c=2094211
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1
X-Requested-With:XMLHttpRequest

The response headers:

Connection:Keep-Alive
Content-Encoding:gzip
Content-Type:application/xml
Date:Mon, 17 Sep 2012 16:21:58 GMT
Keep-Alive:timeout=5, max=100
Server:Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding:chunked
Vary:Accept-Encoding

FIREFOX:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept  */*
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Connection  keep-alive
Content-Length  562
Content-Type    application/x-www-form-urlencoded; charset=UTF-8
Cookie  PHPSESSID=kvfg4fp2trorllim19dmn241c7
Host    hidden.by.me
Referer /?c=2094211
User-Agent  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1
X-Requested-With    XMLHttpRequest

The response headers:

Connection  Keep-Alive
Content-Encoding    gzip
Content-Type    application/xml
Date    Mon, 17 Sep 2012 16:21:23 GMT
Keep-Alive  timeout=5, max=100
Server  Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding   chunked
Vary    Accept-Encoding

Recently I've e across some very strange behavior related to character encoding for AJAX calls made using the POST method. To make a long story short, I have an HTML form with text fields that can accept diacritics (e.g. "ä"). When the form is submitted, the form data is wrapped in an XML block and sent to a server, which stores that information in a MySQL database. Subsequently, that information is retrieved from the database and displayed to regular users, as is.

If the request is sent from Chrome or IE, everything is fine. This means that the data, including the diacritics, is sent, stored, then retrieved and displayed correctly. However, when I use Firefox for this, the XML appears to submit the form data right, but when I reload the web page, the previously sent diacritics don't appear. In other words, they seem to get lost somewhere along the way. For example, if the XML contains the word "tästä", when I load the page I see "tst".

Why is this happening? Is Firefox encoding the post messages differently from IE and Chrome?

In case it helps, I've attached the request and response headers from Chrome and Firefox, for exactly the same form content - only one example:

By the way, I'm not encoding the data before sending it to the server, just simply retrieving the value of the form fields, as is.

CHROME:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:562
Content-Type:application/x-www-form-urlencoded
Cookie:PHPSESSID=rlne2d787j0np52ec5rtn04dm1
Host:83.150.87.220
Origin:http://hidden.by.me
Referer:http://http://hidden.by.me/?c=2094211
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1
X-Requested-With:XMLHttpRequest

The response headers:

Connection:Keep-Alive
Content-Encoding:gzip
Content-Type:application/xml
Date:Mon, 17 Sep 2012 16:21:58 GMT
Keep-Alive:timeout=5, max=100
Server:Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding:chunked
Vary:Accept-Encoding

FIREFOX:

The XML data block:

<request>
<session>{hidden by me}</session>
<builder>Hem i Stan tästä</builder>
</request>

The request headers:

Accept  */*
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Connection  keep-alive
Content-Length  562
Content-Type    application/x-www-form-urlencoded; charset=UTF-8
Cookie  PHPSESSID=kvfg4fp2trorllim19dmn241c7
Host    hidden.by.me
Referer http://hidden.by.me/?c=2094211
User-Agent  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1
X-Requested-With    XMLHttpRequest

The response headers:

Connection  Keep-Alive
Content-Encoding    gzip
Content-Type    application/xml
Date    Mon, 17 Sep 2012 16:21:23 GMT
Keep-Alive  timeout=5, max=100
Server  Apache/2.2.11 (Win32) PHP/5.2.9-1
Transfer-Encoding   chunked
Vary    Accept-Encoding

• javascript
• ajax
• firefox
• character-encoding

• 1 Notice that the Content-type headers are different: Firefox is sending UTF-8 to your server. – Pointy Commented Sep 17, 2012 at 17:03
• But in both cases, the character encoding is UTF-8. Isn't that just an issue of information layout in Firebug as opposed to the Chrome Inspector? – Andrei Oniga Commented Sep 17, 2012 at 17:36
• I meant the Content-Type in the request header. In Firefox, according to what you posted, it's " application/x-www-form-urlencoded; charset=UTF-8", but that "charset" clause is missing from the Chrome information. Whether it's actually being posted in UTF-8 I can't say; your server should be able to tell. The problem has to be something like that, in any case. – Pointy Commented Sep 17, 2012 at 17:38
• I understood what you meant, and it's exactly what I was saying, that I believe that to be a matter of data layout in the 2 applications (Firebug/Chrome's code inspector). In other words, the HTTP request parameters, including the char encoding, are set within the JS script. But Firebug displays it next to the content type. Anyway, I'm not sure about it. – Andrei Oniga Commented Sep 17, 2012 at 21:23
• Well, either the two browsers are sending different encodings, or they're expecting different encodings in the response. Such problems are a real pain to figure out :/ You should be able to tell at the server if the strings are arriving byte-for-byte identically, I think. – Pointy Commented Sep 17, 2012 at 21:56

2 Answers 2

As @Pointy mentioned some time ago, the problem was related to the Content-Type of the POST request, because Firefox appears to encode POST messages differently than other browsers. In my head, Data-Type and Content-Type were the same and so, I didn't realize it's necessary to specify UTF-8 as the character encoding standard in both cases. But once I had changed both the Content-Type and the Data-Type as well to a clear "text/xml; charset=UTF-8", the problem was resolved.

I am soooo happy. Thank you guys for posting and figuring this out earlier. It took me a couple hours to get close enough to the problem to find this through googling, but because of your ments, I got this solved in less than a day; and in time for the big presentation tomorrow! :)

It was so bizarre, seeing that all browsers were sending the very same data string in an AJAX request but getting different results, depending on the browser (Firefox being different.)

I tried this, but it didn't work:

req.setRequestHeader ("encoding", "utf-8");

Then I just did what you said Firefox does and one coding solution works in all browsers.

req.setRequestHeader("Content-type", "application/x-www-form-urlencoded;charset=utf-8");

I've tested on Chrome, MSIE, Firefox, Safari, Opera and Opera Next. Works every time!