Module request how to properly retrieve accented characters? � � �

Question

I'm using: Module: Request -- Simplified HTTP request method to scrape a webpage with accented characters á é ó ú ê ã etc.

I've already tried encoding: utf-8 with no success. I'm still getting this ��� characters in the result.

request.get({
    uri: url,
    encoding: 'utf-8'
    // ...

Is there any configuration to fix it?

I don't know if it is an issue, but I filled one for this module. No answers yet. :/

Solution 1

Since binary is deprecated it seems like a better idea to use iconv and correctly handle the decoding:

var request = require("request"), iconv  = require('iconv-lite');
var requestOptions  = { encoding: null, method: "GET", uri: "http://something.com"};

request(requestOptions, function(error, response, body) {
    var utf8String = iconv.decode(new Buffer(body), "ISO-8859-1");
    console.log(utf8String);
});

The important part is to set the encoding on the HTTP request to be null encoding: null.

Solution 2

Specify the encoding as utf8 not utf-8. Here are a list of possible encodings for a buffer from the Node.js documentation.

• ascii - for 7 bit ASCII data only. This encoding method is very fast, and will strip the high bit if set.
• utf8 - Unicode characters. Many web pages and other document formats use UTF-8.
• base64 - Base64 string encoding.
• 'binary - A way of encoding raw binary data into strings by using only the first 8 bits of each character. This encoding method is depreciated and should be avoided in favor of Buffer objects where possible. This encoding will be removed in future versions of Node.

Comments

[Pablo Cantero]

utf-8 works as utf8. The page that I'm scrapping is iso-8859-1. The only encoding that worked for me was "binary"... too strange... We discussed about it here github.com/mikeal/request/issues/118

[newman]

This works great, but I have two questions. 1. why do you need to create new Buffer for body? I tried to use body directly and didn't see any difference. What do I miss? 2. If the web page says charset=utf-8, why do I have to use iconv-lite to convert it to ISO-8859-1?

[Marcos Mendes]

binary works for me. I`m using request module, i passed encoding: 'binary' in the options. Thank you