ruby `encode': "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
Solution 1
It seems you should use another encoding for the object. You should set the proper codepage to the variable @tree
, for instance, using iso-8859-1 instead of ascii-8bit by using @tree.force_encoding('ISO-8859-1')
. Because ASCII-8BIT
is used just for binary files.
To find the current external encoding for ruby, issue:
Encoding.default_external
If sudo solves the problem, the problem was in default codepage (encoding), so to resolve it you have to set the proper default codepage (encoding), by either:
-
In ruby to change encoding to utf-8 or another proper one, do as follows:
Encoding.default_external = Encoding::UTF_8
-
In bash,
grep
current valid set up:$ sudo env|grep UTF-8 LC_ALL=ru_RU.UTF-8 LANG=ru_RU.UTF-8
Then set them in
.bashrc
properly, in a similar way, but not exactly withru_RU
language, such as the following:export LC_ALL=ru_RU.UTF-8 export LANG=ru_RU.UTF-8
Solution 2
File.open(yml_file, 'w') should be change to File.open(yml_file, 'wb')
Solution 3
I had the same problems when saving to the database. I'll offer one thing that I use (perhaps, this will help someone).
if you know that sometimes your text has strange characters, then before saving you can encode your text in some other format, and then decode the text again after it is returned from the database.
example:
string = "Œuf"
before save we encode string
text_to_save = CGI.escape(string)
(character "Œ" encoded in "%C5%92" and other characters remained the same)
=> "%C5%92uf"
load from database and decode
CGI.unescape("%C5%92uf")
=> "Œuf"
Solution 4
I just suffered through a number of hours trying to fix a similar problem. I'd checked my locales, database encoding, everything I could think of and was still getting ASCII-8BIT encoded data from the database.
Well, it turns out that if you store text in a binary field, it will automatically be returned as ASCII-8BIT encoded text, which makes sense, however this can (obviously) cause problems in your application.
It can be fixed by changing the column encoding back to :text
in your migrations.
Admin
Updated on February 13, 2020Comments
-
Admin about 4 years
Hannibal episodes in tvdb have weird characters in them.
For example:
Œuf
So ruby spits out:
./manifesto.rb:19:in `encode': "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError) from ./manifesto.rb:19:in `to_json' from ./manifesto.rb:19:in `<main>'
Line 19 is:
puts @tree.to_json
Is there a way to deal with these non utf characters? I'd rather not replace them, but convert them? Or ignore them? I don't know, any help appreciated.
Weird part is that script works fine via cron. Manually running it creates error.
-
Martin Zabel about 8 yearsThe posted code in the question does not contain a line
File.open(...)
. So what do you mean? -
amoebe about 8 yearsOkay, so this answer actually helped me. If you write YAML to a file like this:
File.open('/path/to/file.yml', 'wb') { |f| YAML.dump(data, f) }
, you will get the error in the title if you leave out theb
. -
Малъ Скрылевъ about 6 yearsthe question say nothing about the file open
-
alexventuraio about 5 yearsActually it helped me because I was having this error
Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8
when doingfile.puts data
withfile = File.open("response.txt", "w")
but after adding theb
param it worked fine. But it would come in handy if the explanation was included here @unplugandplay