Force strings to UTF-8 from any encoding
Solution 1
Ruby 1.9
"Forcing" an encoding is easy, however it won't convert the characters just change the encoding:
str = str.force_encoding('UTF-8')
str.encoding.name # => 'UTF-8'
If you want to perform a conversion, use encode
:
begin
str.encode("UTF-8")
rescue Encoding::UndefinedConversionError
# ...
end
I would definitely read the following post for more information:
http://graysoftinc.com/character-encodings/ruby-19s-string
Solution 2
This will ensure you have the correct encoding and won't error out because it replaces any invalid or undefined character with a blank string.
This will ensure no matter what, that you have a valid UTF-8 string
str.encode(Encoding.find('UTF-8'), {invalid: :replace, undef: :replace, replace: ''})
Solution 3
Only this solution worked for me:
string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
Note the binary argument.
Solution 4
require 'iconv'
i = Iconv.new('UTF-8','LATIN1')
a_with_hat = i.iconv("\xc2")
Summary: the iconv gem does all the work of converting encodings. Make sure it's installed with:
gem install iconv
Now, you need to know what encoding your string is currently in as Ruby 1.8 treats Strings as an array of bytes (with no intrinsic encoding.) For example, say your string was in latin1 and you wanted to convert it to utf-8
require 'iconv'
string_in_utf8_encoding = Iconv.conv("UTF8", "LATIN1", string_in_latin1_encoding)
Related videos on Youtube
Hayk Saakian
Updated on July 09, 2022Comments
-
Hayk Saakian almost 2 years
In my rails app I'm working with RSS feeds from all around the world, and some feeds have links that are not in UTF-8. The original feed links are out of my control, and in order to use them in other parts of the app, they need to be in UTF-8.
How can I detect encoding and convert to UTF-8?
-
Gromski over 11 yearsTo detect an encoding, you need to parse the accompanying meta information of the documents, i.e. HTTP headers or
<meta>
tags.
-
-
Hayk Saakian over 11 yearsThanks for the answer, but in my case the source data is inconsistent and I don't really have a reliable way to preempt encodings
-
Hackeron almost 11 yearsDoesn't work: whois = whois.force_encoding("UTF-8") \n whois.encoding.name => "UTF-8" \n whois.scan(/role:\s+(.+)/i) -- Throws: ArgumentError: invalid byte sequence in UTF-8
-
kwarrick almost 11 yearsAs stated, force_encoding does not convert the characters and certainly cannot magically interpret invalid UTF-8 byte sequences.
-
basgys almost 11 yearsIconv should not be used anymore. (deprecated) stackoverflow.com/questions/8148762/…
-
Joseworks about 8 yearsCurrent syntax for Ruby 2.2.0 and above is:
str.force_encoding(Encoding::UTF_8)
Encoding