String#encode not fixing "invalid byte sequence in UTF-8" error
Solution 1
It would seem that ruby thinks that the string encoding is already utf8, so when you do
line.encode!('UTF-8', :undef => :replace, :invalid => :replace, :replace => "")
it doesn't actually do anything because the destination encoding is the same as the current encoding (at least that's my interpretation of the code in transcode.c
)
The real question here is whether your starting data is valid in some encoding that isn't utf-8 or whether this is data that is supposed to be utf-8 but has a few warts in it that you want to discard.
In the first case, the correct thing to do is tell ruby what this encoding is. You can do this when you open the file
File.open('somefile', 'r:iso-8859-1')
will open the file, interpreting its contents as iso-8859-1
You can even get ruby to transcode for you
File.open('somefile', 'r:iso-8859-1:utf-8')
will open the file as iso-8859-1, but when you read data from it the bytes will be converted to utf-8 for you.
You can also call force_encoding
to tell ruby what a string's encoding is (this doesn't modify the bytes at all, it just tells ruby how to interpret them).
In the second case, where you just want to dump whatever nasty stuff has got into your utf-8, you can't just call encode!
as you have because that's a no-op. In ruby 2.1 and higher, you can use String#scrub, in previous versions you can do this
line.encode!('UTF-16', :undef => :replace, :invalid => :replace, :replace => "")
line.encode!('UTF-8')
We first convert to utf-16. Since this is a different encoding, ruby will actually replace our invalid sequences. We can then convert back to utf-8. This won't lose us any extra data because utf-8 and utf-16 are just two different ways of encoding the same underlying character set.
Solution 2
Maybe you are running this code in IRB. I have had a lot of encoding issues with IRB. In this case, try saving this code as a .rb
file and run the code from the command line.
joshm1
Updated on June 05, 2022Comments
-
joshm1 almost 2 years
I know there are multiple similar questions about this error, and I've tried many of them without luck. The problem I'm having involves the byte
\xA1
and is throwingArgumentError: invalid byte sequence in UTF-8
I've tried the following with no success:
"\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").sub('', '') "\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").force_encoding('UTF-8').sub('', '') "\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").encode('UTF-8').sub('', '')
Each line throws the error for me. What am I doing wrong?
UPDATE:
The above lines fail only in IRB. However, I modified my application to encode lines of a CVS file using the same String#encode method and arguments, and I get the same error when reading the line from a file (note: it works if you perform the operations on the same string w/o using IO).
bad_line = "col1\tcol2\tbad\xa1" bad_line.sub('', '') # does NOT fail puts bad_line # => col1 col2 bad? tmp = Tempfile.new 'foo' # write the line to a file to emulate real problem tmp.puts bad_line tmp.close tmp2 = Tempfile.new 'bar' begin IO.foreach tmp.path do |line| line.encode!('UTF-8', :undef => :replace, :invalid => :replace, :replace => "") line.sub('', '') # fail: invalid byte sequence in UTF-8 tmp2.puts line end tmp2.close # this would fail if the above error didn't halt execution CSV.foreach(tmp2.path) do |row| puts row.inspect # fail: invalid byte sequence in UTF-8 end ensure tmp.unlink tmp2.close tmp2.unlink end
-
joshm1 almost 12 yearsYes, you're right. I was trying to resolve this in IRB after I found in error in a real application (parsing a CVS file with CVS#read). I'll look into encoding the file to UTF-8 before reading it.
-
Patrick Oscity almost 12 yearsGlad to hear. If that solved your issue, please consider accepting my answer.
-
joshm1 almost 12 yearsIt seems that the problem exists in files when a line with that byte is read from a file (and not just a hard-coded string). I modified my original post with a better example.
-
joshm1 almost 12 yearsThanks. Encoding it to UTF-16 then back to UTF-8 did what I needed. The encoding of the input file is not well defined by the source, so I can't use the first option.
-
JBoy over 9 yearsThis is just fantastic, thank you very much @Frederick Cheung
-
JBoy over 9 yearsHi, thx, is this a problem happening only in lower version than 2.1? if it is i will provide an immediate update
-
Frederick Cheung over 9 yearsNo, just noting that ruby 2.1 includes an extra method for dealing with this