Encoding::UndefinedConversionError

ruby encoding sinatra sequel

46,887

Solution 1

menu.to_s.encode('UTF-8', invalid: :replace, undef: :replace, replace: '?')

This worked perfectly, I had to replace some extra characters but there are no more errors.

Solution 2

What do you expect for "\xC2"? Probably a Â

With ASCII-8BIT you have binary data, and ruby cant decide, what should be.

You must first set the encoding with force_encoding.

You may try the following code:

Encoding.list.each{|enc|
  begin
    print "%-10s\t" % [enc]
    print "\t\xC2".force_encoding(enc)
    print "\t\xC2".force_encoding(enc).encode('utf-8')
  rescue => err
    print "\t#{err}"
  end
  print "\n"
}

The result are the possible values in different encodings for your "\xC2".

The result may depend on your Output format, but I think you can make a good guess, which encoding you have.

When you defined the encoding you need (probably cp1251) you can

menu.force_encoding('cp1252').to_json

Solution 3

If you don't care about losing the strange characters, you can blow them away:

str.force_encoding("ASCII-8BIT").encode('UTF-8', undef: :replace, replace: '')

Solution 4

Your auto-accepted solution doesn't work, there are effectively no errors, but it is NOT JSON.

I solved the problem using the oj gem, it now works find. It is also faster than the standard JSON library.

Writting :

   menu_json = Oj.dump menu

Reading :

   menu2 = Oj.load menu_json

https://github.com/ohler55/oj for more details. I hope it will help.

Solution 5

:fallback option can be useful if you know what chars you want to replace

"Text 🙂".encode("ASCII", "UTF-8", fallback: {"🙂" => ":)"})
#=> hello :)

From docs:

Sets the replacement string by the given object for undefined character. The object should be a Hash, a Proc, a Method, or an object which has [] method. Its key is an undefined character encoded in the source encoding of current transcoder. Its value can be any encoding until it can be converted into the destination encoding of the transcoder.

View more solutions

46,887

Author by

martriay

Product at @OpenZeppelin. Previously @decentraland.

Updated on July 24, 2022

Comments

martriay almost 2 years
I keep getting an Encoding::UndefinedConversionError - "\xC2" from ASCII-8BIT to UTF-8 every time I try to convert a hash into a JSON string. I tried with [.encode | .force_encoding](["UTF-8" | "ASCII-8BIT" ]), chaining .encode with .force_encoding, backwards, switching parameters but nothing seemed to work so I caught the error like this:
```
begin
  menu.to_json
rescue Encoding::UndefinedConversionError
  puts $!.error_char.dump
  p $!.error_char.encoding
end
```
Where menu is a sequel's dataset.to_hash with content from a MySQL DB, utf8_general_ci encoding and returned this:

"\xC2"

<#Encoding:ASCII-8BIT>

The encoding never changes, no matter what .encode/.force_encoding I use. I've even tried to replace the string .gsub!(/\\\xC2/) without luck.

Any ideas?
martriay over 11 years

this is what I did: ´Encoding.list.each{|enc| begin print "%-10s\t" % [enc] print menu.to_json.force_encoding(enc) print menu.to_json.force_encoding(enc).encode('utf-8') rescue => err print "\t#{err}" end print "\n" }´ and this is what I've got for each result: ´SJIS-KDDI "\xC2" from ASCII-8BIT to UTF-8´
martriay over 11 years

Didn't worked :( Encoding::UndefinedConversionError at /menu "\xC2" from ASCII-8BIT to UTF-8
martriay over 11 years

menu.to_s.encode('UTF-8', {:invalid => :replace, :undef => :replace, :replace => '?'}) -> this worked! :D
martriay over 10 years

The problem was the error, not the JSON part. Therefore, my auto-accepted answers works. Anyway, I'll upvote you for giving an alternative solution.
gvo over 10 years

Well, I agree with you, there are no longer errors, but it's not a json string. I don't know what was your purpose, but I needed to load back my json, and I wanted a valid JSON String. Or maybe I have missed something in your proposed solution?
martriay over 10 years

This question was only about the error, I'm not saying my answer is the best choice, clearly it isn't for your purpose, but solves the problem presented: the encoding error. The JSON I mention in my question is for contextualization purposes.
Marc Clifton over 10 years

Fantastic solution - solved my problem dealing with strange types in SQL Server. Thank you!
jmoreira almost 10 years

Thanks! It works for me too, official ruby doc for future reference here
Cyberwiz over 6 years

Thanks @gvo ! Hopefully others googling for this error will find this solution...
pdobb over 5 years

This discovery loop for all encodings is brilliant. Helped me solve my own variant on this problem. Thanks!