How to get rid of non-ascii characters in ruby

Question

I have a Ruby CGI (not rails) that picks photos and captions from a web form. My users are very keen on using smart quotes and ligatures, they are pasting from other sources. My web app does not deal well with these non-ASCII characters, is there a quick Ruby string manipulation routine that can get rid of non-ASCII chars?

Solution 1

Use String#encode

The official way to convert between string encodings as of Ruby 1.9 is to use String#encode.

To simply remove non-ASCII characters, you could do this:

some_ascii   = "abc"
some_unicode = "áëëçüñżλφθΩ𠜎😸"
more_ascii   = "123ABC"
invalid_byte = "\255"

non_ascii_string = [some_ascii, some_unicode, more_ascii, invalid_byte].join

# See String#encode documentation
encoding_options = {
  :invalid           => :replace,  # Replace invalid byte sequences
  :undef             => :replace,  # Replace anything not defined in ASCII
  :replace           => '',        # Use a blank for those replacements
  :universal_newline => true       # Always break lines with \n
}

ascii = non_ascii_string.encode(Encoding.find('ASCII'), encoding_options)
puts ascii.inspect
  # => "abce123ABC"

Notice that the first 5 characters in the result are "abce1" - the "á" was discarded, one "ë" was discarded, but another "ë" appears to have been converted to "e".

The reason for this is that there are sometimes multiple ways to express the same written character in Unicode. The "á" is a single Unicode codepoint. The first "ë" is, too. When Ruby sees these during this conversion, it discards them.

But the second "ë" is two codepoints: a plain "e", just like you'd find in an ASCII string, followed by a "combining diacritical mark" (this one), which means "put an umlaut on the previous character". In the Unicode string, these are interpreted as a single "grapheme", or visible character. When converting this, Ruby keeps the plain ASCII "e" and discards the combining mark.

If you decide you'd like to provide some specific replacement values, you could do this:

REPLACEMENTS = { 
  'á' => "a",
  'ë' => 'e',
}

encoding_options = {
  :invalid   => :replace,     # Replace invalid byte sequences
  :replace => "",             # Use a blank for those replacements
  :universal_newline => true, # Always break lines with \n
  # For any character that isn't defined in ASCII, run this
  # code to find out how to replace it
  :fallback => lambda { |char|
    # If no replacement is specified, use an empty string
    REPLACEMENTS.fetch(char, "")
  },
}

ascii = non_ascii_string.encode(Encoding.find('ASCII'), encoding_options)
puts ascii.inspect
  #=> "abcaee123ABC"

Update

Some have reported issues with the :universal_newline option. I have seen this intermittently, but haven't been able to track down the cause.

When it happens, I see Encoding::ConverterNotFoundError: code converter not found (universal_newline). However, after some RVM updates, I've just run the script above under the following Ruby versions without problems:

• ruby-1.9.2-p290
• ruby-1.9.3-p125
• ruby-1.9.3-p194
• ruby-1.9.3-p362
• ruby-2.0.0-preview2
• ruby-head (as of 12-31-2012)

Given this, it doesn't appear to be a deprecated feature or even a bug in Ruby. If anyone knows the cause, please comment.

Solution 2

class String
 def remove_non_ascii(replacement="") 
   self.gsub(/[\u0080-\u00ff]/, replacement)
 end
end

Solution 3

Here's my suggestion using Iconv.

class String
  def remove_non_ascii
    require 'iconv'
    Iconv.conv('ASCII//IGNORE', 'UTF8', self)
  end
end

Solution 4

If you have active support you can use I18n.transliterate

I18n.transliterate("áëëçüñżλφθΩ𠜎")
"aee?cunz?????"

Or if you don't want the question marks...

I18n.transliterate("áëëçüñżλφθΩ𠜎", replacement: "")
"aeecunz"

Note that this doesn't remove invalid byte sequences it just replaces non ascii characters. For my use case this was what I wanted though and was simple.

Solution 5

With a bit of help from @masakielastic I have solved this problem for my personal purposes using the #chars method.

The trick is to break down each character into its own separate block so that ruby can fail.

Ruby needs to fail when it confronts binary code etc. If you don't allow ruby to go ahead and fail its a tough road when it comes to this stuff. So I use the String#chars method to break the given string into an array of characters. Then I pass that code into a sanitizing method that allows the code to have "microfailures" (my coinage) within the string.

So, given a "dirty" string, lets say you used File#read on a picture. (my case)

dirty = File.open(filepath).read    
clean_chars = dirty.chars.select do |c|
  begin
    num_or_letter?(c)
  rescue ArgumentError
    next
  end
end
clean = clean_chars.join("")

def num_or_letter?(char)
  if char =~ /[a-zA-Z0-9]/
    true
  elsif char =~ Regexp.union(" ", ".", "?", "-", "+", "/", ",", "(", ")")
    true
  end
end

View more solutions

Comments

[Admin]

Yes, I found that but it does not deal with unicode double byte chars right? Well, I will test this one, thanks for the help!

[jshea]

This is the simplest way to create an ASCII projection dropping Unicode characters. It does not create a clean translation and injects multiple replacement chars for a single multi-byte Unicode char. It was the right tool for my job, though.

[jshea]

This looks like the legitimate way to convert from Unicode to Ascii.

[e3matheus]

In ruby 1.9, I get an exception of "invalid multibyte escape". To fix it, instead of \x80-\xff, I used \u0080-\u00ff

[klochner]

. . . but, you need to remove the universal_newline option in ruby build p194 (1.9.3-p194).

[kellzerIrl]

I'm seeing the code converter not found (universal_newline) for ruby-1.9.3-p429

[Dex]

Changing the symbol :universal_newline to :UNIVERSAL_NEWLINE_DECORATOR fixes the problem for me.

[FastSolutions]

This helped me a lot, this was the only thing that was working for me! Thanks Nathan!

[dwn]

Thank you, finally! (but for me it only worked after negation /[^\u0080-\u00ff]/)

[Joe Alamo]

On Ruby 2.5.0 I was experiencing a Encoding::UnknownConversionError when trying to strip out Unicode characters from text. I fixed this by adding :undef => :replace, to the Encoding options hash

[Janosch]

This is a really bad solution. There are 0x10FF7F non-ASCII chars. This will work for 0.01% of these, and will not cover those mentioned by the OP. If you want to use gsub, the Regexp should be /[^\x00-\x7F]/.

[Rabin Poudyal]

Bad solution removed a space in some cases too

[danielricecodes]

Update for Ruby 2.7. Add double splat to fix the last argument deprecation warning. encode(Encoding.find('ASCII'), **encoding_options).

[jpw]

super-helpful newer addition to this thread