Is there a way to remove the BOM from a UTF-8 encoded file?

Question

Is there a way to remove the BOM from a UTF-8 encoded file?

I know that all of my JSON files are encoded in UTF-8, but the data entry person who edited the JSON files saved it as UTF-8 with the BOM.

When I run my Ruby scripts to parse the JSON, it is failing with an error. I don't want to manually open 58+ JSON files and convert to UTF-8 without the BOM.

Solution 1

So, the solution was to do a search and replace on the BOM via gsub! I forced the encoding of the string to UTF-8 and also forced the regex pattern to be encoded in UTF-8.

I was able to derive a solution by looking at http://self.d-struct.org/195/howto-remove-byte-order-mark-with-ruby-and-iconv and http://blog.grayproductions.net/articles/ruby_19s_string

def read_json_file(file_name, index)
  content = ''
  file = File.open("#{file_name}\\game.json", "r") 
  content = file.read.force_encoding("UTF-8")

  content.gsub!("\xEF\xBB\xBF".force_encoding("UTF-8"), '')

  json = JSON.parse(content)

  print json
end

Solution 2

With ruby >= 1.9.2 you can use the mode r:bom|utf-8

This should work (I haven't test it in combination with json):

json = nil #define the variable outside the block to keep the data
File.open('file.txt', "r:bom|utf-8"){|file|
  json = JSON.parse(file.read)
}

It doesn't matter, if the BOM is available in the file or not.

---

Andrew remarked, that File#rewind can't be used with BOM.

If you need a rewind-function you must remember the position and replace rewind with pos=:

#Prepare test file
File.open('file.txt', "w:utf-8"){|f|
  f << "\xEF\xBB\xBF" #add BOM
  f << 'some content'
}

#Read file and skip BOM if available
File.open('file.txt', "r:bom|utf-8"){|f|
  pos =f.pos
  p content = f.read  #read and write file content
  f.pos = pos   #f.rewind  goes to pos 0
  p content = f.read  #(re)read and write file content
}

Solution 3

You can also specify encoding with the File.read and CSV.read methods, but you don't specify the read mode.

File.read(path, :encoding => 'bom|utf-8')
CSV.read(path, :encoding => 'bom|utf-8')

Solution 4

the "bom|UTF-8" encoding works well if you only read the file once, but fails if you ever call File#rewind, as I was doing in my code. To address this, I did the following:

def ignore_bom
  @file.ungetc if @file.pos==0 && @file.getc != "\xEF\xBB\xBF".force_encoding("UTF-8")
end

which seems to work well. Not sure if there are other similar type characters to look out for, but they could easily be built into this method that can be called any time you rewind or open.

Solution 5

Server side cleanup of utf-8 bom bytes that worked for me:

csv_text.gsub!("\xEF\xBB\xBF".force_encoding(Encoding::BINARY), '')

View more solutions

Comments

[Markus Strauss]

one-liner for ruby (takes file paths on STDIN): ruby -ne 'File.open($_.strip, "r:bom|utf-8") {|f| content=f.read(f.size); f.close; File.open($_.strip, "w").write(content) }

[Jan]

I would recommend sub!(/^\xEF\xBB\xBF/, '') as the BOM occures only once at the beginning of the string.

[Farley Knight]

I would actually suggest String#delete! over String#gsub!(..., '')

[Andrew Schwartz]

So this failed for me because I used File#rewind. At least in ruby 1.9.2, using "bom|UTF-8" simply advances the file pointer when you open it, and this gets forgotten when you rewind. Seems to me like a poor implementation of open given this encoding parameter.

[knut]

@AndrewSchwartz You are right. I extended my answer with a work around for your problem.

[Ryan]

@Jan your suggestion is good unless your setup is defaulting to UTF-8 or at least that's what I believe to be my problem because your command does not work on mine. Just doing content.sub!(/^\xEF\xBB\xBF/, '') gives an "invalid UTF-8 byte sequence" error.

[Bo Jeanes]

@Ryan I think @Jan simply meant using sub instead of gsub, but didn't necessarily mean to drop the force_encoding.