Removing BOM characters using Java

Question

What needs to happen to a string using Java to be an equivalent of vis

:set nobomb

Assume that BOM comes from the file I am reading.

Solution 1

Java does not handle BOM properly. In fact Java handles a BOM like every other char.

Found this:

http://www.rgagnon.com/javadetails/java-handle-utf8-file-with-bom.html

public static final String UTF8_BOM = "\uFEFF";

private static String removeUTF8BOM(String s) {
    if (s.startsWith(UTF8_BOM)) {
        s = s.substring(1);
    }
    return s;
}

May be I would use apache IO instead:

http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/BOMInputStream.html

Solution 2

For UTF-8 the BOM is: 0xEF, 0xBB, 0xBF

Comments

[fge]

Strings in Java do not have BOM... Unless you read from a source which has one

[James Raitsev]

This is precisely what happens. I am reading the file that happens to have this mark

[fge]

Do you at least know what encoding is used (UTF-8, UTF-16 LE/BE)?

[Durandal]

If you have the option just open the file with Notepad++ or SublimeText and resave it without a BOM. Otherwise you'd need to know the encoding type to do it programatically

[Walter Tross]

UTF8_BOM is a wrong name. There is nothing in the BOM that links it to UTF-8. On the contrary, UTF-8 does NOT need the BOM, while UTF-16 MAY (and Microsoft has the bad habit of writing UTF-16 files with a BOM, which often get converted to UTF-8 with BOM by bad tools).

[Krzysztof Tomaszewski]

UTF-8 BOM consists of 3 bytes, not 2.