remove non-UTF-8 characters from xml with declared encoding=utf-8 - Java

Question

I have to handle this scenario in Java:

I'm getting a request in XML form from a client with declared encoding=utf-8. Unfortunately it may contain not utf-8 characters and there is a requirement to remove these characters from the xml on my side (legacy).

Let's consider an example where this invalid XML contains £ (pound).

1) I get xml as java String with £ in it (I don't have access to interface right now, but I probably get xml as a java String). Can I use replaceAll(£, "") to get rid of this character? Any potential issues?

2) I get xml as an array of bytes - how to handle this operation safely in that case?

Solution 1

1) I get xml as java String with £ in it (I don't have access to interface right now, but I probably get xml as a java String). Can I use replaceAll(£, "") to get rid of this character?

I am assuming that you rather mean that you want to get rid of non-ASCII characters, because you're talking about a "legacy" side. You can get rid of anything outside the printable ASCII range using the following regex:

string = string.replaceAll("[^\\x20-\\x7e]", "");

2) I get xml as an array of bytes - how to handle this operation safely in that case?

You need to wrap the byte[] in an ByteArrayInputStream, so that you can read them in an UTF-8 encoded character stream using InputStreamReader wherein you specify the encoding and then use a BufferedReader to read it line by line.

E.g.

BufferedReader reader = null;
try {
    reader = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(bytes), "UTF-8"));
    for (String line; (line = reader.readLine()) != null;) {
        line = line.replaceAll("[^\\x20-\\x7e]", "");
        // ...
    }
    // ...

Solution 2

UTF-8 is an encoding; Unicode is a character set. But the GBP symbol is most definitely in the Unicode character set and therefore most certainly representable in UTF-8.

If you do in fact mean UTF-8, and you are actually trying to remove byte sequences that are not the valid encoding of a character in UTF-8, then...

CharsetDecoder utf8Decoder = Charset.forName("UTF-8").newDecoder();
utf8Decoder.onMalformedInput(CodingErrorAction.IGNORE);
utf8Decoder.onUnmappableCharacter(CodingErrorAction.IGNORE);
ByteBuffer bytes = ...;
CharBuffer parsed = utf8Decoder.decode(bytes);
...

Solution 3

"test text".replaceAll("[^\\u0000-\\uFFFF]", "");

This code removes all 4-byte utf8 chars from string.This can be needed for some purposes while doing Mysql innodb varchar entry

Solution 4

I faced the same problem while reading files from a local directory and tried this:

BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(filePath), "UTF-8"));
DocumentBuilder db = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document xmlDom = db.parse(new InputSource(in));

You might have to use your network input stream instead of FileInputStream.

-- Kapil

Solution 5

Note that the first step should be that you ask the creator of the XML (which is most likely a home grown "just print data" XML generator) to ensure that their XML is correct before sending to you. The simplest possible test if they use Windows is to ask them to view it in Internet Explorer and see the parsing error at the first offending character.

While they fix that, you can simply write a small program that change the header part to declare that the encoding is ISO-8859-1 instead:

<?xml version="1.0" encoding="iso-8859-1" ?>

and leave the rest untouched.

View more solutions

Comments

[HitchHiker]

Thanx a lott!! my problem was different but this ended my misery of 2 days :)

[Digao]

this answer is 8 years old and still valid ! Thanks a lot !