I have UTF-8 - but still get "Invalid byte 1 of 1-byte UTF-8 sequence"

Question

I create a XML String on the fly (NOT reading from a file). Then I use Cocoon 3 to transform it via FOP to a PDF. Somewhere in the middle Xerces runs. When I use the hardcoded stuff everything works. As soon as I put a german Umlaut into the database and enrich my xml with that data I get:

Caused by: org.apache.cocoon.pipeline.ProcessingException: Can't parse the XML string.
at org.apache.cocoon.sax.component.XMLGenerator$StringGenerator.execute(XMLGenerator.java:326)
at org.apache.cocoon.sax.component.XMLGenerator.execute(XMLGenerator.java:104)
at org.apache.cocoon.pipeline.AbstractPipeline.invokeStarter(AbstractPipeline.java:146)
at org.apache.cocoon.pipeline.AbstractPipeline.execute(AbstractPipeline.java:76)
at de.grobmeier.tab.webapp.modules.documents.InvoicePipeline.generateInvoice(InvoicePipeline.java:74)
... 87 more

Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
    at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:684)
    at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:554)

I have then debugged my app and found out, my "Ä" (which comes frome the database) has the byte value of 196, which is C4 in hex. This is what I have expected according to this: http://www.utf8-zeichentabelle.de/

I do not know why my code fails.

I have then tried to add a BOM manually, like that:

byte[] bom = new byte[3];
bom[0] = (byte) 0xEF;
bom[1] = (byte) 0xBB;
bom[2] = (byte) 0xBF;
String myString = new String(bom) + inputString;

I know this is not exactly good, but I tried it - of course it failed. I have tried to add a xml header in front:

<?xml version="1.0" encoding="UTF-8"?>

Which failed too. Then I combined it. Failed.

After all I tried something like that:

xmlInput = new String(xmlInput.getBytes("UTF8"), "UTF8");

Which is doing nothing in fact, because it is already UTF-8. Still it fails.

So... any ideas what I am doing wrong and what Xerces is expecting from me?

Thanks Christian

Solution 1

If your database contains only a single byte (with value 0xC4) then you aren't using UTF-8 encoding.

The character "LATIN CAPITAL LETTER A WITH DIAERESIS" has a code-point value U+00C4, but UTF-8 can't encode that in a single byte. If you check the third column "UTF-8 (hex.)" on UTF8-zeichentabelle.de you'll see that UTF-8 encodes that as 0xC3 84 (two bytes).

Please read Joel's article "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" for more info.

---

EDIT: Christian found the answer himself; turned out it was a problem in the Cocoon 3 SAX component (I guess it's the alpha 3 version). It turns out that if you pass an XML as a String into the XMLGenerator class, something will go wrong during SAX parsing causing this mess.

I looked up the code to find the actual problem in Cocoon-stax:

if (XMLGenerator.this.logger.isDebugEnabled()) {
    XMLGenerator.this.logger.debug("Using a string to produce SAX events.");
}
XMLUtils.toSax(new ByteArrayInputStream(this.xmlString.getBytes()), XMLGenerator.this.getSAXConsumer();

As you can see, the call getBytes() will create a Byte array with the JRE's default encoding which will then fail to parse. This is because the XML declares itself to be UTF-8 whereas the data is now in bytes again, and likely using your Windows codepage.

As a workaround, one can use the following:

new org.apache.cocoon.sax.component.XMLGenerator(xmlInput.getBytes("UTF-8"),
       "UTF-8");

This will trigger the right internal actions (as Christian found out by experimenting with the API).

I've opened an issue in Apache's bug tracker.

EDIT 2: The issue is fixed and will be included in an upcoming release.

Solution 2

The C4 you see on that page refers to the unicode code point, U+00C4. The byte sequence used to represent such a code point in UTF-8 is NOT "\xC4". What you want is what's in the UTF-8 (hex.) column, namely "\xC3\x84".

Therefore, your data is not in UTF-8.

You can read about how data is encoded in UTF-8 here.

Comments

[Pops]

Way to go above and beyond!