XML Signature: How to calculate the digest value?

Question

I have an XML like this

<?xml version="1.0" encoding="utf-8"?>
<foo>
  <bar>
    <value>A</value>
  </bar>
  <bar>
    <value>B</value>
  </bar>
  <baz>
    <value>C</value>
  </baz><Signature xmlns="http://www.w3.org/2000/09/xmldsig#"><SignedInfo><CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" /><SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" /><Reference URI=""><Transforms><Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" /><Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" /></Transforms><DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" /><DigestValue>WqpRWHxXA0YgH+p3Sxy6hRo1XIk=</DigestValue></Reference></SignedInfo><SignatureValue>EoRk/GhR4UA4D+8AzGPPkeim1dZrlSy88eF73n/T9Lpeq9IxoGRHNUA8FEwuDNJuz3IugC0n2RHQQpQajiYvhlY3XG+z742pgsdMfFE4Pddk4gF1T8CVS1rsF7bjX+FKT/c8B2/C8FNgmfkxDlB/ochtbRvuAGPQGtgJ3h/wjSg=</SignatureValue><KeyInfo><X509Data><X509Certificate>MIIB8zCCAVygAwIBAgIQgfzbrIjhLL9FobStI2ub3zANBgkqhkiG9w0BAQQFADATMREwDwYDVQQDEwhUZXN0ZUFjbjAeFw0wMDAxMDEwMDAwMDBaFw0zNjAxMDEwMDAwMDBaMBMxETAPBgNVBAMTCFRlc3RlQWNuMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDO+yAZ8/qJbhSVH/+2wMmzix3jM/CExb6sTgaiPwe6ylcHgF45zeQDq06OSJZCSns34em/ULINZddDf8z0b9uk/2sOGr1pYqsunLLBvw2FkvWJQDkhx2SzCm8v4xGX2kyXNbjiY/K56oPOMjpayKoAFnnvk7p2iFAxNZK/6lpZ7wIDAQABo0gwRjBEBgNVHQEEPTA7gBCOOHcajwnATYZ0t6w7LVU0oRUwEzERMA8GA1UEAxMIVGVzdGVBY26CEIH826yI4Sy/RaG0rSNrm98wDQYJKoZIhvcNAQEEBQADgYEABL9Qhi6f1Z+/t8oKXBQFx3UUsNF9N2o4k6q1c3CKZYqx2E/in+nARIYRdh5kbeLfomi6GIyVFeXExp8crob3MAzOQMvXf9+ByuezimMPIHDvv0u3kmmeITXfoZrHCDxLoWWlESN1owBfKPqe7JKAuu9ORDC0pUiUfCHWxCoqNos=</X509Certificate></X509Data></KeyInfo></Signature>
</foo>

How the digest value (WqpRWHxXA0YgH+p3Sxy6hRo1XIk=) in the reference is created? I mean how can I compute this value manually?

Solution 1

I came across this question when attempting to find out the exact same thing. I later worked out how to do it, so figured I'd post the answer here.

The things that need to happen are:

• canonicalization
• create digest value, typically SHA1 (but could be SHA256 amongst others)
• base64 encode it

The canonicalization part was fairly simple, as the Java libraries did that for me. What I struggled with was the next bit, the creating of the digest, because I made a fatal error in that the SHA1 digest I generated was the SHA1 in HEX form. SHA1 is 160 bits, so 20 bytes, but if you output these 160 bits in HEX, you get 40 characters. If you then base64 encode that, you get totally the wrong value compared to what should be in the DigestValue.

Instead, you should generate the SHA1 digest and base64 encode the 20 byte output. Don't try to output the 20 bytes to STDOUT as it's highly unlikely to be readable (which is why people often output the HEX equivalent, since it is readable). Instead, just base64 encode the 20 bytes and that's your DigestValue.

Solution 2

Is very simple, use openssl in the console:

openssl dgst -binary -sha1 file | openssl enc -base64

Done

Solution 3

I have encountered exactly this problem myself: I was generating an XML signature in Java & validating in .NET, and the validation always failed. In my case the cause was the 'print XML to file' function XMLWrite.m (yes, in MATLAB*) which was 'pretty printing' the XML, inserting tabs, spaces, and newlines as it saw fit. Since these are part of the document, naturally the validation failed (it failed in Java, too). Looking at your source, this may be happening to you. Use a Transformer (javax.xml.transform.*) to serialise your DOM properly without changing the content.

*You did know that MATLAB understands Java as well? You can just type Java statements into the interpreter console & they will be executed like native m-code.

Solution 4

This is a JAVA solution which requires the following jars:

• commons-logging-1.2.jar
• commons-codec-1.6.jar
• Saxon-HE-9.4.jar
• xmlsec-1.3.0.jar

This solution uses http://www.w3.org/2001/10/xml-exc-c14n# as the canonicalization algorithm, and uses SHA256 as the hashing algorithm and base64 encoding.

Note: document represents the XML document as a DOM object in JAVA.

Code sample:

        // create the transformer in order to transform the document from
        // DOM Source as a JAVA document class, into a character stream (StreamResult) of
        // type String writer, in order to be converted to a string later on
        TransformerFactory tf = new net.sf.saxon.TransformerFactoryImpl();
        Transformer transformer = tf.newTransformer();
        transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
        transformer.setOutputProperty(OutputKeys.METHOD, "xml");
        transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");

        // create the string writer and transform the document to a character stream
        StringWriter sw = new StringWriter();
        transformer.transform(new DOMSource(document), new StreamResult(sw));

        String documentAsString = sw.toString();

        // initialize the XML security object, which is necessary to run the apache canonicalization
        com.sun.org.apache.xml.internal.security.Init.init();

        // canonicalize the document to a byte array and convert it to string
        Canonicalizer canon = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
        byte canonXmlBytes[] = canon.canonicalize(documentAsString.getBytes());
        String canonXmlString = new String(canonXmlBytes);

        // get instance of the message digest based on the SHA-256 hashing algorithm
        MessageDigest digest = MessageDigest.getInstance("SHA-256");

        // call the digest method passing the byte stream on the text, this directly updates the message
        // being digested and perform the hashing
        byte[] hash = digest.digest(canonXmlString.getBytes(StandardCharsets.UTF_8));

        // encode the endresult byte hash
        byte[] encodedBytes = Base64.encodeBase64(hash);

        return new String(encodedBytes);

View more solutions

Comments

[Oscar Jara]

I understand you solved the "Digest verification failed for Reference" error? would it be possible to get a detailed explanation of how you in fact did it? I'm kinda having that issue with a signed soap message using Metro and a .Net WCF application wouldn't recognize the Digest Value.

[ewernli]

This is already addressed by XML Signature which rely on XML Canonicalization. XML is normalized first and the syntactic differences (white spaces, namespaces, etc.) will be handled correctly.

[Max]

True: but it is the DOM, not its seralization, which has been canonicalized. The extra whitespace characters were inserted during serialization (to file) AFTER the digest was computed, & they were still there when the validating code created its DOM, canonicalized it and recomputed the digest. Whitespace is significant in parsed character data nodes.

[Filippo Mazza]

I thought the same, but I get different value also like that. Did you digested directly Canonicalization Bytes or the digested String bytes?

[Vbakke]

OP did not specify 'manually' as 'programmatically'. This is anyway a good way of checking.

[Dustin Sun]

SHA1 digest of what?

[Harry]

@lonelyloner: SHA1 of the canonicalized input xml after the Signature Node was removed. In my case the xml was created by .NET, so i also had to remove \r \n and whitespaces between tags before canonicalizing and creating SHA1

[dev4life]

Would you know how to get the signaturevalue as well?

[InspiredBy]

Found this very useful. Also looking hot to generate signturevalue , like @dev4life above though.