Getting BouncyCastle to decrypt a GPG-encrypted message

Question

How can I get BouncyCastle to decrypt a GPG-encrypted message?

I have created a GPG key pair at the CentOS 7 command line using gpg --gen-key. I chose RSA RSA as the encryption types, and I exported the keys using gpg --export-secret-key -a "User Name" > /home/username/username_private.key and gpg --armor --export 66677FC6 > /home/username/username_pubkey.asc

I am able to import username_pubkey.asc into a remote Thunderbird client of another email account and successfully send an encrypted email to [email protected]. But when my Java/BouncyCastle code running at mydomain.com tries to decrypt the GPG-encoded data, it gives the following error:

org.bouncycastle.openpgp.PGPException:  
Encrypted message contains a signed message - not literal data.

If you look at the code below, you will see this corresponds with the line in PGPUtils.decryptFile() which states else if (message instanceof PGPOnePassSignatureList) {throw new PGPException("Encrypted message contains a signed message - not literal data.");

The original code for this came from the blog entry at this link, though I made minor changes to get it to compile in Eclipse Luna with Java 7. A user of the linked blog reported the same error, and the blog author replied by saying that it does not work with GPG. So how do I fix this to make it work with GPG?

The Java decryption code starts when the GPG-encoded-file and the GPG-secret-key are passed into Tester.testDecrypt() as follows:

Tester.java contains:

public InputStream testDecrypt(String input, String output, String passphrase, String skeyfile) throws Exception {
    PGPFileProcessor p = new PGPFileProcessor();
    p.setInputFileName(input);//this is GPG-encoded data sent from another email address using Thunderbird
    p.setOutputFileName(output);
    p.setPassphrase(passphrase);
    p.setSecretKeyFileName(skeyfile);//this is the GPG-generated key
    return p.decrypt();//this line throws the error
}

PGPFileProcessor.java includes:

public InputStream decrypt() throws Exception {
    FileInputStream in = new FileInputStream(inputFileName);
    FileInputStream keyIn = new FileInputStream(secretKeyFileName);
    FileOutputStream out = new FileOutputStream(outputFileName);
    PGPUtils.decryptFile(in, out, keyIn, passphrase.toCharArray());//error thrown here
    in.close();
    out.close();
    keyIn.close();
    InputStream result = new FileInputStream(outputFileName);//I changed return type from boolean on 1/27/15
    Files.deleteIfExists(Paths.get(outputFileName));//I also added this to accommodate change of return type on 1/27/15
    return result;
}

PGPUtils.java includes:

/**
 * decrypt the passed in message stream
 */
@SuppressWarnings("unchecked")
public static void decryptFile(InputStream in, OutputStream out, InputStream keyIn, char[] passwd)
    throws Exception
{
    Security.addProvider(new BouncyCastleProvider());

    in = org.bouncycastle.openpgp.PGPUtil.getDecoderStream(in);

    //1/26/15 added Jca prefix to avoid eclipse warning, also used https://www.bouncycastle.org/docs/pgdocs1.5on/index.html
    PGPObjectFactory pgpF = new JcaPGPObjectFactory(in);
    PGPEncryptedDataList enc;

    Object o = pgpF.nextObject();
    //
    // the first object might be a PGP marker packet.
    //
    if (o instanceof  PGPEncryptedDataList) {enc = (PGPEncryptedDataList) o;}
    else {enc = (PGPEncryptedDataList) pgpF.nextObject();}

    //
    // find the secret key
    //
    Iterator<PGPPublicKeyEncryptedData> it = enc.getEncryptedDataObjects();
    PGPPrivateKey sKey = null;
    PGPPublicKeyEncryptedData pbe = null;

    while (sKey == null && it.hasNext()) {
        pbe = it.next(); 
        sKey = findPrivateKey(keyIn, pbe.getKeyID(), passwd);
    }

    if (sKey == null) {throw new IllegalArgumentException("Secret key for message not found.");}

    InputStream clear = pbe.getDataStream(new BcPublicKeyDataDecryptorFactory(sKey));

    //1/26/15 added Jca prefix to avoid eclipse warning, also used https://www.bouncycastle.org/docs/pgdocs1.5on/index.html
    PGPObjectFactory plainFact = new JcaPGPObjectFactory(clear);

    Object message = plainFact.nextObject();

    if (message instanceof  PGPCompressedData) {
        PGPCompressedData cData = (PGPCompressedData) message;
        //1/26/15 added Jca prefix to avoid eclipse warning, also used https://www.bouncycastle.org/docs/pgdocs1.5on/index.html
        PGPObjectFactory pgpFact = new JcaPGPObjectFactory(cData.getDataStream()); 
        message = pgpFact.nextObject();
    }

    if (message instanceof  PGPLiteralData) {
        PGPLiteralData ld = (PGPLiteralData) message;

        InputStream unc = ld.getInputStream();
        int ch;

        while ((ch = unc.read()) >= 0) {out.write(ch);}
    } else if (message instanceof  PGPOnePassSignatureList) {
        throw new PGPException("Encrypted message contains a signed message - not literal data.");
    } else {
        throw new PGPException("Message is not a simple encrypted file - type unknown.");
    }

    if (pbe.isIntegrityProtected()) {
        if (!pbe.verify()) {throw new PGPException("Message failed integrity check");}
    }
}

/**
 * Load a secret key ring collection from keyIn and find the private key corresponding to
 * keyID if it exists.
 *
 * @param keyIn input stream representing a key ring collection.
 * @param keyID keyID we want.
 * @param pass passphrase to decrypt secret key with.
 * @return
 * @throws IOException
 * @throws PGPException
 * @throws NoSuchProviderException
 */
public  static PGPPrivateKey findPrivateKey(InputStream keyIn, long keyID, char[] pass)
    throws IOException, PGPException, NoSuchProviderException
{
    //1/26/15 added Jca prefix to avoid eclipse warning, also used https://www.bouncycastle.org/docs/pgdocs1.5on/index.html
    PGPSecretKeyRingCollection pgpSec = new JcaPGPSecretKeyRingCollection(PGPUtil.getDecoderStream(keyIn));
    return findPrivateKey(pgpSec.getSecretKey(keyID), pass);

}

/**
 * Load a secret key and find the private key in it
 * @param pgpSecKey The secret key
 * @param pass passphrase to decrypt secret key with
 * @return
 * @throws PGPException
 */
public static PGPPrivateKey findPrivateKey(PGPSecretKey pgpSecKey, char[] pass)
    throws PGPException
{
    if (pgpSecKey == null) return null;

    PBESecretKeyDecryptor decryptor = new BcPBESecretKeyDecryptorBuilder(new BcPGPDigestCalculatorProvider()).build(pass);
    return pgpSecKey.extractPrivateKey(decryptor);
}  

The complete code of all three Java files can be found on a file sharing site by clicking on this link.

The complete stack trace for the error can be found by clicking on this link.

For reference, the GUI instructions for encryption by the remote Thunderbird sender are summarized in the following screen shot:

I have read many postings and links about this. In particular, this other SO posting looks similar, but is different. My Keys use RSA RSA, but the other posting does not.

EDIT#1

As per @DavidHook's suggestion, I have read SignedFileProcessor, and I am starting to read the much longer RFC 4880. However, I need actual working code to study in order to understand this. Most people who find this via google searches will also need working code to illustrate the examples.

For reference, the SignedFileProcessor.verifyFile() method recommended by @DavidHook is as follows. How should this be customized to fix the problems in the code above?

private static void verifyFile(InputStream in, InputStream keyIn) throws Exception {
    in = PGPUtil.getDecoderStream(in);
    PGPObjectFactory pgpFact = new PGPObjectFactory(in);
    PGPCompressedData c1 = (PGPCompressedData)pgpFact.nextObject();
    pgpFact = new PGPObjectFactory(c1.getDataStream());
    PGPOnePassSignatureList p1 = (PGPOnePassSignatureList)pgpFact.nextObject();
    PGPOnePassSignature ops = p1.get(0);
    PGPLiteralData p2 = (PGPLiteralData)pgpFact.nextObject();
    InputStream dIn = p2.getInputStream();
    int ch;
    PGPPublicKeyRingCollection  pgpRing = new PGPPublicKeyRingCollection(PGPUtil.getDecoderStream(keyIn));
    PGPPublicKey key = pgpRing.getPublicKey(ops.getKeyID());
    FileOutputStream out = new FileOutputStream(p2.getFileName());
    ops.initVerify(key, "BC");
    while ((ch = dIn.read()) >= 0){
        ops.update((byte)ch);
        out.write(ch);
    }
    out.close();
    PGPSignatureList p3 = (PGPSignatureList)pgpFact.nextObject();
    if (ops.verify(p3.get(0))){System.out.println("signature verified.");}
    else{System.out.println("signature verification failed.");}
}

EDIT#2

The SignedFileProcessor.verifyFile() method recommended by @DavidHook is almost identical to the PGPUtils.verifyFile() method in my code above, except that PGPUtils.verifyFile() makes a copy of extractContentFile and calls PGPOnePassSignature.init() instead of PGPOnePassSignature.initVerify(). This may be due to a version difference. Also, PGPUtils.verifyFile() returns a boolean, while SignedFileProcessor.verifyFile() gives SYSO for the two boolean values and returns void after the SYSO.

If I interpret @JRichardSnape's comments correctly, this means that the verifyFile() method might best be called upstream to confirm the signature of the incoming file using the sender's public key, and then, if the signature on the file is verified, using another method to decrypt the file using the recipient's private key. Is this correct? If so, how do I restructure the code to accomplish this?

Solution 1

if anyone is interested to know how to encrypt and decrypt gpg files using bouncy castle openPGP library, check the below java code:

The below are the 4 methods you going to need:

The below method will read and import your secret key from .asc file:

public static PGPSecretKey readSecretKeyFromCol(InputStream in, long keyId) throws IOException, PGPException {
in = PGPUtil.getDecoderStream(in);
PGPSecretKeyRingCollection pgpSec = new PGPSecretKeyRingCollection(in, new BcKeyFingerprintCalculator());

PGPSecretKey key = pgpSec.getSecretKey(keyId);

if (key == null) {
    throw new IllegalArgumentException("Can't find encryption key in key ring.");
}
return key;
}

The below method will read and import your public key from .asc file:

@SuppressWarnings("rawtypes")
public static PGPPublicKey readPublicKeyFromCol(InputStream in) throws IOException, PGPException {
    in = PGPUtil.getDecoderStream(in);
    PGPPublicKeyRingCollection pgpPub = new PGPPublicKeyRingCollection(in, new BcKeyFingerprintCalculator());
    PGPPublicKey key = null;
    Iterator rIt = pgpPub.getKeyRings();
    while (key == null && rIt.hasNext()) {
        PGPPublicKeyRing kRing = (PGPPublicKeyRing) rIt.next();
        Iterator kIt = kRing.getPublicKeys();
        while (key == null && kIt.hasNext()) {
            PGPPublicKey k = (PGPPublicKey) kIt.next();
            if (k.isEncryptionKey()) {
                key = k;
            }
        }
    }
    if (key == null) {
        throw new IllegalArgumentException("Can't find encryption key in key ring.");
    }
    return key;
}

The below 2 methods to decrypt and encrypt gpg files:

public void decryptFile(InputStream in, InputStream secKeyIn, InputStream pubKeyIn, char[] pass) throws IOException, PGPException, InvalidCipherTextException {
    Security.addProvider(new BouncyCastleProvider());

    PGPPublicKey pubKey = readPublicKeyFromCol(pubKeyIn);

    PGPSecretKey secKey = readSecretKeyFromCol(secKeyIn, pubKey.getKeyID());

    in = PGPUtil.getDecoderStream(in);

    JcaPGPObjectFactory pgpFact;


    PGPObjectFactory pgpF = new PGPObjectFactory(in, new BcKeyFingerprintCalculator());

    Object o = pgpF.nextObject();
    PGPEncryptedDataList encList;

    if (o instanceof PGPEncryptedDataList) {

        encList = (PGPEncryptedDataList) o;

    } else {

        encList = (PGPEncryptedDataList) pgpF.nextObject();

    }

    Iterator<PGPPublicKeyEncryptedData> itt = encList.getEncryptedDataObjects();
    PGPPrivateKey sKey = null;
    PGPPublicKeyEncryptedData encP = null;
    while (sKey == null && itt.hasNext()) {
        encP = itt.next();
        secKey = readSecretKeyFromCol(new FileInputStream("PrivateKey.asc"), encP.getKeyID());
        sKey = secKey.extractPrivateKey(new BcPBESecretKeyDecryptorBuilder(new BcPGPDigestCalculatorProvider()).build(pass));
    }
    if (sKey == null) {
        throw new IllegalArgumentException("Secret key for message not found.");
    }

    InputStream clear = encP.getDataStream(new BcPublicKeyDataDecryptorFactory(sKey));

    pgpFact = new JcaPGPObjectFactory(clear);

    PGPCompressedData c1 = (PGPCompressedData) pgpFact.nextObject();

    pgpFact = new JcaPGPObjectFactory(c1.getDataStream());

    PGPLiteralData ld = (PGPLiteralData) pgpFact.nextObject();
    ByteArrayOutputStream bOut = new ByteArrayOutputStream();

    InputStream inLd = ld.getDataStream();

    int ch;
    while ((ch = inLd.read()) >= 0) {
        bOut.write(ch);
    }

    //System.out.println(bOut.toString());

    bOut.writeTo(new FileOutputStream(ld.getFileName()));
    //return bOut;

}

public static void encryptFile(OutputStream out, String fileName, PGPPublicKey encKey) throws IOException, NoSuchProviderException, PGPException {
    Security.addProvider(new BouncyCastleProvider());

    ByteArrayOutputStream bOut = new ByteArrayOutputStream();

    PGPCompressedDataGenerator comData = new PGPCompressedDataGenerator(PGPCompressedData.ZIP);

    PGPUtil.writeFileToLiteralData(comData.open(bOut), PGPLiteralData.BINARY, new File(fileName));

    comData.close();

    PGPEncryptedDataGenerator cPk = new PGPEncryptedDataGenerator(new BcPGPDataEncryptorBuilder(SymmetricKeyAlgorithmTags.TRIPLE_DES).setSecureRandom(new SecureRandom()));

    cPk.addMethod(new BcPublicKeyKeyEncryptionMethodGenerator(encKey));

    byte[] bytes = bOut.toByteArray();

    OutputStream cOut = cPk.open(out, bytes.length);

    cOut.write(bytes);

    cOut.close();

    out.close();
}

Now here is how to invoke/run the above:

try {
         decryptFile(new FileInputStream("encryptedFile.gpg"), new FileInputStream("PrivateKey.asc"), new FileInputStream("PublicKey.asc"), "yourKeyPassword".toCharArray());

        PGPPublicKey pubKey = readPublicKeyFromCol(new FileInputStream("PublicKey.asc"));

        encryptFile(new FileOutputStream("encryptedFileOutput.gpg"), "fileToEncrypt.txt", pubKey);




    } catch (PGPException e) {
        fail("exception: " + e.getMessage(), e.getUnderlyingException());
    }

Solution 2

It just means that content has been signed and then encrypted, the routine provided does not know how to deal with it, but at least tells you that. PGP protocol presents as a series of packets some of which can be wrapped in other ones (for example compressed data can also wrap signed data or simply literal data, these can be used to generate encrypted data as well, actual content always appears in literal data).

If you look at the verifyFile method in the SignedFileProcessor in the Bouncy Castle OpenPGP examples package you will see how to handle the signature data and get to the literal data containing the actual content.

I would also recommend looking at RFC 4880 so you have some idea of how the protocol works. The protocol is very loose and both GPG, BC, and the variety of products out there reflect this - that said the looseness does mean that if you try and cut and paste your way to a solution you'll end up with a disaster. It's not complicated, but understanding is required here as well.

Comments

[CodeMed]

Thank you. I very much want to understand. But I need more specifics. Can you please show code to fix this? I am good at decomposing working code samples. This material is complicated. Your paragraphs seem esoteric without working code to illustrate what you mean.

[CodeMed]

I added the code for SignedFileProcessor.verifyFile() to my edit at the end of my OP.

[J Richard Snape]

The issue here is with understanding the PGP specification (argh!). I have now sent a message that is encoded but not signed and used a (slightly modified) version of your code to decode it. However, when the message is signed, the PGP object stream that you want to get at (with the LiteralData objects) is effectively 'nested' within the CompressedData object. We need to be totally clear what you want to do - have you already got the attachment part of your email out of the raw email, or do you want to fully decode the email exactly as it arrives?

[J Richard Snape]

Basically - I think you really want to save off the encrypted attachment and then decode it as a file. This suits the model of your decryptFile() code better and I think might suit your purposes better. I think this question is getting rather long for SO. Maybe we should continue it in chat?

[CodeMed]

Thank you and +1 for taking the time to add insight to this old question.