Strength of RSA Encrypting an AES Key

Question

I'm currently developing a system to transmit data between client and server, and was wondering what the strength of the encryption I planned to use was.

My thought was to have a private/public RSA key pair and hand out the public key to each client (leaving the private key solely on the server). Each client would then generate their own AES key and RSA encrypt it. They would then AES encrypt their data and send the encrypted data and encrypted AES key to the server. The server would then decrypt the AES key using the private key, and then decrypt the data using the AES key.

Are there any security flaws I am missing and/or is there a better way to do this?

Solution 1

This is almost exactly how SSL/TLS works (check out the handshake section). The only thing to make it stronger is to generate the AES key for each connection, rather than using the same key each time. You might also want to digitally sign messages that go back and forth to avoid man-in-the-middle and other spoofing attacks.

Generally speaking, creating a good cryptosystem is very difficult. When possible, you should always favor an existing (trusted) application to help out. In this case, you might consider sending your messages using HTTPS, rather than creating your own system.

Solution 2

You should give us more information about the language and platform you are using, so that we can give you specific recommendations about libraries that already exist and wich will handle the details for you. Using cryptographic primitives directly is not trivial and difficult to get exactly right, and with cryptography, you have to be "not exactly right" only once for your security to be broken.

---

To answer your question, it's generally a better idea to create a session secret (the AES key) through a Diffie-Hellman exchange, and each side use its private RSA key to sign its key-exchange data.

Otherwise, if the session secret is not established through a DH exchange, an adversary that gains access to the private RSA key (which has to be stored somewhere) could decrypt all traffic that was ever sent between the client and server.

If the secret is established through a DH exchange, then only the authentication part of the protocol would be exposed. Although an adversary in possession of the private RSA key would then not be able to read any previous communication, he still could either enter an authenticated dialog with the client/server or launch a man-in-the-middle attack (which may or may not be easily done, depending on the network).