How to parse and validate a WebSocket frame in Java?

Question

I wrote a WebSocket frame decoder in Java:

private byte[] decodeFrame(byte[] _rawIn) {
        int maskIndex = 2;
        byte[] maskBytes = new byte[4];

        if ((_rawIn[1] & (byte) 127) == 126) {
            maskIndex = 4;
        } else if ((_rawIn[1] & (byte) 127) == 127) {
            maskIndex = 10;
        }

        System.arraycopy(_rawIn, maskIndex, maskBytes, 0, 4);

        byte[] message = new byte[_rawIn.length - maskIndex - 4];

        for (int i = maskIndex + 4; i < _rawIn.length; i++) {
            message[i - maskIndex - 4] = (byte) (_rawIn[i] ^ maskBytes[(i - maskIndex - 4) % 4]);
        }

        return message;
    }

It works, but I have no idea how to validate a frame in order to make sure that it decodes only valid frames.

The protocol description http://tools.ietf.org/html/rfc6455 unfortunately does not tell much about frame-validation.

Solution

Parsing a raw websocket frame is easy enough. But you have to inspect the header one byte at a time.

Here's a rough example:

I left a few TODO's for you to work out on your own (after reading the RFC-6455 spec of course)

Things you can validate:

Base Framing Protocol: RFC-6455 - Section 5.2

• Is the opcode found one of the valid ones defined in the spec?
• Are RSV bits being used improperly?

Client-to-Server Masking: RFC 6455 - Section 5.3

• If Frame was sent by Client, is the Frame Masked?
• Is the Mask random from frame to frame?
• Don't allow [0x00, 0x00, 0x00, 0x00] as a mask.

Fragmentation: RFC 6455 - Section 5.4

• Is it a fragmented Control frame?
• Is the fragmentation of a large message, consisting of multiple frames, out of order?
• Was a new message started before prior one completed with a FIN flag?

Control Frames: RFC 6455 - Section 5.5

• Does the payload length of a control frame exceed 125 bytes?
• Is the payload fragmented?

Close Frames: RFC 6455 - Section 5.5.1

• If a status code is provided in the payload, does the status code conform to one of the status codes declared in section 7.4.1? Don't forget to to check the IANA registry of websocket status codes that were added after the RFC was finalized)
• Is the status code one that is allowed to be sent over the network in a Frame? (see codes 1005, and 1006 for example)
• If a /reason/ is provided in the frame, does it conform to UTF-8 encoding rules?
• Have you received any frames, of any kind, after a Close frame? (this is a no-no)

Data Frames: RFC 6455 - Section 5.6

• If you receive a TEXT payload data (from TEXT + CONTINUATION frames), does the payload data conform to UTF-8 encoding rules?

While you can validate at the individual frame level, you will find that some of the validations above are validations of state and behavior between multiple frames. You can find more of these kinds of validations in Sending and Receiving Data: RFC 6455 - Section 6.

However, if you have extensions in the mix, then you will also need to process the frames from the point of view of the negotiated extension stack as well. Some tests above would appear to be invalid when an extension is being used.

Example: You have Compression Extension (RFC-7692) (such as permessage-deflate) in use, then the validation of TEXT payload cannot be done with the raw frame off the network, as you must first pass the frame through the extension. Note that the extension can change the fragmentation to suit its needs, which might mess up your validation as well.

package websocket;

import java.nio.ByteBuffer;
import java.nio.charset.Charset;

public class RawParse
{
    public static class Frame
    {
        byte opcode;
        boolean fin;
        byte payload[];
    }

    public static Frame parse(byte raw[])
    {
        // easier to do this via ByteBuffer
        ByteBuffer buf = ByteBuffer.wrap(raw);

        // Fin + RSV + OpCode byte
        Frame frame = new Frame();
        byte b = buf.get();
        frame.fin = ((b & 0x80) != 0);
        boolean rsv1 = ((b & 0x40) != 0);
        boolean rsv2 = ((b & 0x20) != 0);
        boolean rsv3 = ((b & 0x10) != 0);
        frame.opcode = (byte)(b & 0x0F);

        // TODO: add control frame fin validation here
        // TODO: add frame RSV validation here

        // Masked + Payload Length
        b = buf.get();
        boolean masked = ((b & 0x80) != 0);
        int payloadLength = (byte)(0x7F & b);
        int byteCount = 0;
        if (payloadLength == 0x7F)
        {
            // 8 byte extended payload length
            byteCount = 8;
        }
        else if (payloadLength == 0x7E)
        {
            // 2 bytes extended payload length
            byteCount = 2;
        }

        // Decode Payload Length
        while (--byteCount > 0)
        {
            b = buf.get();
            payloadLength |= (b & 0xFF) << (8 * byteCount);
        }
        
        // TODO: add control frame payload length validation here

        byte maskingKey[] = null;
        if (masked)
        {
            // Masking Key
            maskingKey = new byte[4];
            buf.get(maskingKey,0,4);
        }
        
        // TODO: add masked + maskingkey validation here

        // Payload itself
        frame.payload = new byte[payloadLength];
        buf.get(frame.payload,0,payloadLength);

        // Demask (if needed)
        if (masked)
        {
            for (int i = 0; i < frame.payload.length; i++)
            {
                frame.payload[i] ^= maskingKey[i % 4];
            }
        }

        return frame;
    }

    public static void main(String[] args)
    {
        Charset UTF8 = Charset.forName("UTF-8");

        Frame closeFrame = parse(hexToByteArray("8800"));
        System.out.printf("closeFrame.opcode = %d%n",closeFrame.opcode);
        System.out.printf("closeFrame.payload.length = %d%n",closeFrame.payload.length);

        // Examples from https://www.rfc-editor.org/rfc/rfc6455#section-5.7
        Frame unmaskedTextFrame = parse(hexToByteArray("810548656c6c6f"));
        System.out.printf("unmaskedTextFrame.opcode = %d%n",unmaskedTextFrame.opcode);
        System.out.printf("unmaskedTextFrame.payload.length = %d%n",unmaskedTextFrame.payload.length);
        System.out.printf("unmaskedTextFrame.payload = \"%s\"%n",new String(unmaskedTextFrame.payload,UTF8));

        Frame maskedTextFrame = parse(hexToByteArray("818537fa213d7f9f4d5158"));
        System.out.printf("maskedTextFrame.opcode = %d%n",maskedTextFrame.opcode);
        System.out.printf("maskedTextFrame.payload.length = %d%n",maskedTextFrame.payload.length);
        System.out.printf("maskedTextFrame.payload = \"%s\"%n",new String(maskedTextFrame.payload,UTF8));
    }

    public static byte[] hexToByteArray(String hstr)
    {
        if ((hstr.length() < 0) || ((hstr.length() % 2) != 0))
        {
            throw new IllegalArgumentException(String.format("Invalid string length of <%d>",hstr.length()));
        }

        int size = hstr.length() / 2;
        byte buf[] = new byte[size];
        byte hex;
        int len = hstr.length();

        int idx = (int)Math.floor(((size * 2) - (double)len) / 2);
        for (int i = 0; i < len; i++)
        {
            hex = 0;
            if (i >= 0)
            {
                hex = (byte)(Character.digit(hstr.charAt(i),16) << 4);
            }
            i++;
            hex += (byte)(Character.digit(hstr.charAt(i),16));

            buf[idx] = hex;
            idx++;
        }

        return buf;
    }
}

Comments

[Philipp]

There are more opcodes which make sense. Continuation, close, ping and pong can all be sent by the web browser when it feels like doing so. I also saw other opcodes in the wild. I witnessed Firefox sending the reserved opcode 0xb on closing the connection in some cases.

[JavaIntermediate]

@Philipp your answer unfortunately is not helpful at all. The handshake occurs before the data frames come into play. His question is simple how to tell that it is a vaild frame before his server trys to decode it. The only unknown that is left out of the question is what kind of frames does the server intend to accept. My -1 for my answer thus far is invalid in my opinion because currently my answer is the most on target.

[JavaIntermediate]

Also random opcodes in the wild that does not follow RFC 6455 IMO is the problem with browser compatibility. There's a specific design for specific reasons not to say your 0xb example was invalid because it simple says reserved for future control frames. Google Chrome's design that doesn't allow Sec-WebSocket-Protocol: if it isn't given is a good example. Firefox's choice to ignore this at least my version 23.0.1 is what makes coding that much more difficult.

[Damian Kołakowski]

Hi man. A section "// Decode Payload Length" parses the extended length incorrectly. After changing the implementation to: gist.github.com/glock45/5b4a0c38cfba32c85139, it works. What'a your opinion ?

[Joakim Erdfelt]

@DamianKołakowski you don't need an if check on the bytecount for that loop, it will still be valid. The 8800 example case shows this to be true (in the main method of this example)

[Damian Kołakowski]

Joakim: yeah, that's right :) The check is not needed. I updated the gist.