ANOMALY: meaningless REX prefix used

Question

What does the error ANOMALY: meaningless REX prefix used mean? I have googled and all information I got was completly random that it is related to java or avg or minecraft (because of java).

However, I got this error in the console output of my Visual Studio console application after I merged several branches of my c++ opengl 4.0 graphics engine and it suddenly popped up. I might have updated the AMD graphics driver between the time points I have written them, so this could be one source. After the error popped up also the depth buffer test was suddenly disabled.

After using clean and rebuild in visual studio the error is gone now, I therefore do not need help in fixing the error but I would like to know what it means and what in general causes this error. It makes me curious as I have not found ANYTHING useful searching for this error.

Solution

Myria in the comments said:

It's referring to an x86-64 assembly instruction using a REX prefix byte when it didn't need to

To expand upon this, REX prefixes are ignored in a few different scenarios.

1. If the ModR/M field specifies other registers or an extended opcode.

2. If more than 1 REX prefix is used in an instruction (though I read on osdev.org this is undefined

3. If general formatting isn't followed. For example the REX prefix must precede the opcode or escape opcode byte unless being used in conjunction with a mandatory prefix. In which case REX can be right after the opcode/escape byte.

4. If you try to use the single byte form of INC/DEC in 64 bit mode.

Looks like this ANOMALY message displays in a variety of contexts from git to Java related programs (maybe the one you are referencing) in which a new driver seems to have been the problem. The culprit: Raptr, which comes with AMD's Radeon drivers. In the Java post someone reported using SAPPHIRE Radeon HD 5850 and on the next site I'll link you to, one person was using AMD R9 390 and another the 380. In this context someone saw the message on the console of their 64-bit Win7 sys. Now this person's site took me through a hook Raptr was using (which connects to the opengl32.dll) called mhook, I started digging through this 'Windows API hooking library' and found this starting on line 1230:

assert(X86Instruction->AddressSize >= 4);
    if (X86Instruction->rex.w)
    {
        X86Instruction->OperandSize = 8;
        X86Instruction->HasOperandSizePrefix = FALSE;
    }
    else if (X86Instruction->HasOperandSizePrefix)
    {
        assert(X86Instruction->OperandSize == 2);
    }
    else if (X86Instruction->rex_b == REX_PREFIX_START)
    {
        if (!Instruction->AnomalyOccurred)
        {
            if (!SuppressErrors) printf("[0x%08I64X] ANOMALY: meaningless REX prefix used\n", VIRTUAL_ADDRESS);
            Instruction->AnomalyOccurred = TRUE;
        }
        X86Instruction->rex_b = 0;
    }

To summarize, this ANOMALY message occurs when software handles a REX prefix ignore, like this Windows API library does.

So there you have it, you were in all the right places. The mhook library even has a long list of Visual Studio files to ignore.
additional note* I found this comment from the os2museum site a good clue to this whole mystery

The Windows amd64 ABI requires that the first opcode of a function be at least 2 bytes in length. (I think this is so the function can be hotpatched.) Many times the first instruction is “push ” but the instruction has a 1-byte encoding! To comply with the ABI, a rex prefix is added to the instruction, making it 2 bytes — “rex push rbp” or “rex push rbx” or whatever. The compiler does this for you, but if you are writing a function in assembler, you need to remember the rule.

Other fun error messages (just a few of many!) in this particular hook library include

ANOMALY: Meaningless segment override

ANOMALY: REX prefix before legacy prefix 0x%02X\n

ANOMALY: Conflicting prefix\n

ANOMALY: Reached maximum prefix count %d\n

and my favorite:

ANOMALY: branch into the middle of an instruction\n

And just because I can't help myself, it might be worth noting these are the instructions that default to 64-bit operands:

+--------------+------------+-------------+
| CALL (near)  | ENTER      | Jcc         |
+--------------+------------+-------------+
| JrCXZ        | JMP (near) | LEAVE       |
+--------------+------------+-------------+
| LGDT         | LIDT       | LLDT        |
+--------------+------------+-------------+
| LOOP         | LOOPcc     | LTR         |
+--------------+------------+-------------+
| MOV CR(n)    | MOV DR(n)  | POP reg/mem |
+--------------+------------+-------------+
| POP reg      | POP FS     | POP GS      |
+--------------+------------+-------------+
| POPFQ        | PUSH imm8  | PUSH imm32  |
+--------------+------------+-------------+
| PUSH reg/mem | PUSH reg   | PUSH FS     |
+--------------+------------+-------------+
| PUSH GS      | PUSHFQ     | RET (near)  |
+--------------+------------+-------------+