How does the assembly TEST instruction work with these jump instructions?

Question

Using AT&T assembly syntax, I'm trying to understand how testl is used in assembly code. Specifically:

testl  %edx, %edx
jle    .L3

I know testl does a bitwise and of the same value to set the condition flags, but how can I interpret 'jump if less than or equal to' if it isn't comparing two values?

Solution 1

Here's an excerpt from the official documentation from Intel on test:

Operation
TEMP ← SRC1 AND SRC2;
SF ← MSB(TEMP);
IF TEMP = 0
    THEN ZF ← 1;
    ELSE ZF ← 0;
FI:
PF ← BitwiseXNOR(TEMP[0:7]);
CF ← 0;
OF ← 0;

And the same on jle:

Jump if less or equal (ZF=1 or SF≠OF)

So, the jump will be performed if edx is 0 (because edx AND edx = edx and that's 0 only when edx is 0, and because ZF is set to 1 when the result of AND is 0) or if the most significant bit of edx is 1 (because SF = most significant bit of edx AND edx (or, equivalently, of edx itself) and OF is always 0, which means SF ≠ OF is only true when SF ≠ 0).

IOW, the jump will be performed only if edx is ≤ 0 when interpreted as a signed integer or, equivalently, when edx is either 0 or greater or equal than 0x80000000 when interpreted as an unsigned integer.

Solution 2

TESTL with identical arguments (like edx and edx) sets the flags based on the value of that argument itself (since x AND x is identical to x). So we can forget about the AND altogether here since it's discarded - all we need to concern ourselves with is the value in edx.

With TESTL, the zero flag ZF is set to 1 only if the value is zero. TESTL also forces the overflow flag OF to 0 and sets the sign flag SF only if the high bit is set.

JLE will then jump if either ZF is set to 1, or SF <> OF.

So, the jump will execute if either:

• edx was zero; or
• edx had its high bit set.

Hence it will jump for edx values of 0 or 0x80000000 - 0xffffffff.

Most likely this is a check to ensure that the number is a natural number 0x00000001 - 0x7fffffff, the jump would be to an error handling routine of some sort and a valid natural number would continue without the jump, something like:

loop_for_number:
    call   get_number_into_edx
    testl  %edx, %edx
    jle    loop_for_number

    ; carry on here knowing that edx >= 1

---

For a description of the various jumps and the flags they use, see here.

Solution 3

In x86 assembly almost all conditional jumps are based on flags (except jcxz, jecxz, loop and loopne/ loopnz). This means that all that matters are the values the flags have.

jle is synonymous with jng. The jump condition is ZF = 1 or SF <> OF. You may want to check Intel x86 JUMP quick reference.

test does set all flags except AF link, so everything looks good so far.

According to this link logical operations always zero OF. That means your jump would practically be ZF = 1 or SF = 1, so in your code jle would jump if edx was 0 or between range 0x80000000 ... 0xffffffff.

Comments

[us2012]

+1, great catch about the signed integers. That makes a lot more sense than checking for 0 or >= 0x80000000!

[amorimluc]

IOW, in the case of signed integers, the jump instructions always compares the tested value (testl) to 0. Thanks!

[Alexey Frunze]

There's nothing else to effectively compare with. If you did cmp edx, 5 or sub edx, 5, you'd be looking at edx-5 vs 0 instead of just edx vs 0.