November 8, 2016
While debugging a problem in some C code recently, I ran across an assembly
instruction causing problems for a function return value. The culprit:
never seen this instruction before, and it was the cause a serious problem for a
function returning a pointer I had just written.
The code I was writing was spread across two functions. Here is a simplified version of it:
I compiled the files like this:
Then when I ran the executable, this happened:
Why is the pointer returned to
main incorrect? Notice that it is not totally
wrong, just mostly wrong (which is still wrong enough when it comes to pointers).
These are the two values in binary:
The high 32-bits of the pointer (I was compiling this on a 64-bit processor) are all 1’s after being returned from the function. This calls for a look at the assembly code!
The assembly code
Let’s generate the assembly code here to see what is really happening:
Here is the relevant part of the
main function, where it calls
Everything looks good until that odd
cltq instruction appears. What is it doing
there? Stack Overflow to the rescue!
I never forward declared the
ReturnsAPointer method, so GCC assumed the return
int. Therefore, the 64-bit pointer return value was treated as a 32-bit
int that needs to be extended to 64 bits, so
cltq is the correct instruction. It
widens a 32-bit value to 64 bits, filling in the upper 32 bits with the value in
the highest bit of the lower 32 bits.
Intel assembly has a number of similar instructions that
might be used instead of
cltq in cases like this. The are
cdqe. The solution to any of these cases is to forward declare the
ReturnsAPointer function, so the compiler can know the proper type of its return
The rest of the story
I wasn’t totally honest above. It turns out that this behavior is not nearly as surprising as I first thought it was. Here is the complete output of the compile step I showed earlier:
GCC is clearly indicating the exact problem via a warning. Unfortunately, the code I was using causes lots of warnings when it is compiled, so I missed this one and cost myself some debugging time. The moral of this story is: compile without warnings if at all possible.