Having a problem with inline assembly and calling conventions

[Psytherium]

I'm working on an assignment that creates a function using inline assembly to compute the value of an expression. The following is the signature of the function.


long evaluate(char *code, bool trace);


The expression pointed to by the code parameter computes a signed 32-bit value using a simple expression language. Each expression contains three fields. The first field is required but the other two vary in size depending on the value of the first field. The memory layout is shown below.

Expression encoding:
1-byte op-code | optional left operand bytes | optional right operand bytes

Op-codes include three fields as shown below.

Bits Description
7-6 Operation code
5-3 Left operand value
2-0 Right operand value

The operation code indicates how the operands will be transformed into the final resulting expression value.

Code Operation
00 minimum of left and right operand
01 maximum of left and right operand
10 sum of left and right operands
11 difference between left and right operands (left-right)

Each operation requires two operands (left and right). The value of each operand is given in the table below.

Code Value of the operand
000 constant 0
001 constant 1
010 constant 2
011 constant 4
100 1-byte signed value
101 2-byte signed value
110 4-byte signed value
111 an expression (as defined here) (We'll ignore this one).

Constants require no additional bytes. One-, two-, and four-byte values require the appropriate number of bytes.

Here are some example inputs and what the output should be.

Input | Output
00 | min(0,0) = 0
4A | max(1,2) = 2
9C 7F | 4 + 127 = 131


The following is my code. It compiles without any errors, but when i run it it just crashes.


//add2.cpp

long __declspec(naked) evaluate(const char *code, bool trace)
{
_asm{
push ebp
mov esp, ebp
push dword ptr [ebp+8]
push code

call eval

add esp, 12
ret

eval:
mov eax, 0
mov ebx, 0
mov ecx, 0
mov edx, 0
mov esi, [ebp+8]
mov ebx, [esi]
mov dl, [esi]
and dl, 56
shr dl, 3
cmp dl, 7
jne notexp

nextthreebits:
cmp ecx, 1
je op
mov dl, [ebx]
and dl, 7 ;00000111
cmp dl, 7
mov ecx,1 ;checked nextthreebits
jne notexp

jp op

notexp:
cmp dl, 4 ;00000100 (check to see if byte value)
jge signedvalue
cmp dl, 3 ;00000011 (check to see if const 4)
je constant4
and edx, 255 ;11111111
push edx
jp nextthreebits

constant4:
push 4
jp nextthreebits

signedvalue:
add dl, 4
cmp dl, 0
je onebyte
cmp dl, 1
je twobyte
and edx, 255
push edx
add esi, 4
jp nextthreebits

onebyte:
inc esi
mov dl, [esi]
and edx, 255
push edx
jp nextthreebits

twobyte:
inc esi
mov dx, [esi]
and edx, 255
push edx
add esi, 2
jp nextthreebits


op:
mov dl, [ebx]
and dl, 192 ;11000000
cmp dl, 0
je min
cmp dl, 64 ;01000000
je max
cmp dl, 128 ;10000000
je sum
cmp dl, 192 ;11000000
je subtract
jp done

min:
pop ebx ; right value
pop eax ; left value
cmp ebx,eax ; right-left
jle right
jp done

right:
mov eax, ebx
jp done

max:
pop ebx
pop eax
cmp ebx,eax
jg right
jp done

sum:
pop ebx
pop eax
add eax, ebx
jp done

subtract:
pop ebx
pop eax
sub eax, ebx
jp done

recursive:
jp done

done:
}
}


Here is the test file.


//testeval.cpp

#include <stdio.h>
#include <assert.h>

long evaluate(const char *code, bool traceFlag);

int main()
{
static const char test1[] = { 10 };
static const char test2[] = { 0x4A };
static const char test3[] = { 0x9C, 0x7F };

long x = evaluate(test1,true);
printf("test1 result %d\n\n",x);

x = evaluate(test2,true);
printf("test2 result %d\n\n",x);

x = evaluate(test3,true);
printf("test3 result %d\n\n",x);

return 0;
}


Any help would be GREATLY appreciated. I need help on this ASAP. I don't necessarily need the code revised for me(though that would be excellent) but atleast just some help on where I'm going wrong or which parts in the code i need to look at, etc. Thanks!

[SpinningVertex]

I'm afraid I can't give you any exact instructions for a solution. But since it crashes, we can assume that you either cause an access violation or that you do not preserve the registers well enough for C so that the your thread starts trying to run code at some random place in the memory.

If the later is the case, it can easily be fixed by adding PushAd (Which pushes All general purpos registers to the stack) at the start of your asm and PopAd(which pops them all) before "ret".

If it still crashes after that,.. then you probably have an access violation. I usually make those when I treat a pointer as a value or the otherway around.

Oh, and try to avoid using the registers directly, if your compiler translates the parameters and result names to registers when compiling. It makes it a bit easier to keep track of all the variables.

Sorry if this isn't much help but it is all I can think of at the moment.

Hope it turns out well!