Function Calls, Part 2 (Stack and Calling Conventions)

Introduction

In Part 1 of the article series, you looked at the basics of a function call from the perspective of the generated code. In the process, you got familiar with two registers used by the processor in the x86 family—the EIP (instruction pointer) and the ESP (stack pointer). In this part, you will take a little more detailed look into stack and its relevance in the context of function calls. In the process, you will explore two most commonly used calling conventions in Windows/C programming.

Stack. What Is It?

Stack is a memory area (in the 4GB process space) dedicated for a thread where in it can store data that it will need for execution. The data could be things like local variables used by your code, temporary variables used by the compiler, function arguments, and such. The stack is called so because its behavior is like a stack of cards, as far as the processor goes. This means, when you stack items on it, it always goes on the top, and when you remove items from it, you always remove from the top. In technical terms, it is a LIFO (Last In First Out) data structure.

As explained in Part 1, a dedicated stack is made available to each thread by the system. The size of the stack defaults to 1MB, but it can be overridden by the process’ image header value if there is one. It can also be overridden by specifying a value explicitly in a call to CreateThread or _beginthreadex APIs.

At any time, the processor needs to know where the top of the stack is. This top of the stack is what the ESP register points to. Unlike the EIP register (which cannot be explicitly modified), the ESP register can be implicitly modified by the processor, or explicitly modified by instructions.

For example,

  • A push instruction implicitly decrements the ESP value by 4 and pushes a 32-bit value specified at that location. It is like putting a card on the top of a stack of cards. The stack is now grown by 4 bytes.
  • The pop instruction implicitly retrieves 32 bits at the current ESP location onto the target and then increments the value of ESP by 4. Using the card analogy, it is like removing a card from the top of the stack. The stack has now shrunk by 4 bytes.
  • Instructions such as mov ESP, [source], or sub esp, [value] explicitly increase/decrease/modify the ESP value, effectively changing the top of the stack location. These are explicit because the ESP register is specified in the instruction as the target.

How Are Parameters Passed to a Function?

Simple answer. Via the stack. The caller of the function knows what to pass; the caller knows how many parameters to pass. So, if a caller calls a function sum that takes two ints as parameters like below,

int sum(int argument1, int argument2)

the caller, in broad terms, does this:

  • It pushes the two parameters to the stack via two push instructions. As a result of this, the stack pointer (ESP) is now implicitly decremented by 2 * 4 bytes; in other words, the top of the stack has been raised by 8 bytes.
  • It issues a call instruction to the function’s address. As a result of this, the stack pointer (ESP) is now implicitly decremented by another 4 bytes because the act of issuing a call instruction implicitly pushes the location of the instruction following the call onto the stack.

How Does the Function Retrieve the Parameters?

Simple answer again. Via the stack. Once inside the function sum, before executing any instruction, this is what it the stack will look like. The current ESP value (top of the stack) will be pointing to the location where to continue execution after the sum function returns. If you go down the stack by 4 bytes (in other words, contents at ESP + 4), there you will find one of the parameters to the sum function. Go down another 4 bytes (in other words, contents at ESP + 8); there you will find the other parameter to the function.

As you see, once inside the function, the code has all that it needs to execute.

Calling Conventions

In this context, I need to talk about calling conventions. A calling convention is the protocol for passing arguments to functions. It is an agreement between the caller and the callee. What you just saw in the “How Are Parameters Passed?” and “How Does the Function Retrieve the Parameters?” sectionss is that protocol. That was a general view. But, if you are working in Microsoft land, there are specific calling conventions. Most notable of those are:

  • __cdecl
  • __stdcalll
  • thiscall

In this part, you will dig into the mechanism of the __cdecl and _stdcall calling conventions and learn how the stack and the compiled code look in each case. This will be all hands-on.

More by Author

Must Read