Visual C++ Network: Why do my machine send an RST packet in reply to a SYN/ACK pac

[Mathew Joy]

Q: Raw sockets - Why do my machine send an RST packet in reply to a SYN/ACK packet?

A: This is one of the most frequently asked question by someone who is experimenting with raw sockets and TCP/IP. It is known that the 'IP_HDRINCL' socket option allows you to include the IP header along with the data. Since TCP encapsulates the IP header, we can also build a TCP packet and send it over a network. But the problem is, a TCP connection can never be established this way. The scenario is as follows:

A TCP connection is always made by a three-way handshake. So, initially you send a 'SYN' packet to the remote machine. If it is actively listening on the port, you get a 'SYN/ACK' packet. So far so good. But before you can respond, your machine sends an 'ACK/RST' packet and connection attempt is ended. For the connection to be complete, instead of the 'RST' packet, your machine should be sending an 'ACK' to the remote machine.

The difference lies where the connection is exactly made. Although the programs are communicating after the connection is complete, the TCP connection is never between two programs but rather between the TCP stacks of the two machines. Here 'stack' means a layer of programs that communicates between each other. TCP stack stands for the protocol driver or the actual network transport protocol. Now lets look at exactly what happens when you send a 'SYN' packet...

Since you are using raw sockets ('SOCK_RAW') and not TCP/Stream sockets ('SOCK_STREAM') the TCP stack has no information about what you are doing at program level. And since the 'IP_HDRINCL' allows you to build any type of IP packet and send it along with the data, you can build a 'SYN' packet and send it to the TCP server program which is actively listening. But the point is that the 'SYN' packet is being sent from your program and not the stack. In other words the TCP stack of your machine has no idea how of sending the 'SYN' packet.

On the other side the 'SYN' packet is received by the stack at the remote machine and not exactly by the program. As with the case of the arrival of any 'SYN' packet, the stack at the remote machine responds with a 'SYN/ACK' packet. This packet is now received by the TCP stack of your machine. In other words, the incoming TCP packet ('SYN/ACK') will be processed by the stack. Since it has no information of the previous sent 'SYN' packet, it responds with a 'RST' packet, as in the case of any improper or unacceptable packet for a connection.

So the difference between sending and receiving a TCP packet using raw sockets is, the former is not processed while the latter is processed by the TCP stack of your machine.

Thanks to Mr. Andreas Masur for his help.