Step by Step Developing a SOHO HTTP Filter

1.Abstract

Most available web filters work inline; this means that all outgoing and incoming packets are passed through a filter driver. This approach, along with its own benefits, has a big flaw; the filtering process affects data transfer throughput. This project represents an experimental remedy to this issue by putting the filter engine in sniffer mode. This way, the filtering process and data transfer act independently.

2. Requirements

  • The article expects the reader to be familiar with C++ and TCP/IP concepts.
  • The source code uses the following libraries:
    • Winsock
    • Ethereal: A packet capture and network analyzer
  • The code was compiled and built with VC7 on Windows server 2003.

3. Introduction

The main goal of this article is to explain the practical details of low-level network programming. There are many commercial and open source firewalls and HTTP filters available on both Linux and Windows. But internally, most of them follow the same approach to find their targets. The reference section of this article provides you with handy books related to this topic.

Specifically, a web filter could act in two modes to inspect outgoing packets for blacklist keywords: Inline mode and Sniffer mode. I explain both modes of operation and compare them.

4. Background

This section explains the TCP/IP protocol stack, HTTP protocol behavior, and a Boyer-Moore algorithm to perform fast pattern matching. If you think that you have enough experience and knowledge and want to get your hands dirty, please continue to Section 7 (Implementation).

A mission critical system is totally different from an office application. Imagine your team plans to develop a firewall, web filter, or even a secure proxy server that processes tons of packets in a few seconds. What are the main characteristics of these systems?

Being fail-safe, high-performance, full-feature, and GREEN are a few. Green means the system should not eat up CPU cycles and the memory of the hosted platform. Meanwhile, in such an expensive project, there is no room for logical mistakes due to a lack of technical knowledge! In a typical environment, your last result must be deployed in the model below:

4.1 TCP/IP protocol stack

As the name describes, TCP/IP refers to a number of protocols, each of which was developed for a purpose. To understand a HTTP session establishment process, you should know at least the following protocols: Ethernet II, ARP, IP, TCP, UDP, HTTP, and DNS.

4.2 Ethernet II

The Ethernet II frame format was defined by the Ethernet specification created by Digital, Intel, and Xerox before the IEEE 802.3 Specification. The Ethernet II frame format is also known as the DIX frame format.

Ethernet II consists of the following fields: (Totally 26 bytes + payload from 46 bytes to 1500 bytes)

  • The Preamble (8 bytes) consists of 7 bytes of alternating 1s and 0s (each byte is the bit sequence 10101010) to synchronize a receiving station and a 1-byte 10101011 sequence that indicates the start of a frame. The Preamble provides receiver Synchronization and frame delimitation services.
  • The Destination Address (6 bytes) indicates the destination's address. The destination can be a unicast, a multicast, or the Ethernet broadcast address. The unicast address is also known as an individual, physical, hardware, or MAC address. For the Ethernet broadcast address, all 48 bits are set to 1 to create the address 0xFF-FF-FF-FF-FF-FF.
  • The Source Address (6 bytes) indicates the sending node's unicast address.
  • The EtherType (2 bytes) indicates the upper layer protocol contained within the Ethernet frame. For an IP datagram, the field is set to 0x0800. For an ARP message, the EtherType field is set to 0x0806. For a complete list, refer to the references.
  • The Payload field for an Ethernet II frame consists of a protocol data unit (PDU) of an upper-layer protocol. Ethernet II can send a maximum-sized payload of 1500 bytes. Because of Ethernet's collision detection facility, Ethernet II frames must send a minimum payload size of 46 bytes.
  • The FCS (4 bytes) provides bit-level integrity verification on the bits in the Ethernet II frame using the CRC algorithm.

A typical capture shows you the following fields: (FCS and preamble are excluded)

  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  | destination   | source       | protocol |                   |
  | mac address   | mac address  | type     | IP DATAGRAM       |
  | (6 bytes)     | (6 bytes)    | 0X0800   |                   |
  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

4.3 ARP

ARP is a broadcast-based, request-reply protocol that provides a dynamic address resolution facility to map next-hop IP addresses to their corresponding MAC addresses.

There are two facts regarding the datalink layer that show you need ARP:

  1. When an Ethernet frame is sent from one host on a LAN to another, it is the 48-bit Ethernet address that determines for which interface the frame is destined. The device driver software never looks at the destination IP address in the IP datagram.
  2. The next-hop IP address is not necessarily the same as the destination IP address of the IP datagram, the result of the route. The determination process for every outgoing IP datagram is a next-hop interface and a next-hop IP address.

For direct deliveries to destinations on the same subnet, the next-hop IP address is the datagram's destination IP address. For indirect deliveries to remote destinations, the next-hop IP address is the IP address of a router on the same Subnet as the forwarding host. To get that device as a next hop, the packet needs its hardware address.

4.4 IP

IP, the heart of a TCP/IP protocol suite, provides a connectionless, unreliable delivery of data. By unreliable, I mean that there is no guarantee that a datagram successfully gets to its destination. By connectionless, I mean that the IP doesn't maintain any information regarding successive datagrams. On the other hand, each datagram is handled independently. The IP makes a best effort to deliver packets to the next hop or the final destination. End-to-end reliability is the responsibility of upper-layer protocols such as TCP.

The IP header contains the following fields that you need to know for later packet processing:

  • Version (4 bytes): Indicates the format of Internet header.
  • IHL (4 bytes): Is the length of IP header in 32-bit words.
  • Type of service or TOS (1 byte): As the name indicates, it specifies how important the IP packet is for you. Some intermediate devices evaluate this field in the case of high load and prioritize the datagram. In RFC 791 (Internet protocol), this field is structured as follows:
  •      0     1     2     3     4     5     6     7
    
         +-----+-----+-----+-----+-----+-----+-----+-----+
         |                 |     |     |     |     |     |
         |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
         |                 |     |     |     |     |     |
         +-----+-----+-----+-----+-----+-----+-----+-----+
    
         D                >> Delay
         T                >> Throughput
         R                >> Reliability
         Bit6 and Bit7    >> reserved
    
  • Total length (2 bytes): Total size of header + payload. The total length field conceptually allows the length of a datagram to be up to 65,535 bytes, although such a long packet is impractical for most hosts and network devices.

    Later, you will learn what an MTU is and how it may help you put this in reality. Anyway, remember that the only IP header is 20 bytes and if there are any options, the length can go as high as 60 bytes. No more!

  • ID (2 bytes): Assigned by the sender so that the receiver can decrement the fragmented IP packets due to the MTU value.

    For the second time, I've mentioned "MTU;" see what the MTU is: Simply put, most of the data you generate while, for example, surfing the web, are bulk data. It means that the size of the data is big. The underlying media access protocol splits the bulk into smaller parts so that it can send seamlessly over the network infrastructure. In case of HTTP 802.3 Ethernet Protocol, the maximum size of a datagram is 1500 bytes. This number is the MTU; it stands for Maximum Transmission Unit.

    In case you want to transmit a 15000-byte data stream to your mate, the protocol stack splits your message to 10 * 1500 bytes and transmits them one by one. If you put 20 bytes for the IP header, 1480 byes remain for the transport layer, header, and payload. There is where ID comes into the picture. The protocol stack splits the message into 10 smaller messages and assigns a unique ID in the IP header. When the receiver takes all the pieces, it can do further processing over the whole message.

  • Flags (3 bits): Says whether or not the datagram is a part of a fragmented data.
  •         0   1   2
            +---+---+---+
            |   | D | M |
            | 0 | F | F |
            +---+---+---+
    
  • Fragment Offset (13 bits): An 8-byte chunk of data is called a fragment block. The number in the Fragment Offset field reports the size of the offset in fragment blocks. The Fragment Offset field is 13 bits long, so offsets can range from 0 to 8191 fragment blocks—corresponding to offsets of from 0 to 65,528 bytes.
  • TTL (1 byte): This field says how long a datagram could remain alive in a network system. It measures time in seconds.
  • Protocol (1 byte): Indicates the upper layer protocol type. For example:
      1 >> ICMP
      2 >> GIMP
      4 >> IP in IP encapsulation
      6 >> TCP
    17 >> ADP
    41 >> IPV6
    47 >> Generic routing encapsulation (ARE)
    50 >> IP security encapsulation security payload (ESP)
    51 >> IP security authentication header (AH)
    89 >> ASP
  • Header checksum (2 bytes): To measure the integrity of the header, the protocol stack handler performs a CRC on the header and compares it with the checksum value. It is a kind of sanity check.
  • Source IP Address and Destination IP Address (each 4 bytes).
  • Options (variable length): Maintains a list of optional information for the datagram.

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |Version|  IHL  |Type of Service|          Total Length       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         Identification        |Flags|      Fragment Offset  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  Time to Live |    Protocol   |         Header Checksum     |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                       Source Address                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Destination Address                      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Options                    |    Padding  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    

Step by Step Developing a SOHO HTTP Filter

4.5 TCP

TCP is a fully formed Transport Layer protocol that provides a reliable data-transfer service and a method to pass TCP-encapsulated data to an application Layer protocol. TCP has the following characteristics:

  • Connection oriented: Before any send or receive, both sides of data transfer must negotiate a TCP connection using a TCP three-way handshake process. TCP connections also terminated through a graceful TCP four-way termination process.
  • Full duplex: From one pipe, data can be sent and received simultaneously. TCP does this job perfectly, gaining from the sequence number and acknowledgment number fields in its header.
  • Reliable data sent on a TCP connection is sequenced and a positive acknowledgment is expected from the receiver. If no acknowledgment is received, the segment is retransmitted.
  • Flow control: To avoid sending too much data at one time and congesting the routers of the the IP internetwork, TCP implements sender-side flow control that gradually scales the amount of data sent at one time. To avoid having the sender send data that the receiver cannot buffer, TCP implements receiver-side flow control that indicates the amount of space left in the receiver's buffer.
  • Segmentation of application layer data: TCP segments data obtained from the Application Layer process so that it will fit within an IP datagram sent on the Network Interface Layer link. TCP peers exchange the maximum-sized segment that each can receive and adjust the TCP maximum segment size using Path Maximum Transmission Unit (PMTU) discovery.
  • One-to-one delivery of data: CP connections are a logical point-to-point circuit between two application Layer protocols. TCP does not provide a one-to-many delivery service.

    The following figure shows a TCP segment encapsulated in an IP datagram:

    <----------------- IP datagram -------------------->
    ++++++++++++++++++++++++++++++++++++++++++++++++++++
    |   IP      |  TCP      |          TCP             |
    | Header    | Header    |         Data             |
    ++++++++++++++++++++++++++++++++++++++++++++++++++++
    20 bytes    20 bytes
    At-least    At-least
    

    TCP header:

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |          Source Port          |       Destination Port      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        Sequence Number                      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Acknowledgment Number                    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  Data |           |U|A|P|R|S|F|                             |
    | Offset| Reserved  |R|C|S|S|Y|I|            Window           |
    |       |           |G|K|H|T|N|N|                             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |           Checksum            |         Urgent Pointer      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                    Options                    |   Padding   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                             data                            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    
  • Source port (2 bytes): The source port number.
  • Destination port (2 bytes): The destination port number.
  • Sequence Number (4 bytes): Indicates the outgoing byte-stream-based sequence number of the segment's first octet.
  • Acknowledgment Number (4 bytes): If the ACK control bit is set, this field indicates the sequence number of the next octet in the incoming byte stream that the receiver expects to receive.
  • Data Offset (4 bits): Indicates where the TCP segment data begins. The Data Offset field is also the TCP header's size. Just as the IP header's Header Length field, the Data Offset field is the number of 32-bit words (4-byte blocks) in the TCP header. For the smallest TCP header (no options), the Data Offset Ffield is set to 5 (0x5), indicating that the segment data begins in the twentieth octet offset starting from the beginning of the TCP segment (the offset starts its count at 0). With a Data Offset field set to its maximum value of 15 (0xF), the largest TCP header, Including TCP options, can be 60 bytes long.
  • Reserved (6 bits): Reserved for future use. Must be zero.
  • Control Bits (6 bits from left to right): URG: Urgent Pointer field significant:
    • ACK: Acknowledgment field significant
    • PSH: Push Function
    • RST: Reset the connection
    • SYN: Synchronize sequence numbers
    • FIN: No more data from sender
  • Window (2 bytes): The number of data octets beginning with the one indicated in the acknowledgment field that the sender of this segment is willing to accept.
  • Checksum (2 bytes): The checksum field is the 16-bit one's complement of the one's complement sum of all 16-bit words in the header and text. While computing the checksum, the checksum field itself is replaced with zeros.

    To compute a TCP checksum, you need to consider a TCP pseudo header. The TCP pseudo header is used to associate the TCP segment with the IP header. The TCP pseudo header is added to the beginning of the TCP segment only during the checksum calculation and is not sent as part of the TCP segment. The use of the TCP pseudo header assures the receiver that a routing or fragmentation process did not improperly modify key fields in the IP header.

    +--------+--------+--------+--------+
    |           Source Address          |
    +--------+--------+--------+--------+
    |         Destination Address       |
    +--------+--------+--------+--------+
    |  zero  |PROTOCOL|    TCP Length   |
    +--------+--------+--------+--------+
    
  • Urgent Pointer (2 bytes): This field communicates the current value of the urgent pointer as a positive offset from the sequence number in this segment. The urgent pointer points to the sequence number of the octet following the urgent data. This field is be interpreted only in segments with the URG control bit set.
  • Options (variable): Options may occupy space at the end of the TCP header and are a multiple of 8 bits in length.
  • Padding (variable): The TCP header padding is used to ensure that the TCP header ends and data begins on a 32-bit boundary. The padding is composed of zeros.

4.6 UDP

UDP is a simple, datagram-oriented, transport layer protocol: Each output operation by a process produces exactly one UDP datagram, which causes one IP datagram to be sent. This is different from a stream-oriented protocol such as TCP where the amount of data written by an application may have little relationship to what actually gets sent in a single IP datagram.

+--------+--------+--------+--------+  -----
|     Source   2  | Destination   2 |    |
|      Port  bytes|    Port    bytes|    |
+--------+--------+--------+--------+  8 bytes
|     2 bytes     |     2 bytes     |    |
|     Length      |    Checksum     |    |
+--------+--------+--------+--------+  -----
|                                   |  
|          data octets              |
+-----------------------------------+

4.7 DNS

The Domain Name System, or DNS, is a distributed database that is used by TCP/IP applications to map between hostnames and IP addresses, and to provide electronic mail routing information. You use the term distributed because no single site on the Internet knows all the information. Each site (university department, campus, company, or department within a company, for example) maintains its own database of information and runs a server program that other systems across the Internet (clients) can query. The DNS provides the protocol that allows clients and servers to communicate with each other.

From an application's point of view, access to the DNS is through a resolver. On Windows and Unix hosts, the resolver is accessed primarily through two library functions, gethostbyname() and gethostbyaddr(); they are linked with the application when the application is built. The first takes a hostname and returns an IP address; the second takes an IP address and looks up a hostname. The resolver contacts one or more name servers to do the mapping.

Note: The DNS is an application layer protocol that implements over the UDP transport protocol.

4.8 HTTP

Simply put, HTTP is the protocol behind the World Wide Web. With every web transaction, HTTP is invoked.

HTTP headers

There four types of headers for HTTP:

  • General headers indicate general information such as the date, or whether the connection should be maintained. They are used by both clients and servers.
  • Request headers are used only for client requests. They convey the client's configuration and desired document format to the server.
  • Response headers are used only in server responses. They describe the server's configuration and information about the requested URL.
  • Entity headers describe the document format of the data being sent between client and server.

Although Entity headers are most commonly used by the server when returning a requested document, they are also used by clients when using the POST or PUT methods.

Client requests
  • GET retrieves a resource on the server.
  • HEAD retrieves some information about the document, but doesn't need the document itself.
  • POST says that you are providing some information of your own (for example, in forms). This may change the state of the server in some way, such as creating a record in the database.
  • PUT replaces or creates a new document on the server.
  • DELETE removes a document on the server.
  • TRACE is used for protocol debugging purposes.
  • OPTIONS is used when the client looks for other methods that can be used on the document.
  • CONNECT is used when a client needs to talk to a HTTPS server through a proxy server.

Other HTTP methods that you may see (LINK, UNLINK, and PATCH) are less clearly defined.

For reader the of this article, only an understanding of the GET method is needed. This is the main method used for document retrieval. The response to a GET request can be generated by the server in many ways.

After the client uses the GET method in its request, the server responds with a status line, headers, and data requested by the client. If the server cannot process the request, due to an error or lack of authorization, the server usually sends an explanation in the entity-body of the response.

For example:

A Client request

GET / HTTP/1.1\r\n
Accept: */*\r\n
Referer: http://www.google.com\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2;
   SV1; .NET CLR 1.1.4322)\r\n
Host: www.google.com\r\n
Connection: Keep-Alive\r\n
Cache-Control: no-cache\r\n
Cookie: PREF=ID=76325601198c7def:LD=en:CR=2:TM=1184565890:
   LM=1186242386:S=ofiC-IBOMCVySJCP\r\n
\r\n

Server response

HTTP/1.0 200 OK\r\n
Cache-Control: private\r\n
Content-Type: text/html; charset=UTF-8\r\n
Content-Encoding: gzip\r\n
Server: GWS/2.1\r\n
Content-Length: 2976\r\n
Date: Sat, 04 Aug 2007 15:46:45 GMT\r\n
X-Cache: MISS from netcache-1\r\n
Connection: keep-alive\r\n
\r\n

Step by Step Developing a SOHO HTTP Filter

4.9 Performing a Pattern Search by Using the "Boyer-Moore" Algorithm

The Boyer-Moore approach is very interesting. It doesn't need to actually check every character of the string to be searched but rather skips over some of them. Say you want to find the word "ROTTEN" in a string such as "HTTP://WWW.WEBSERVER.COM/ROTTEN HTTP/1.1. By using the Boyer-Moore algorithm, you have the following steps:

HTTP://WWW.WEBSERVER.COM/ROTTEN HTTP/1.1

ROTTEN

  • First, you fetch "/" because it stands in the same position as the last character of the pattern.
  • There is no "/" in the pattern. The search fails.
  • Move the pattern right along six characters. So, 'N' stands under 'W'. The match fails because there's no 'w' either.
  • Move the pattern right along six characters. So, 'N' stands under 'V'. The match fails because there's no 'V' either. Also, for 'M' a similar state will happen.
  • Move the pattern right along six characters. So, 'N' stands under 'E'. A match is found. There is an 'E' in the pattern. So, move the pattern right along a single place so both 'E's line up, and compare right to left until the exact match is found.

This is exactly what you want. I had a comparison between Boyer-Moore and other case insensitive pattern matchings. The result was much more efficient when I used Boyer-Moore. For more information about the Boyer-Moore pattern matching algorithm, please Google the topic.

5. A HTTP Request Lifetime

This section describes what happens until a page is downloaded from the Internet. Say you open your favorite Web browser and type http://www.google.com in your address bar. Again, remember your network model.

Please run Ethereal, select your outgoing interface, and start capturing. You can check all information below in Ethereal.

5.1 Browser tries to resolve the address to its corresponding IP address

To do this, it makes a single function call to gethostbyname(). This function generates a DNS query. A DNS query, as described in section 4.7, is a UDP datagram that is destined to the corresponding DNS server that is set in TCP/IP properties. The TCP/IP handler takes a look at the destination IP address and realizes that the client IP address and server IP address are in different subnets, so it decides to forward the packet to the next hop, which is your gateway. To do this, the driver looks for the gateway's MAC address in the local ARP table. If, for any reason, the gateway's MAC address is not among the resolved addresses, TCP/IP broadcasts an ARP request in the subnet to get the corresponding MAC address. Upon receiving the gateway's MAC address, the client makes the final packet and puts it on the wire. Then, the packet is received by the gateway. The gateway looks at the destination IP address and realizes that this packet doesn't belong to itself, so it decides to forward this to the next hop.

From now on, the packet traverses a global route until it successfully gets to the destination. The destination protocol handler then hands away the packet to the proper application layer service, which is DNS in this case. Finally, the DNS server fetches any related IP addresses and makes a DNS response destined to the sender.

5.2 Establishing a TCP session with the peer

Destination IP address in hand, your client protocol stack now can establish a TCP session with the peer. The "three-way handshake" is the procedure used to establish a connection. This procedure normally is initiated by one TCP and responded to by another TCP.

The following figure shows the exact process in your example:

You                                               Web server

1. CLOSED                                           LISTEN

2. SYN-SENT    --> <SEQ=ISN1><CTL=SYN><ACK=0>    --> SYN-RECEIVED

3. ESTABLISHED <-- <SEQ=ISN2><ACK=ISN1+1>        <-- SYN-RECEIVED
                   <CTL=SYN,ACK>

4. ESTABLISHED --> <SEQ=ISN1+1><ACK=ISN2+1>      --> ESTABLISHED
                   <CTL=ACK>

5. ESTABLISHED --> <SEQ=ISN1+1><ACK=ISN2+1>      --> ESTABLISHED
                   <CTL=ACK><DATA>

In line 2 of the figure, you begin by sending a SYN segment indicating that you will use sequence numbers starting with sequence number ISN1. In line 3, the Server sends a SYN and acknowledges the SYN it received from you. Note that the acknowledgment field indicates the server is now expecting to hear sequence ISN1+1, acknowledging the SYN that occupied sequence ISN1.

At line 4, you respond with an empty segment containing an ACK for the server's SYN.

5.3 The application layer transactions begins

In line 5, you send some data. Note that the sequence number of the segment in line 5 is the same as in line 4 because the ACK does not occupy sequence number space. As a result, <DATA> contains the following statements:

GET / HTTP/1.1\r\n
Accept: */*\r\n
Referer: http://www.google.com\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;
   .NET CLR 1.1.4322)\r\n
Host: www.google.com\r\n
Connection: Keep-Alive\r\n
Cache-Control: no-cache\r\n
Cookie: PREF=ID=76325601198c7def:LD=en:CR=2:TM=1184565890:
   LM=1186242386:S=ofiC-IBOMCVySJCP\r\n
\r\n

5.4 Receiving a response

The server receives the client's request and responds back with some data.

5.5 Finishing data transfer

To do this, TCP provides you two choices: TCP reset and TCP graceful termination.

TCP resets are usually sent when something goes wrong either for the client or server. To see a real TCP reset, simply type http://www.codeguru.com in your address bar and press the Esc button after a few moments. A TCP reset packet is a 40-byte datagram that TCP flags set to 0x0004 (RST). On the other hand, you have a graceful connection termination that is a four-way sequence.

You                                                Web server

1. ESTABLISHED                                     ESTABLISHED

2. (Close)
   FIN-WAIT-1 --> <SEQ=ISN1><ACK=ISN2>             --> CLOSE-WAIT
                  <CTL=FIN,ACK>

3. FIN-WAIT-2 <-- <SEQ=ISN2><ACK=ISN1+1>           <-- CLOSE-WAIT
                  <CTL=ACK>

4.                                                     (Close)
   TIME-WAIT  <-- <SEQ=ISN2><ACK=ISN1+1>           <-- LAST-ACK
                  <CTL=FIN,ACK>

5. TIME-WAIT  --> <SEQ=ISN1+1><ACK=ISN2+1>         --> CLOSED
                  <CTL=ACK>

6. (2 MSL)
   CLOSED
Note: TCP connection terminations do not have to use four segments. In some cases, Segments 2 and 3 are combined. The result is a FIN-ACK/FIN-ACK/ACK sequence.

6. The Solution for Implementing a Web Filter

The solution is fairly simple. The following sequence is used to filter a matched web request:

  1. Capture packets.
  2. Distinguish TCP packets that are destined to port 80 or 8080.
  3. Look for a 'GET /' in the first four bytes of data. If not found, continue with next captured packet.
  4. Perform a Boyer-Moore pattern match over data against blacklisted keywords. If not found, continue with the next captured packet.
  5. Send a TCP reset to the server and a block page to the client and finish the client transaction by sending a FIN packet.

7. Implementation

This section explains the core functionality of your HTTP filter in detail. All functionalities are available in the source code, in the CProcessPacket class.

7.1 Capturing packets with raw socket

The fastest way to capture incoming traffic without using a NDIS low-level driver is to use a Winsock raw socket. However, you should note that a raw socket cannot capture in promiscuous mode. This means the socket captures only traffic that is destined to its own address.

if((m_sniffSocket = socket(AF_INET, SOCK_RAW,
                           IPPROTO_IP))==SOCKET_ERROR)
   {
      return 0;
   }
// Adapter information
PIP_ADAPTER_INFO pAdapterInfo = m_AdapterInfo;
u_long in = 0;
do {
   if (strcmp (in_szSourceDevice, pAdapterInfo->AdapterName ) == 0)
   {
      break;    // we found the selected adapter
   }
in++;
pAdapterInfo = pAdapterInfo->Next;    // Progress through
}
while(pAdapterInfo);


struct sockaddr_in src;
memset(&src, 0, sizeof(src));
src.sin_addr.S_un.S_addr = inet_addr
   (pAdapterInfo->IpAddressList.IpAddress.String);
src.sin_family      = AF_INET;
src.sin_port        = 0;
if (bind(m_sniffSocket,(struct sockaddr *)&src,sizeof(src))
    == SOCKET_ERROR)
{
   return 0;
}

int j=1;
if (WSAIoctl(m_sniffSocket, SIO_RCVALL, &j, sizeof(j), 0, 0,
    &in,0, 0) == SOCKET_ERROR)
{
   return 0;
}

You can use that function in a loop to capture packets contiguously.

int  res = 0;
   char *pkt_data = (char *)malloc(65536);
   char                    m_pLogString[256];
   Packet                  p;

   if (pkt_data == NULL)
   {
        return 0;
   }

   SetEvent(_this->m_hThrdReadyEvent);

   // Capture packet
   do
   {
      res = recvfrom(_this->m_sniffSocket,pkt_data,65536,0,0,0);
      if(res > 0)
      {
         ZeroMemory(&p, sizeof (Packet));

         DecodeIP((u_int8_t*)pkt_data, res, &p);

         if (p.banned == 1)
         {
            FilterHttpRequest(&p);

            char ip_string_src[17];
            char ip_string_dst[17];

            memcpy (ip_string_src, inet_ntoa(p.iph->ip_src), 17);
            memcpy (ip_string_dst, inet_ntoa(p.iph->ip_dst), 17);

            sprintf (m_pLogString,
               "Keyword \'%s\' is detected in a request from %s to
                %s. Session Dropped!",
               p.matched, ip_string_src, ip_string_dst);

            m_pFilterLog->AddLog(m_pLogString);

         }

      }
   }
   while (res > 0);

With each single call to pcap_next_ex, you get a buffer containing a complete Ethernet frame.

7.2 Distinguishing HTTP packets

In this section, you process portions of a captured packet to find Ethernet, IP, and TCP header fields. To perform this, you need a couple of handy structs to lay data over them.

struct IPHdr
{
   u_int8_t ip_verhl;        /* version & header length */
   u_int8_t ip_tos;          /* type of service */
   u_int16_t ip_len;         /* datagram length */
   u_int16_t ip_id;          /* identification  */
   u_int16_t ip_off;         /* fragment offset */
   u_int8_t ip_ttl;          /* time to live field */
   u_int8_t ip_proto;        /* datagram protocol */
   u_int16_t ip_csum;        /* checksum */
   struct in_addr ip_src;    /* source IP */
   struct in_addr ip_dst;    /* dest IP */
};

struct TCPHdr
{
   u_int16_t th_sport;       /* source port */
   u_int16_t th_dport;       /* destination port */
   u_int32_t th_seq;         /* sequence number */
   u_int32_t th_ack;         /* acknowledgement number */
   u_int8_t th_offx2;        /* offset and reserved */
   u_int8_t th_flags;
#define    TH_FIN    0x01
#define    TH_SYN    0x02
#define    TH_RST    0x04
#define    TH_PSH    0x08
#define    TH_ACK    0x10
#define    TH_URG    0x20
   u_int16_t th_win;         /* window */
   u_int16_t th_sum;         /* checksum */
   u_int16_t th_urp;         /* urgent pointer */
};

// our helper struct to hold packet information in one place
struct Packet
{
   
   /* base pointer to the raw packet data */
   u_int8_t            *pkt;

  
   /* and orig. headers for ICMP_*_UNREACH family */
   IPHdr               *iph;
   u_int32_t            ip_options_len;
   u_int8_t            *ip_options_data;


   TCPHdr             *tcph;
   u_int32_t           tcp_options_len;
   u_int8_t           *tcp_options_data;


   u_int8_t           *data;     /* packet payload pointer */
   u_int16_t           dsize;    /* packet payload size */

   u_int8_t           *http_uri_content;
   u_int32_t           http_payload_len;
   /* HTTP request / HTTP response */
   u_int8_t            http_state;
   /* Indicates if the request should be censored */
   u_int8_t            banned;
   /* Keyword that this request matched to - maximum 128 byte*/
   unsigned char       matched[128];

#define    CLIENT_REQUEST     0x01
#define    SERVER_RESPONSE    0x02
#define    NOT_HTTP           0x04


   u_int8_t frag_flag;     /* flag to indicate a fragmented packet */
   u_int16_t frag_offset;  /* fragment offset number */
   u_int8_t mf;            /* more fragments flag */
   u_int8_t df;            /* don't fragment flag */
   u_int8_t rf;            /* IP reserved bit */
}; 

Now, you can retrieve all header information stored in the captured packet. To perform this step, you should know the related protocol specifications that were explained in section 4.

Packet p;

/* lay the IP struct over the raw data */
p->iph = (IPHdr *) (pkt_data); 

/* Decode only TCP headers */
if (p->iph->ip_proto == 6)
{
   /* lay TCP on top of the data because there is enough of it! */
   // set pinter at the beginning of TCP header
   p->tcph = (TCPHdr *) (pkt_data + IP_HEADER_LEN);
   if (p->tcph->th_flags & TH_ACK    /* If is request */
      && p->tcph->th_flags & TH_PSH)
         {
            /* If target service is not HTTP */
            if(p->tcph->th_dport != htons(80) &&
            p->tcph->th_dport    != htons(8080) )
            return ;
            // continue in 7-3 }
         }
}

Step by Step Developing a SOHO HTTP Filter

7.3 Looking for 'GET /' in the first four bytes

// find payload position
p->data = (unsigned byte*)(pkt_data + ETHERNET_HEADER_LEN +
   IP_HEADER_LEN + TCP_HEADER_LEN)

if( p->data[0] == 'G' &&
   p->data[1] ==  'E' &&
   p->data[2] ==  'T' &&
   p->data[3] ==  ' ' &&
   p->data[4] ==  '/'
   )
{
   // continue in 8.4
}

7.4 Performing a Boyer-Moore pattern matching algorithm on the payload

Boyer-Moore is a fast pattern matching algorithm that follows a one-liner approach to find a match.

if (CheckPattern(p->data, len))
   {
      // A match found, session must be filtered!
      // continue in 7.5
   }

7.5 Finishing data transaction by sending a bunch of packets to both directions

Perhaps this section is the trickiest part of the article. Here again, I remind you that your HTTP filter works in sniffer mode. As a result, you should do your best effort to prevent both sides' reliable TCP engines to live up the session by resynchronizing.

7.5.1 Getting a target packet

Assume that you grasp a target HTTP GET packet with the following TCP information:


SEQ = ISN1
ACK = ISN2
Direction = To Server.
7.5.2 Eliminating server resynchronation

Again, due to the fact that you are operating in monitor mode, you should get rid of server resynchronization attempts. For this purpose, I sent a TCP reset to the server to stop the TCP state machine for a while.

SEQ = ISN1
ACK = ISN2
FLAGS = RST
Direction = To Server.
7.5.3 Sending a block page to the client
SEQ = ISN2
ACK = ISN1 + Len(GET)
FLAGS = FIN | ACK
Direction = To Client.

8 Special Considerations

Although it's a choice for a SOHO environment, your HTTP filter is a less common type of filtering solution for enterprise networks. This comes from the fact that the sniffing mode doesn't guarantee a synchronized reaction against TCP sessions because it doesn't stand against packet flows. Compare it with a football stadium before and after the game. Before the game, guards can inspect fans one by one and check their tickets. After the game, fans just rush to the outdoors and it is fairly easy for an insurgent to get out of sight. At least, in my city it's this way!

It's the matter of one-by-one inspection! Along with its own reliability, a one-by-one inspection has a big flaw: "A slow inspector could slow down the whole movement."

9. Future Work

  • Make the concept more reliable by performing asynchronous analysis in background threads.
  • Replace core capturing functionalities with a network driver hook available in both Windows and Unix platforms.
  • Replace Boyer-Moore with a multi-pattern search so that dozens of patterns can be evaluated in a single search.
  • Look for your ideas or requests.

10. Revision History

  • Initial release: 2007-08-06
  • Total review: Excluding Winpcap and Libnet from the project. Performing both capture and send raw packets by raw socket functionality available in Wincock: 2007-08-26

11. Test Environments

  • Ethernet LAN
  • WLAN 802.11/bg standard sub-network

12. References

  1. TCP/IP Illustrated, Volume 1 The Protocols. W. Richard Stevens.
  2. 2. RFC 793, RFC 791, RFC 768, RFC 826, RFC 1034, RFC 1035. Refer to ietf.
  3. Low-level network and network security programming forum.
  4. Ethereal
  5. Windows server 2003 TCP/IP and services. Technical reference by Josef Davies and thomas Lee.
  6. HTTP pocket reference by Clinton Wong. O'Rielly, May 2000.


About the Author

r1 s2

Developing mission critical communication and network computer software. Network programming, Network security programming, Networking concepts

Downloads

Comments

  • There are no comments yet. Be the first to comment!

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Agile methodologies give development and test teams the ability to build software at a faster rate than ever before. Combining DevOps with hybrid cloud architectures give teams not just the principles, but also the technology necessary to achieve their goals. By combining hybrid cloud and DevOps: IT departments maintain control, visibility, and security Dev/test teams remain agile and collaborative Organizational barriers are broken down Innovation and automation can thrive Download this white paper to …

  • As mobile devices have pushed their way into the enterprise, they have brought cloud apps along with them. This app explosion means account passwords are multiplying, which exposes corporate data and leads to help desk calls from frustrated users. This paper will discover how IT can improve user productivity, gain visibility and control over SaaS and mobile apps, and stop password sprawl. Download this white paper to learn: How you can leverage your existing AD to manage app access. Key capabilities to …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds