Linux TCP/IP Stack Architecture, 3-Way Handshake & Socket States Tutorial

A comprehensive deep dive into DoD 4-Layer vs OSI 7-Layer models, TCP packet mechanics, connection establishment & teardown, TCP vs UDP, and kernel sysctl tuning.

The Transmission Control Protocol / Internet Protocol (TCP/IP) suite is the universal foundational communication standard that connects billions of computers, servers, cloud virtual machines, and mobile devices across the global Internet. Developed originally under the auspices of the United States Department of Defense (DARPA), TCP/IP provides reliable, end-to-end data transmission across heterogenous physical networks.

For system administrators, DevOps engineers, and students preparing for university exams or Linux certifications (RHCSA, LFCS), mastering TCP/IP goes far beyond memorizing layer names. It requires understanding how sequence numbers work, how the 3-way handshake builds a session, why the TIME_WAIT socket state protects your server, and how to tune Linux kernel network parameters via sysctl.

1. TCP/IP 4-Layer Architecture vs OSI 7-Layer Model

While academia commonly teaches the theoretical 7-layer Open Systems Interconnection (OSI) reference model, Linux and the actual Internet implement the practical 4-layer TCP/IP (DoD) architecture. Each layer encapsulates the payload of the layer above it, prepending its own protocol header:

Figure 42.1: The 4-Layer TCP/IP Protocol Stack mapped to Linux commands and utilities

2. Anatomy of the TCP Header

The Transmission Control Protocol (defined in RFC 793) prepends a 20-byte minimum header (up to 60 bytes with options) to ensure complete transmission reliability over an unreliable IP network:

• Source Port (16 bits) & Destination Port (16 bits): Identifies sending and receiving applications (e.g., client ephemeral port 54321 → server port 80).
• Sequence Number (32 bits): Tracks the byte order of outgoing data streams and identifies the Initial Sequence Number (ISN) during session setup.
• Acknowledgment Number (32 bits): Informs the sender of the next expected sequence number (cumulative ACK).
• Data Offset (4 bits): Specifies header length in 32-bit words (e.g., 5 words = 20 bytes).
• Control Flags (9 bits):
  • SYN (Synchronize): Requests connection establishment and synchronizes sequence numbers.
  • ACK (Acknowledgment): Indicates that the Acknowledgment field contains valid data.
  • FIN (Finish): Sender has finished sending data and requests connection closure.
  • RST (Reset): Immediately aborts a malformed or rejected connection.
  • PSH (Push): Directs the receiving kernel to push buffered data immediately to user-space without waiting for a full buffer.
  • URG (Urgent): Flags urgent out-of-band data (rarely used today).
• Window Size (16 bits): Implements flow control, telling the sender how many bytes of unacknowledged data the receiver's buffer can currently accept.
• Checksum (16 bits): Verifies header and payload data integrity against corruption.

3. The TCP 3-Way Handshake (Connection Establishment)

Before an application can transmit a single HTTP request or SSH keystroke, TCP guarantees both peers agree to communicate and establish synchronized sequence numbers through the Three-Way Handshake:

Figure 42.2: TCP 3-Way Handshake, Full-Duplex Data Transfer, and 4-Step Connection Teardown

Step 1: Client sends SYN

The client generates a cryptographically randomized 32-bit Initial Sequence Number (e.g., ISN = 1000). It sends a packet with the SYN flag set and payload length 0. The client enters the SYN_SENT socket state.

Step 2: Server responds with SYN-ACK

The server receives the SYN on a listening port (LISTEN state). It allocates memory in its SYN backlog queue, chooses its own random Sequence Number (e.g., Seq = 5000), sets Ack = 1001 (client ISN + 1), and flags both SYN and ACK. The server transitions to the SYN_RECV socket state.

Step 3: Client acknowledges with ACK

The client receives the SYN-ACK, verifies Ack = 1001, and sends back an acknowledgment with Seq = 1001 and Ack = 5001 (server ISN + 1). Both endpoints now transition to the ESTABLISHED state. Application data can now flow in full duplex.

Exam Insight: Why 3 packets instead of 2? A 2-way handshake would leave the server vulnerable. If an old duplicate SYN packet delayed in router buffers arrives at the server after a session closed, a 2-way handshake would force the server to open an orphaned connection and allocate memory without confirming the client actually initiated it.

4. TCP 4-Step Connection Teardown & Socket Lifecycle States

Because TCP is full-duplex (both directions transmit independently), terminating a session requires closing both channels gracefully:

1. Step 1 (FIN from Client): When the client finishes sending data (e.g., browser completes download), it sends a packet with the FIN flag set and enters FIN_WAIT_1.
2. Step 2 (ACK from Server): The server kernel acknowledges the FIN by sending an ACK. The client enters FIN_WAIT_2, while the server enters CLOSE_WAIT (notifying the server application that the client has closed its send channel).
3. Step 3 (FIN from Server): Once the server application finishes sending remaining buffered response data and calls close(), the server sends its own FIN packet and enters LAST_ACK.
4. Step 4 (Final ACK from Client): The client receives the server's FIN, replies with a final ACK, and enters the TIME_WAIT state. The server receives the ACK and immediately enters CLOSED.

The 11 Standard TCP Socket States in Linux

5. Architectural Comparison: TCP vs UDP

Both TCP and UDP reside at Layer 4 (Transport Layer), but they serve opposite engineering priorities:

6. Linux Kernel TCP Tuning via sysctl

Linux allows real-time tuning of the network subsystem via /proc/sys/net/ipv4/ or permanently through /etc/sysctl.d/99-network-tuning.conf. On high-throughput web servers or API gateways, standard kernel defaults can bottleneck connections:

# 1. Protect against SYN Flood DoS attacks (SYN Cookies)
sudo sysctl -w net.ipv4.tcp_syncookies=1

# 2. Increase maximum SYN backlog queue (pending handshakes)
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096

# 3. Reduce TIME_WAIT timeout from 60 seconds to 30 seconds
sudo sysctl -w net.ipv4.tcp_fin_timeout=30

# 4. Allow kernel to reuse TIME_WAIT sockets for outgoing connections (vital for reverse proxies)
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# 5. Expand ephemeral outgoing port range
sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"

# 6. Apply all sysctl settings permanently from config file
sudo sysctl -p /etc/sysctl.d/99-network-tuning.conf

7. Practical Terminal Lab: Inspecting TCP/IP in Linux

Lab 7.1: Capturing a Real TCP 3-Way Handshake with tcpdump

Open two terminal windows. In Terminal 1, initiate a network capture filtering on port 80 or 443:

# Terminal 1: Capture TCP packets with SYN or ACK flags set
sudo tcpdump -nn -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' -c 3

In Terminal 2, trigger an HTTP request to any web server:

# Terminal 2: Make a quick HTTP request
curl -I http://example.com

Terminal 1 will display the exact 3-way packet sequence:

10:14:20.123 IP 192.168.1.100.48292 > 93.184.216.34.80: Flags [S], seq 182749102, win 64240
10:14:20.145 IP 93.184.216.34.80 > 192.168.1.100.48292: Flags [S.], seq 382910492, ack 182749103, win 65535
10:14:20.146 IP 192.168.1.100.48292 > 93.184.216.34.80: Flags [.], ack 382910493, win 64240
# Note: [S] = SYN, [S.] = SYN-ACK, [.] = ACK

Lab 7.2: Filtering Active Socket States with ss

Query the Linux kernel's internal socket tables with ss:

# 1. View overall socket statistics summary
ss -s

# 2. View all currently ESTABLISHED TCP connections with process names
ss -tanp state established

# 3. View sockets currently lingering in TIME_WAIT
ss -tan state time-wait | head -n 10

# 4. View all listening sockets on TCP and UDP
ss -tulpn

8. Student Viva & Exam Practice Q&A

9. Frequently Asked Questions (FAQ)