Class 33 of 37 • Advanced Linux System Programming

Linux I/O Multiplexing in C - select, poll & High-Performance epoll Guide

In high-load internet services, servers must handle tens or hundreds of thousands of concurrent client connections simultaneously—a milestone known as the C10K Problem. Traditional multithreaded (1 thread per client via pthreads) or multiprocess (fork per client) models collapse under severe memory consumption and CPU context-switching overhead.

Linux solves this challenge through I/O Multiplexing. Instead of blocking on a single network socket, an event loop monitors thousands of file descriptors simultaneously, waking up only when network data is ready to be processed. In this guide, you will master the Linux kernel's high-performance epoll subsystem that powers production engines like Nginx, Node.js (libuv), and Redis.

---

1. Evolution: select() vs poll() vs epoll()

System Call	Time Complexity	Max Descriptors	Kernel Data Structure	Key Limitation
select()	O(N) Linear	1024 (FD_SETSIZE)	Fixed Bitmap bitmasks	Re-scans all 1024 bits; copies bitmaps between user and kernel on every tick.
poll()	O(N) Linear	Unlimited (File limits)	Array of struct pollfd	No 1024 limit, but kernel still scans every descriptor linearly in an O(N) loop.
epoll()	O(1) Event-Driven	Unlimited (RAM limits)	Red-Black Tree + Ready List	Zero scanning overhead; hardware interrupts place ready sockets directly onto ready list!

---

2. The 3 Core epoll System Calls

The epoll API is managed via three clean C system calls declared in <sys/epoll.h>:

• 1. int epoll_create1(int flags);
  Creates an epoll instance in the kernel and returns an epoll file descriptor. Use 0 or EPOLL_CLOEXEC as flags.
• 2. int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);
  Registers, modifies, or deletes descriptors from the epoll interest tree. Operations include EPOLL_CTL_ADD, EPOLL_CTL_MOD, and EPOLL_CTL_DEL. Events include EPOLLIN (ready to read), EPOLLOUT (ready to write), and EPOLLET (edge-triggered).
• 3. int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);
  Waits for I/O events. Blocks until monitored descriptors are ready, or until the timeout expires. Returns the exact count of ready descriptors and populates the events[] array in O(1) time!

---

3. Level-Triggered (LT) vs Edge-Triggered (ET) Modes

---

4. Compilable High-Performance epoll TCP Server in C

Here is a complete, concurrent event-driven TCP Echo Server implemented with epoll:

/* epoll_server.c - Concurrent TCP Echo Server using epoll */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <sys/epoll.h>

#define MAX_EVENTS 64
#define PORT 8080
#define BUFFER_SIZE 1024

/* Make socket non-blocking */
static int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    if (flags == -1) return -1;
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

int main(void) {
    int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
    int opt = 1;
    setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

    struct sockaddr_in server_addr;
    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = INADDR_ANY;
    server_addr.sin_port = htons(PORT);

    bind(listen_fd, (struct sockaddr *)&server_addr, sizeof(server_addr));
    listen(listen_fd, SOMAXCONN);
    set_nonblocking(listen_fd);

    /* 1. Create epoll instance */
    int epfd = epoll_create1(0);

    /* 2. Register listen socket with epoll */
    struct epoll_event ev;
    ev.events = EPOLLIN;
    ev.data.fd = listen_fd;
    epoll_ctl(epfd, EPOLL_CTL_ADD, listen_fd, &ev);

    struct epoll_event events[MAX_EVENTS];
    printf("epoll TCP Server listening on port %d...\n", PORT);

    /* 3. Event Loop */
    while (1) {
        int nfds = epoll_wait(epfd, events, MAX_EVENTS, -1);

        for (int i = 0; i < nfds; i++) {
            if (events[i].data.fd == listen_fd) {
                /* New incoming connection */
                struct sockaddr_in client_addr;
                socklen_t client_len = sizeof(client_addr);
                int client_fd = accept(listen_fd, (struct sockaddr *)&client_addr, &client_len);

                set_nonblocking(client_fd);
                ev.events = EPOLLIN | EPOLLET; /* Edge-Triggered */
                ev.data.fd = client_fd;
                epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);
                printf("[New Connection] Socket FD: %d\n", client_fd);
            } else {
                /* Existing client sent data */
                char buffer[BUFFER_SIZE];
                int client_fd = events[i].data.fd;

                while (1) {
                    ssize_t bytes_read = read(client_fd, buffer, sizeof(buffer));
                    if (bytes_read > 0) {
                        /* Echo back to client */
                        write(client_fd, buffer, bytes_read);
                    } else if (bytes_read == 0) {
                        /* Client closed connection */
                        printf("[Disconnected] Socket FD: %d\n", client_fd);
                        epoll_ctl(epfd, EPOLL_CTL_DEL, client_fd, NULL);
                        close(client_fd);
                        break;
                    } else {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            /* All current data read in ET mode */
                            break;
                        }
                        close(client_fd);
                        break;
                    }
                }
            }
        }
    }

    close(listen_fd);
    close(epfd);
    return 0;
}

Compile and test with multiple concurrent clients using telnet or nc:

$ gcc -Wall epoll_server.c -o epoll_server
$ ./epoll_server
epoll TCP Server listening on port 8080...

# In separate terminals:
$ nc localhost 8080
Hello Veewom Linux
Hello Veewom Linux

---

---

5. University Viva & Job Interview Q&A

---

6. Frequently Asked Questions (FAQ)