Built a TCP Load Balancer in C to understand how it actually works.

A lot of engineers use load balancers every day. But most of them never go deep enough to understand how it works. I did, and I implemented a Layer 4 load balancer in C to understand how it works. It is not a production-grade solution but is enough to give a core conceptual idea of modern load balancers.

Why I Built It?

I’m a great fan of a quote attributed to Nobel Prize-winning physicist Richard Feynman: “What I cannot create, I do not understand.” That is why I try to build the things I find interesting.

What Is a Load Balancer?

I’ll put it in simple words. You hosted a website on a server. Suppose that server can handle 10,000 concurrent connections. The exact capacity depends on the workload, hardware, and application. In the future, traffic may grow beyond that capacity. You then have two broad options:

  1. Increase server resources (vertical scaling).
  2. Add one more server to the system (horizontal scaling).

Suppose you choose the second approach and add another server. Now clients need a way to reach the correct backend without knowing which instance should handle the connection. A load balancer sits in front of those servers, accepts client connections, selects a backend, and forwards traffic between the client and that backend. Load balancers have several algorithms to decide which server to connect to. You can read about those algorithms here.

Just for a visual demonstration, below is a diagram.

2 Types of Load Balancers

  1. Layer 4 load balancers operate at the transport layer and commonly forward TCP or UDP traffic.
  2. Layer 7 load balancers operate at the application layer and can make routing decisions using protocols such as HTTP.

I implemented a Layer 4 load balancer to keep things simple and work on direct TCP instead of diving into HTTP complexities.

The Simplest Possible Architecture

A load balancer has 4 simple responsibilities:

  • Accept client connections
  • Select a backend server
  • Connect to that backend
  • Forward traffic in both directions

So the architecture is simple. A load balancer listens for client connections on a TCP port. Once a client connects, the load balancer selects a backend server and forwards traffic between the two connections. When the server responds to the load balancer, the response will be sent to the client.

But How Do We Handle Multiple Connections?

In this simplified architecture, the load balancer is single-threaded, so a blocking operation could prevent it from handling other connections. That would limit concurrency even if the backend servers themselves could handle much more traffic.

The Threaded Approach

A threaded design could assign a thread to each connection or unit of work. That can be simple to reason about, but large numbers of simultaneously runnable threads can create memory and scheduling overhead. Real load balancers also use thread pools, event loops, asynchronous I/O, processes, or combinations of these approaches. This is related to the classic C10K problem, which popularized the challenge of handling roughly 10,000 concurrent connections efficiently. For this project, we therefore want an event-driven design rather than one thread per connection.

But Why Is There a Problem With Threads?

For those who are not very familiar with operating systems, here is a simple explanation of why the thread-per-request approach fails.

Let’s say you built your load balancer in Java with a thread-per-request approach. A Java thread has a configurable, implementation-dependent stack size; it is not universally fixed at 1 MB of physical RAM. If you had 10,000 simultaneously active threads, the memory and scheduling overhead could still be substantial, but it would be misleading to simply multiply 10,000 by 1 MB and call that physical RAM usage.

There is another problem: a typical connection flow might look like this: client connects → thread is created → load balancer forwards traffic → backend processes the request → backend returns a response → load balancer returns it to the client. While the backend is waiting or processing, a dedicated thread may spend much of its time blocked.

Event-Driven Approach

We have another approach. Instead of creating a thread for every connection, event-driven load balancers use readiness notifications so a small number of workers can handle many connections. The idea is surprisingly simple. Rather than asking the operating system to create 10,000 threads, we ask it to notify us when interesting events occur on sockets. Linux provides a mechanism called “epoll” for exactly this purpose.

Think of it like this. Instead of hiring 10,000 employees and assigning one employee to each customer, you hire a single receptionist. Customers come and go throughout the day. Most of the time they are waiting for something. The receptionist simply keeps track of who needs attention right now and only talks to those people. That is exactly what epoll does.

A load balancer may have thousands of client and server connections open at the same time. Most of these connections are idle and waiting for data. Instead of continuously checking every connection, the load balancer goes to epoll and asks:

“Which connections are ready right now?”

The operating system returns only the connections that have something interesting happening on them, such as:

  • New client connection arrived
  • Client sent data
  • Backend server sent response
  • Connection closed

The load balancer then processes only those sockets and immediately goes back to waiting for more events. A typical flow now looks like this:

Client Connected → epoll notifies load balancer → Load Balancer accepts connection

Client Sent Data → epoll notifies load balancer → Load Balancer forwards data to backend

Backend Sent Response → epoll notifies load balancer → Load Balancer forwards response to client

Notice something important here. While the backend server is processing a request, the load balancer is not blocked. It is free to handle thousands of other connections. There is no dedicated thread sitting idle waiting for a response. This solves both problems we discussed earlier.

Memory Usage Drops

With thread-per-request model, 10,000 requests could mean 10,000 threads and several gigabytes of memory consumed by thread stacks. With epoll, a single thread can manage thousands or even hundreds of thousands of connections. Memory consumption becomes proportional to connection metadata instead of thread stacks.

Less Context Switching

Operating systems constantly switch between runnable threads. This operation is called a context switch and it is not free. CPU registers must be saved and restored every time the scheduler moves between threads. When thousands of threads exist, the CPU spends a significant amount of time performing context switches instead of doing useful work.

An event-driven load balancer can often run with a small number of worker threads and use readiness notifications to handle many connections efficiently. This dramatically reduces context switching overhead and allows the CPU to spend more time forwarding traffic. This is one reason high-performance systems such as NGINX, HAProxy, and many modern proxies rely heavily on event-driven architectures instead of spawning a thread for every connection.

Implementation

I implemented the same event-driven idea from scratch in C. Instead of epoll, I used kqueue because epoll is Linux-specific and I am developing on macOS, which provides kqueue. The underlying idea—waiting for socket readiness—is similar.

You can find source code on GitHub