Writing a Load Balancer From Scratch in 250 Lines of Code
Hey, everyone. It’s another weekend, and I was exploring what to build. So I decided to build a simple yet completely functional load balancer. Let’s discuss it in this post.
What Is a Load Balancer?
A load balancer is a component that distributes incoming traffic across multiple backend servers according to a chosen policy. It is used in large-scale systems where a large number of requests need to be handled effectively.
In large applications several instances of the same application are deployed to provide horizontal scaling. In order to understand how a load balancer and multiple instances of an application can help in scalability, let’s do some math.
1Total Requests: 1000000/second
2Application Instance: 1
3Requests Per Instance: 1000000/1 = 1000000
So a single instance will need to serve all those 1,000,000 requests. It may fail to serve, or it may crash. Who knows?
But what if we deploy 5 instances of the same application?
1Total Requests: 1000000/second
2Application Instances: 5
3Request Per Instance: 1000000/5 = 200000
Now every instance needs to handle 200000 requests, which will sum up to 1000000 requests combined by 5 instances.
But the question is, how do we uniformly distribute requests to these 5 instances? This is where the load balancer comes into place. See the below diagram for more clarity.
Instead of making direct requests to the application, our client will make requests on the load balancer. After that, the load balancer will redirect that request to the appropriate instance to evenly distribute load.
Load Balancing Strategies
There are several popular strategies that a load balancer can follow to distribute requests across instances. Each strategy has its own use case depending on the type of workload.
- Round Robin: In this strategy, the load balancer sends each new request to the next server in line. Once it reaches the end of the list, it starts again from the beginning. This ensures that traffic is evenly distributed in a circular fashion.
- Least Connection: Here, the load balancer forwards the request to the server with the fewest active connections at that moment. This works well when the load on each request can vary significantly.
- IP Hash: With IP Hash, the client’s IP address is used to determine which server should handle the request. This helps in maintaining session persistence, as the same client will always be directed to the same server.
- Weighted Round Robin: This is a variation of Round Robin, where each server is assigned a weight based on its capacity. Servers with higher weights receive more requests. It’s useful when instances have different processing power.
In our code we will implement Round Robin. But in the end, you will write such a code that can be extended with any strategy.
Code Implementation
Server Pool
Below piece of code is responsible for managing our pool of backend servers. Instead of dealing with individual servers separately, we group them together using a structure called ServerPool.
1type ServerPool struct {
2 Servers []*types.Server
3}
Here, Servers is a slice of pointers to types.Server. That means every server we add to this pool will be stored here.
1func NewServerPool() *ServerPool {
2 return &ServerPool{
3 Servers: make([]*types.Server, 0),
4 }
5}
This function initializes a new empty list of servers and returns a pointer to the ServerPool. We’ll use this function to create our server pool when the program starts.
1func (svp *ServerPool) AddServer(server *types.Server) error {
2 svp.Servers = append(svp.Servers, server)
3 return nil
4}
5
6func (svp *ServerPool) GetAllServers() []*types.Server {
7 return svp.Servers
8}
Every time we want to add a new backend server, we call this AddServer method. It simply appends the new server to our list.
Finally, if we want to get the list of all available servers, we use GetAllServers method.
This returns all the servers currently present in the pool. Later, when the load balancer needs to choose a server, it can pick from this list.
File Source: https://github.com/x-sushant-x/Balancer/blob/main/pool/pool.go
Server Struct
To keep track of each backend server, we define a Server struct. This struct holds all the important information related to a single server instance.
1package types
2
3import "time"
4
5type Server struct {
6 ID string `json:"id"`
7 Name string `json:"name"`
8 Protocol string `json:"protocol"`
9 Host string `json:"host"`
10 Port int `json:"port"`
11 URL string `json:"url"`
12 HealthCheckURL string `json:"health_check_url"`
13 IsHealthy bool `json:"is_healthy"`
14 Timeout time.Duration `json:"timeout"`
15 LastHealthCheck time.Time `json:"last_health_check"`
16 FailureCount int `json:"failure_count"`
17 SuccessCount int `json:"success_count"`
18 HealthyAfter int `json:"healthy_after"`
19 UnhealthyAfter int `json:"unhealthy_after"`
20 RetryCount int `json:"retry_count"`
21}
- ID: A unique identifier for the server. This can help us distinguish one server from another in the pool.
- Name: A human-readable name like “Server 1” or “Server A” to help us refer to the server more easily.
- Protocol: This tells us whether the server is using HTTP, HTTPS, or some other protocol.
- Host: The address of the server, such as localhost or an IP like 192.168.1.10.
- Port: The port number on which the server is listening. For example, 3000 or 8080.
- URL: A full URL string that combines protocol, host, and port. This makes it easy to send requests to the server.
- IsHealthy: A boolean flag that tells us whether the server is currently healthy or not. The load balancer can use this to avoid sending requests to servers that are down.
- LastHealthCheck: This stores the time when the last health check was performed on the server. It helps us monitor when the server was last verified to be healthy.
File Source: https://github.com/x-sushant-x/Balancer/blob/main/types/types.go
Round Robin Implementation
This piece of code implements the Round Robin strategy for our load balancer. The goal here is to pick the next server from the pool in a circular order—one after another—so that the load is evenly distributed.
1type RoundRobinBalancer struct {
2 pool *pool.ServerPool
3 mu sync.Mutex
4 idx int
5}
- pool holds the list of all available servers.
- mu is a mutex that makes sure two requests don’t try to access or update the index at the same time. This is important because our load balancer might be handling multiple requests at once.
- idx keeps track of the last server we selected. We’ll use this to pick the next one.
1func NewRoundRobinBalancer(pool *pool.ServerPool) *RoundRobinBalancer {
2 return &RoundRobinBalancer{
3 pool: pool,
4 idx: -1,
5 }
6}
Here, we start with idx as -1, which means no server has been selected yet.
Now comes the core logic in the GetNextServer() method:
1func (rb *RoundRobinBalancer) GetNextServer() (*types.Server, error) {
2 servers := rb.pool.GetAllServers()
3
4 if len(servers) == 0 {
5 return nil, errors.New("no servers found")
6 }
7
8 rb.mu.Lock()
9 defer rb.mu.Unlock()
10
11 rb.idx = (rb.idx + 1) % len(servers)
12
13 selectedServer := servers[rb.idx]
14
15 return selectedServer, nil
16}
We first get the list of all servers. If there are none, we return an error.
We lock access to idx so that only one request can change it at a time. Once the method is done, it automatically unlocks (because of defer).
This line moves to the next server in the list. Once we reach the end, % len(servers) brings us back to the start—creating a circular loop.
So in Simple Terms:
Whenever a request comes in, the load balancer calls GetNextServer(), picks the next server in line, and sends the request there. This keeps the load evenly distributed across all available servers.
File Source: https://github.com/x-sushant-x/Balancer/blob/main/core/round-robin.go
Balancer
Now that we have a list of servers and a strategy (like Round Robin), the final piece is to actually forward incoming requests to those servers. That’s exactly what this file is doing.
Let’s understand how this works step by step.
1type BalancerStrategy interface {
2 GetNextServer() (*types.Server, error)
3}
This interface expects any load balancing strategy (like Round Robin) to implement a GetNextServer() method. This allows us to swap different strategies without changing the core logic.
1type LoadBalancer struct {
2 strategy BalancerStrategy
3}
This struct holds the selected strategy. It will use that strategy to pick which server the next request should go to.
1func NewLoadBalancer(strategy BalancerStrategy) LoadBalancer {
2 return LoadBalancer{
3 strategy: strategy,
4 }
5}
We use this function to create a new instance of LoadBalancer, and we pass in the strategy we want it to follow.
1func (lb *LoadBalancer) Serve(w http.ResponseWriter, r *http.Request) {
2 server, err := lb.strategy.GetNextServer()
3 if err != nil {
4 http.Error(w, "internal server error", 500)
5 return
6 }
7
8 targetURL, err := url.Parse(server.URL)
9 if err != nil {
10 http.Error(w, "Invalid backend URL", http.StatusInternalServerError)
11 return
12 }
13
14 targetPath := strings.TrimRight(targetURL.String(), "/") + r.URL.Path
15 if r.URL.RawQuery != "" {
16 targetPath += "?" + r.URL.RawQuery
17 }
18
19 req, err := http.NewRequest(r.Method, targetPath, r.Body)
20 if err != nil {
21 http.Error(w, "Failed to create request to backend", http.StatusInternalServerError)
22 return
23 }
24
25 for k, values := range r.Header {
26 for _, value := range values {
27 req.Header.Add(k, value)
28 }
29 }
30
31 req.Header.Set("X-Forwarded-For", r.RemoteAddr)
32
33 client := &http.Client{
34 Timeout: time.Second * 30,
35 }
36
37 res, err := client.Do(req)
38 if err != nil {
39 http.Error(w, "Failed to reach backend server", http.StatusBadGateway)
40 return
41 }
42
43 byteResp, err := io.ReadAll(res.Body)
44 if err != nil {
45 http.Error(w, "Failed to read response body", http.StatusInternalServerError)
46 return
47 }
48
49 defer res.Body.Close()
50
51 for k, values := range res.Header {
52 for _, value := range values {
53 w.Header().Add(k, value)
54 }
55 }
56
57 w.WriteHeader(res.StatusCode)
58
59 _, _ = w.Write(byteResp)
60}
The Main Logic: Serve Method
This is the heart of the load balancer. It handles each incoming request and forwards it to the selected backend server.
- We ask the strategy (like Round Robin) to give us the next available server.
- We convert the server’s raw URL string into a proper URL object that we can work with.
- We build the full path we want to send the request to, including any query parameters (like ?search=value).
- We create a new HTTP request that mirrors the original one (same method like GET/POST, same body).
- Then, we copy all headers from the original request.
- We add an
X-Forwarded-Forheader using the client address. Note thatr.RemoteAddrnormally includes the client port; a production implementation should extract the host portion before using it as an IP header value. - We send the request using Go’s http.Client. If the server is down or slow, the HTTP client waits according to its configured timeout before failing the request.
- We read the response body, then copy all response headers from the backend server to the original client.
- We also make sure to return the same status code and response body.
So in simple words:
- A request comes in.
- We pick the next available server.
- We build a new request and send it to that server.
- We copy the response from the backend and send it back to the client.
This is intentionally a small learning project. A production load balancer would need connection/client reuse, health checking, retries, timeouts at multiple layers, streaming rather than buffering entire responses, and careful handling of headers and failures.
File Source: https://github.com/x-sushant-x/Balancer/blob/main/core/balancer.go
main.go — Entry Point
This is the entry point of our load balancer. Everything comes together here—from defining backend servers to starting the load balancer on a specific port.
Defining Our Backend Servers
1
2var servers = []types.Server{
3 {
4 ID: "1",
5 Name: "Server 1",
6 Protocol: "http",
7 Host: "localhost",
8 Port: 3001,
9 URL: "http://localhost:3001",
10 IsHealthy: true,
11 LastHealthCheck: time.Now(),
12 HealthyAfter: 3,
13 UnhealthyAfter: 3,
14 HealthCheckURL: "http://localhost:3001/health",
15 },
16 {
17 ID: "2",
18 Name: "Server 2",
19 Protocol: "http",
20 Host: "localhost",
21 Port: 3002,
22 URL: "http://localhost:3002",
23 IsHealthy: true,
24 LastHealthCheck: time.Now(),
25 HealthyAfter: 3,
26 UnhealthyAfter: 3,
27 HealthCheckURL: "http://localhost:3002/health",
28 }, {
29 ID: "3",
30 Name: "Server 3",
31 Protocol: "http",
32 Host: "localhost",
33 Port: 3003,
34 URL: "http://localhost:3003",
35 IsHealthy: true,
36 LastHealthCheck: time.Now(),
37 HealthyAfter: 3,
38 UnhealthyAfter: 3,
39 HealthCheckURL: "http://localhost:3003/health",
40 },
41}
Here we define three servers. Each server has a unique ID, name, and a URL where it’s running (e.g., http://localhost:3001). These are the backend servers that will actually handle the incoming requests. We also set IsHealthy to true and record the current time as LastHealthCheck. For now, we’re manually marking them as healthy.
1func main() {
2 pool := serverPool.NewServerPool()
3
4 for _, server := range servers {
5 pool.AddServer(&server)
6 }
7
8 rb := balancer.NewRoundRobinBalancer(pool)
9
10 lb := balancer.NewLoadBalancer(rb)
11
12 healthChecker := healthchecker.NewHealthChecker(time.Second*5, pool)
13 go healthChecker.CheckServersHealth()
14
15 distributeLoad(3000, lb)
16}
- We create a new server pool to manage our list of backend servers. Think of this as a container that holds all the available servers.
- Now loop through each server in our predefined list and add it to the pool. This makes them available for the load balancer to use.
- Create a round-robin balancer and pass it our server pool. This will cycle through the healthy servers according to the round-robin strategy.
- Create the actual load balancer and tell it to use the round robin strategy.
- distributeLoad function sets up a new HTTP server on port 3000. This is the port where the client will send all incoming requests. The load balancer will receive those requests and forward them to the appropriate backend server.
Inside distributeLoad
1func distributeLoad(port int, lb balancer.LoadBalancer) {
2 mux := http.NewServeMux()
3
4 mux.HandleFunc("/", lb.Serve)
5
6 server := &http.Server{
7 Addr: fmt.Sprintf(":%d", port),
8 Handler: mux,
9 }
10
11 log.Printf("Starting load balancer on port %d", port)
12
13 if err := server.ListenAndServe(); err != nil {
14 log.Fatalf("Load balancer on port %d failed: %v", port, err)
15 }
16}
- We define a new HTTP router and tell it to handle all incoming requests ("/") using our load balancer’s Serve method.
- Now let’s create an HTTP server bound to the specified port (3000 in our case) and attach our router to it.
File Source: https://github.com/x-sushant-x/Balancer/blob/main/main.go
Complete Source Code: https://github.com/x-sushant-x/Balancer
Summary
So what happens when a client makes a request to http://localhost:3000/?
- The load balancer receives it.
- It asks the Round Robin strategy to pick the next server.
- It forwards the request to that server (like http://localhost:3001/).
- The response from the backend server is sent back to the client.
Health Checking
There is one more thing that our load balancer is missing, and that is health checking. We need to make sure that our load balancer only sends requests to healthy instances.
We are not going to implement that in this same post. Instead, I’ll write another post for that.