Sushant Dhiman

About

Fri, 17 Jul 2026 11:07:26 +0530

Hi, I’m Sushant Dhiman.

I’m a backend software engineer passionate about backend systems, networking, databases, backend architecture, distributed systems and low-level software. This website is where I document my software engineering journey by sharing what I learn, build, and explore.

I believe the best way to learn engineering is by building things. Instead of only reading about how systems work, I try to implement them myself and document the process. Many articles on this site are based on real projects, experiments, and concepts I’ve explored firsthand.

Built a TCP Load Balancer in C to understand how it actually works.

Sat, 06 Jun 2026 05:55:24 +0000

A lot of engineers use load balancers every day. But most of them never go deep enough to understand how it works. I did, and I implemented a Layer 4 load balancer in C to understand how it works. It is not a production-grade solution but is enough to give a core conceptual idea of modern load balancers.

Why I built it?

I’m a great fan of a quote by Nobel Prize-winning physicist Richard Feynman, and it goes like this: “If I can’t build it, I don’t understand it. “That’s why I try to build anything I find interesting.

Kafka Fundamentals - Guide to Distributed Messaging

Fri, 17 Apr 2026 03:54:35 +0000

Why Kafka Exists?

You are building a backend system with multiple services (auth, payments, notifications, analytics). Each service needs to communicate with other services. Issues come when Service A needs to call Service B, and if Service B is down, Service A also fails. This type of communication increases system complexity and scaling problems. This is why Kafka is built.

Kafka introduces a new paradigm; instead of sending requests and waiting for responses, Kafka lets you treat data as a continuous stream of events. You publish events, allowing others to consume them independently.

Understanding gRPC architecture in simple terms

Wed, 15 Apr 2026 06:23:48 +0000

Most tutorials explain how to write a .proto file and call a gRPC method. But very few explain what actually happens after that call.

I recently implemented gRPC in one of my projects and while learning it, I tried to understand its architecture from first principles — not just how to use it, but why it is designed this way and why it is fast.

This post is a simple breakdown of that.

HTTP Under the Hood: Here's What Actually Happens

Fri, 27 Mar 2026 07:05:02 +0000

Most backend engineers think that HTTP is a stateless request-response protocol. That’s technically correct but not complete. For the last few days I have been reading a lot about HTTP in depth. I’m going to talk about same in this post.

In This Post

What is HTTP?
What is a protocol?
What happens when you browser any website?
How is this HTTP structured
How the server actually reads HTTP request
HTTP 1.0 vs HTTP 1.1 vs HTTP 2.x
Compression
HTTP Connection Management
HTTP Caching

HTTP is a protocol that allows us to transfer files and documents over the internet between a server and client (browser). It follows a request-response model in which your browser sends a request to the server and receives a response. This response is then displayed on your browser screen. It can be a webpage, image and even videos. This is what most better engineers know, but do you really know what happens under the hood? Let’s first start with protocol.

Building a Kafka-style commit log from scratch.

Fri, 20 Mar 2026 08:52:57 +0000

Recently I was learning how Kafka works internally. So in order to understand it, I built a commit log system from scratch in Golang.

Some people might think that Kafka is a message queue and not related to logs. But in reality Kafka is an append-only log system. I’ve discussed this more in this post.

In this post we will implement the log system discussed in the previous post from scratch.

Why Kafka is so fast?

Wed, 11 Mar 2026 17:00:09 +0000

Recently I was exploring RabbitMQ and Kafka to do some POCs. I found that Kafka provides very high throughput, which refers to the ability to process a large number of messages in a short amount of time, as compared to RabbitMQ (not RabbitMQ streams). I started exploring why Kafka is so fast.

The main reason behind why Kafka is so fast is the way it stores data on disk and some operating system optimization techniques that it uses.

Things I miss about Spring Boot after switching to Go

Thu, 05 Mar 2026 06:15:07 +0000

I wrote production systems for a startup in Java & Spring Boot for 1.5 years before switching to Golang. That was a high-paced startup, so we were shipping features every week. There I got my hands dirty on some absolute backend technologies that are powering business-orientated backends, such as Java, Spring Boot, MySQL, Redis and Kafka.

Recently I made a switch to Golang because I wanted to move forward from just writing APIs & microservices. Go has been fantastic to work with, but moving away from the Spring ecosystem made me appreciate several things Spring Boot gets right.

Understanding RabbitMQ in simple terms

Sat, 28 Feb 2026 10:47:37 +0000

Hi, I hope you all are doing well. Recently I was exploring RabbitMQ, and I found it fascinating. Previously I’ve used Kafka. RabbitMQ is very different from Kafka. This article is mostly useful for beginners or people who haven’t used RabbitMQ. If you are an experienced developer, you might not find anything new in this post.

What is RabbitMQ?

RabbitMQ (Rabbit Message Queueing) is an Open Source message broker. It is used by applications to interact asynchronously. Simplest use case of RabbitMQ can be establishing a communication between multiple micro-services.

Write your own Shell (Terminal) from scratch.

Sun, 22 Feb 2026 07:27:41 +0000

Hi, I am on my way to becoming a better engineer. So I am building stuff that people don’t build and learn outside their job. I am currently learning about operating systems and how they work. But instead of following the old textbook reading approach, I am doing this by building some projects along the way. In order to understand processes, I have decided to build a shell from scratch. This is part one of this series, and there will be 2 parts. We will be building a full-function shell from scratch.

nil in Go Is Not What You Think

Thu, 12 Feb 2026 17:54:05 +0000

Recently I was watching a GopherCon talk on YouTube that was published in 2016. The speaker spoke about “nil”. After watching that 30 minutes talk, my views about Nil are completely changed. In this post we will discuss exactly the same things that he talked about. This post is not going to be a very long post, so it will be easy for you to get the key points from that talk. But I will highly recommend you invest your 30 minutes in watching that. Here is the video.

How Computers Work: Explained from First Principles

Mon, 02 Feb 2026 03:47:14 +0000

If you are following my blog, you might know that my plan for this year is to learn and program low-level systems. I wanted to develop an operating system since I was in college. I never got enough time back then. But now I have plenty of time to work on skills other than my job.

I spent last month learning about how a computer is basically made and how a CPU performed all these magical tasks from the absolute beginning.

How Uber Shows Millions of Drivers Locations In Realtime | EP: 4 Behind The Screen

Sat, 03 Jan 2026 07:02:59 +0000

This is the 4th episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way.

Here are the other episodes of this series in case if you find something interesting.

EP: 1 How Email Actually Works
EP: 2 How Search Engines Explore the Entire Internet?
EP: 3 How Rate Limiter protect your system from abuse?

Recently I was reading how Uber’s backend system handles so many real-time location events, and I found that the system which handles it is quite interesting. So I’m talking about my learnings in this blog post. All the information provided in this post is taken from the Uber Engineering Blog.

How Rate Limiter protect your system from abuse! EP: 3 Behind The Screen

Sun, 28 Dec 2025 15:04:32 +0000

This is the 3rd episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way.

Here are the other episodes of this series in case if you find something interesting.

EP: 1 How Email Actually Works
EP: 2 How Search Engines Explore the Entire Internet?

In this episode we are going to discuss:

What are rate limiters.?
What problem the solve?
How they work?
Rate Limiting Algorithms.

IMPORTANT - This post is for beginners who want to understand rate limiters. If you are an experienced developer and already know about rate limiters, this post is not for you. You might find my two other posts interesting.

How Search Engines finds your website - Web Crawlers?

Sat, 27 Dec 2025 08:18:24 +0000

This is the 2nd episode of my series Behind The Screen, where I’m discussing the workings of tech that enhance our daily life in the simplest way.

Here are the other episodes of this series in case if you find something interesting.

EP: 1 How Email Actually Works

Web crawlers are software that only a few people know about, but they play a major role in today’s tech. Without web crawlers, our search engines will not be able to show us search results. In this post we will discuss how web crawlers work by learning the core idea behind them.

How Email Actually Works

Thu, 25 Dec 2025 06:38:45 +0000

Email is a technology that you use every single day. It is actually very interesting how emails are handled behind the scenes and reach the recipient’s mailbox. In this post I’ll explain it in a simple way.

Basic Terminologies

These are some terms that you must know before reading this post.

SMTP: Simple Mail Transfer Protocol. A TCP-based communication pattern used to receive and deliver mail.
Mail Server: A server-side application that receives emails, stores and verifies them and then delivers them to the recipient system.
MX Record: It is a DNS record that contains the IP address of the mail server.
MTA: Mail Transfer Agent is a server component that is responsible for transferring email from one server to another (for example, from Gmail Server to Yahoo Server).
DKIM (Domain Keys Identified Mail) : A concept for authentication.
DMARC: Domain-based Message Authentication, Reporting, and Conformance. Also used for authentication.
SPF: Sender Policy Framework. Also an email authentication method.

It’s ok if you did not understand any terminology completely. They are discussed in depth later in this post.

CS Books I'll be reading in 2026.

Mon, 22 Dec 2025 10:59:06 +0000

Hi, 2025 has come to an end. It was a wonderful year for me, and I hope it was for you too. I worked with different tech stacks, including Golang, Java, Spring Boot, Linux, MySQL, Cassandra, ClickHouse, Kafka and encryption algorithms.

But one thing that bothers me is “we work with computers on a high level” and we really don’t know how a computer works, how memory works and all that stuff. That’s why I also started learning some foundational computer science topics in the last 2 months.

Convert VIM to Code Editor in 8 Easy Steps - Beginner Friendly

Tue, 21 Oct 2025 17:20:57 +0000

In this post we are going to convert fresh Neovim Setup to a complete Code Editor from scratch and this post is completely beginner friendly. After reading this post you will be able to customize Neovim according to your wishes.

Result

Let’s first see how our final Neovim setup will look like.

Features Included:

Auto Completion
Code Highlighting
Themes
Error Lens (like VS Code)
LazyGit Integration
Language Server Protocol for any language
Advance Searching using Telescope
Terminal Integration

Config Source Code:

Why Cassandra can handle massive writes without bottlenecks

Tue, 14 Oct 2025 06:45:56 +0000

Hi,

I’ve been reading about Apache Cassandra since the last 2 weeks. I spent time reading a lot of online blogs and then ended up reading Cassandra: The Definitive Guide. In this post we will discuss what makes Cassandra so much faster, more scalable and how it can handle massive writes without bottlenecks.

Before that we need to understand What Cassandra really is and how is it different from other databases.

1 Billion DB Rows Update Challenge

Sun, 06 Jul 2025 19:14:16 +0000

Recently I came across a fascinating engineering challenge. The challenge is about adding a new column to a table and updating its value. But the main catch here is the table contains 1 billion records. I tried Googling this for a solution, but it does not go as a satisfactory solution. So I have decided to write this post and explain my approach and ask for your opinions regarding improvements and drawbacks to my approach.

Let's Write a Threaded File Compression Tool with Memory Control

Sun, 22 Jun 2025 16:32:26 +0000

This weekend I was diving deep into concurrent programming to learn that, I decided to write a threaded file compressor, which will not only compress files concurrently but also limit the memory usage. I have implemented this in GoLang, but these concepts can be implemented in any language.

What I’ve Developed?

I designed a tool that reads files in chunks and limits the overall memory used by introducing a memory-aware semaphore system. It also uses a pool of reusable buffers to make things even faster.

Writing Load Balancer From Scratch In 250 Line of Code

Sun, 15 Jun 2025 05:51:42 +0000

Hey, everyone. It’s another weekend, and I was exploring what to build. So I decided to build a simple yet completely functional load balancer. Let’s discuss it in this post.

What is a load balancer?

A load balancer is a component in a system that distributes requests evenly between servers. It is used in large-scale systems where a large number of requests need to be handled effectively.

In large applications several instances of the same application are deployed to provide horizontal scaling. In order to understand how a load balancer and multiple instances of an application can help in scalability, let’s do some maths.

Let's Write a JSON Parser From Scratch

Sun, 01 Jun 2025 06:43:44 +0000

Hi,

It’s been a long time since I wrote something in this newsletter. Recently I was learning about language parsing and abstract syntax trees. After getting some knowledge about this, I decided to write a JSON parser from scratch.

Parsing:

It is the process of analysing the structure of a string (basically any programming language syntax). Parsing helps us to determine the meaning of the text. Writing a parser for a programming language is a very complex task because programming languages generally have a lot of keywords and syntax rules. Handling all those syntax and keywords can be overwhelming and highly difficult. But in the case of JSON we have a very limited number of keywords and syntax rules. So writing a JSON parser is a relatively easier task.

8 Use Cases of Redis Beyond Key Value Store

Sat, 05 Apr 2025 12:00:02 +0000

Redis is not just a key-value store; it is far beyond that. In this post we are going to discuss the same. Before discovering these use cases, I was using Redis just for simple key-value storage and as a database for my open-source rate limiter.

API Caching Layer

We can use Redis as a caching layer for your API to make them fast. If there is an API that return lot of static data that does not get updated regularly, we can introduce Redis caching layer in it.

Let's Understand & Implement Consistent Hashing

Thu, 27 Feb 2025 01:59:55 +0000

Recently I’ve been learning about distributed systems and I came across a very interesting concept “Consistent Hashing”. It is one of the most basic thing in Distributed Systems.

What is Consistent Hashing?

Consistent hashing is a key distribution technique that ensures easy and smooth mapping of keys to servers to minimize data movement when nodes are added or removed. Unlike traditional hashing methods, where adding or removing a server changes the hash distribution significantly, consistent hashing reduces this impact.

I Made a Configurable Rate Limiter… Because APIs Can’t Say ‘Chill’

Sat, 22 Feb 2025 06:22:01 +0000

I made a configurable rate limiter. In this post I am going to talk about that. We will discuss its architecture and how it works.

Rate Limiter

Rate limiting is a technique that controls the number of requests that can be made to an API within a specific interval of time. Although it is not a very new concept, it has been used for a long time. But I haven’t made a traditional rate limiter; instead, I made a configurable rate limit.