How Rate Limiters Protect Your System From Abuse | EP: 3 Behind The Screen
This is the 3rd episode of my series Behind The Screen, where I explain how the technology behind everyday products works, in the simplest way I can.
In This Episode We Will Discuss:
- What are rate limiters?
- What problem do they solve?
- How do they work?
- Rate Limiting Algorithms.
IMPORTANT - This post is for beginners who want to understand rate limiters. If you are an experienced developer and already know about rate limiters, this post is not for you. You might find my two other posts interesting.
What Is a Rate Limiter?
A rate limiter is a backend system component that controls how often a client can access your service/system.
In some apps that require OTP verification, you might have seen that you can request OTP every 1 minute or whatever interval is set by the developer. This is done to prevent abuse of the system. Sending OTP messages can be an expensive operation, so limiting how often they can be requested helps control abuse and cost.
This blocking is done by a rate limiter.
What Problems Do Rate Limiters Solve?
The answer is simple, and you might have guessed it. They help protect your system from abuse and can also reduce operational cost. Surely you won’t want the user to request an OTP every second because that’s expensive for you.
How Do Rate Limiters Work?
A rate limiter keeps track of request activity and applies a policy that determines whether a new request should be allowed. For example, let’s say you have an API endpoint:
1POST - /auth/otp/
This API is integrated into your mobile app and is responsible for sending OTP to the user.
The rate limiter sits in the request path and checks whether the request is allowed. If it is allowed, the request continues; otherwise, the service can return HTTP 429 (Too Many Requests).
But How Does the Rate Limiter Decide?
The rate limiter applies a configured policy, such as a maximum number of requests during a time interval. It keeps enough state to determine whether the current request stays within that policy.
Which Algorithms Do Rate Limiters Use?
Three commonly discussed rate-limiting algorithms are:
- Token Bucket
- Fixed Window
- Sliding Window
Let’s discuss each of them.
Token Bucket
A token bucket represents a bucket with a maximum capacity of tokens. A request consumes a token, while tokens are added over time according to a configured refill rate.
If requests arrive faster than tokens are replenished, the available tokens eventually reach zero. Once the bucket is empty, the rate limiter rejects requests until enough tokens are available again.
In the illustration, the bucket starts with 60 tokens. During the next 23 seconds the user accesses the API 40 times, leaving 20 tokens.
By 35 seconds, the user has consumed all the available tokens.
In this simplified example, the bucket is refilled with 60 tokens every 60 seconds. A conventional token bucket is usually described using a continuous refill rate, with the bucket capped at a maximum capacity. This means that from 12:21:35 to 12:21:59, the user can’t access the API/resource.
At 12:22:00 the bucket will receive another refill, allowing the user to make more requests.
Disadvantages
This simple model also has trade-offs.
- State Management: A distributed rate limiter still needs to maintain state such as token counts and timestamps. At large scale, that state must be stored and updated efficiently; token buckets themselves are not inherently unscalable.
- Multiple policies: Complex policies such as 10 requests per second plus 1000 requests per day may require multiple limits or additional policy logic.
- Burst behavior depends on the bucket capacity and refill model. A token bucket can intentionally allow bursts up to its available capacity; in this example the capacity is 60, so it cannot allow 200 requests unless the configuration is changed.
Fixed Window
This algorithm divides time into fixed windows. If you see the above diagram, time is divided into windows of 1 minute. This algorithm says that there are at most N requests in a fixed time window of length T.
In our example, 12:36 to 12:37 is one window. We count requests during that window, and once the limit is reached, additional requests are rejected. At 12:37:01, a new window begins and its counter starts from zero.
Advantages
- Less Storage Overhead: A fixed-window counter can be compact and efficient to store, although distributed implementations still need to handle concurrency and state correctly.
Disadvantages
- Boundary bursts: A client can use its full allowance near the end of one window and then another full allowance immediately after the next window begins.
Sliding Window
This algorithm says that at most N requests in the last T seconds are allowed — continuously, not in fixed buckets. So instead of checking a fixed window of time, we check the last T seconds from the time the request is made.
For example, if a request is made at 12:36:24, then we will check how many requests the user has made from 12:35:24 if our interval (T) is set to 60 seconds. This reduces the fixed-window boundary problem, although it does not eliminate every possible burst or abuse pattern.