I Made a Configurable Rate Limiter… Because APIs Can’t Say ‘Chill’

I made a configurable rate limiter. In this post I am going to talk about that. We will discuss its architecture and how it works.

Rate Limiter

Rate limiting is a technique that controls the number of requests that can be made to an API within a specific interval of time. Although it is not a very new concept, it has been used for a long time. But I haven’t made a traditional rate limiter; instead, I made a configurable rate limit.

Configurable Rate Limiter

A configurable rate limiter performs the tasks of a traditional rate limiter while also allowing different rate-limiting rules for different APIs. For example, one API can allow 1000 requests per minute while another allows 100 requests per minute based on predefined rules.

Services

My implementation of this rate limit contains several services:

  1. Limiter Service: It coordinates with the client and applies the appropriate rate-limiting algorithm.

  2. Token Bucket: A simple and minimalistic rate limiting algorithm.

  3. Fixed Window Counter: A fixed-window rate-limiting algorithm.

  4. Sliding Window Counter: An algorithm with all the improvements that Fixed Window Counter lacks.

  5. Limiting Rules: A set of rate limiting rules.

  6. Redis: The main database for the implementation of this whole architecture.

  7. Slack: Notification platform to get notified when something wrong.

So after reading a small summary of all the services, you should look at the below flow architecture, which will make things clearer for you.

rate limit arch

Rules:

Firstly, I am storing rate-limiting rules in Redis. A rule structure looks like this. It tells us which strategy to use for an endpoint and contains additional configuration for that strategy.

 1{
 2  "strategy": "TOKEN_BUCKET",
 3  "endpoint": "/api/v1/resource",
 4  "http_method": "GET",
 5  "allow_on_error": true,
 6  "token_bucket_rule": {
 7    "bucket_capacity": 1000,
 8    "token_add_rate": 10
 9  }
10}

In our case of the token bucket algorithm/strategy we require capacity of bucket and token add rate. Capacity tells us the maximum number of tokens the bucket can hold, while the refill rate determines how quickly new tokens are added. The exact units depend on how the rule is configured.

Another example with sliding window rate limiting strategy may look like this.

 1{
 2  "strategy": "SLIDING_WINDOW",
 3  "endpoint": "/api/v1/resource",
 4  "http_method": "GET",
 5  "allow_on_error": true,
 6  "sliding_window_counter_rule": {
 7    "max_requests": 100,
 8    "window": 60
 9  }
10}

Limiter Service

The Limiter Service is the central part of this architecture. It gets the IP address and endpoint from the client, fetches the appropriate rule, and applies the configured rate-limiting logic. rate limit arch

Flow

The Limiter Service receives the IP address and endpoint, then it fetches the rule from the database. Let’s suppose the rule includes a token bucket algorithm.

  1. If there is no existing bucket, the implementation creates the bucket state in Redis. The TTL is used to clean up inactive state; it is separate from the token refill rate.
  2. If a bucket already exists, the implementation updates its state and consumes one token when the request is allowed.
  3. If there are no tokens available, the request is rejected with HTTP 429 (Too Many Requests).

This describes the flow at a high level; the exact implementation differs for the other algorithms. The source code contains the details of those implementations.

Database & Storage

I am using a combination of Redis single node and cluster. These two serve different purposes.

  1. Redis Single Node: A single node contains the rate-limiting rules. Because this data set is small, I do not need a cluster for this part of the implementation.
  2. Redis Cluster: The runtime state can grow with the number of users, so I store token-bucket, sliding-window, and fixed-window state in a Redis Cluster. This lets the state scale horizontally as the workload grows.

Optimizations

There is one optimization that we can do in this architecture. If you have another optimization idea, feel free to leave a comment.

  • Why access Redis every time for rules? As we know, rules are something that won’t be updated very regularly. They will be updated whenever we want, so why access them every time from Redis? So what I have done is I have implemented a function that globally stores rules in a map whenever my application starts. This way I do not need to go to Redis every time for rules; instead, I can look into that in-memory map.
  • How to update local rules map automatically? There is no automatic mechanism for updating these rules locally. One option is Redis Pub/Sub: publish an invalidation message when rules change, subscribe to that channel in the application, and refresh the local rules map when a message arrives.

Frontend

I have also created a front-end application that you can use to modify rules. rate limit arch rate limit arch

Source Code

Here is the source code for this configurable rate limit in case you want to go through it.

Rate Shield

Limiter Algorithms Implementation

In a future post I can show the implementation of the Token Bucket, Fixed Window Counter, and Sliding Window Counter algorithms with code examples.

Thank You

I am always eager to learn from my mistakes so if you think that there is a mistake or improvement in this architecture please comment it down. Thank you, and have a nice day.