Distributed Rate Limiting: How to Enforce Fair Usage Across Every Node in Your Cluster Without Sacrificing a Millisecond of Throughput

Quick Answer / TL;DR: Distributed rate limiting solves the fundamental flaw of single-node rate limiters — they only see local traffic. By centralizing quota state in Redis using atomic Lua scripts, sliding window counters, or token bucket algorithms, you can enforce exact per-user, per-tenant, or per-endpoint limits across every pod in your Kubernetes cluster — typically adding less than 3–5ms of overhead per request. This guide walks through the architecture, code, and tradeoffs.

If your API is running on more than one server — and in 2025, virtually every production system is — your rate limiter is probably broken. Not broken in an obvious way. Broken in the way that lets a determined user hammer your backend with 10× their allowed quota simply by distributing their requests across your load balancer. Distributed rate limiting is the engineering discipline that closes this gap. It ensures that a user's quota is tracked globally, not per-node, so no matter which pod handles a request, the limit holds. At Apargo, we've implemented distributed rate limiting across multiple SaaS platforms and within our AI Greentick WhatsApp automation product to protect high-throughput messaging pipelines. Here's exactly how to build it right.

Why Single-Node Rate Limiting Fails in Production

The classic in-memory rate limiter stores a counter in the application process. It's fast, zero-dependency, and completely useless at scale. Here's the math: if you have 10 pods behind a round-robin load balancer and a user is allowed 100 requests per minute, each pod will independently allow 100 requests — meaning the user can actually send 1,000 requests per minute before hitting any wall. That's a 10× amplification of your intended quota.

This isn't a theoretical edge case. It's the default behavior of every popular in-process rate limiting library when deployed without a shared state backend. The problem compounds in autoscaling environments where pod count fluctuates dynamically — your effective rate limit changes every time Kubernetes spins up or tears down an instance.

The Three Failure Modes to Understand

• Quota multiplication: N pods × per-pod limit = N× effective quota per user
• Inconsistent enforcement: Sticky sessions partially fix the problem but break when pods restart or traffic spikes cause rebalancing
• Race conditions: Even with a shared store, non-atomic read-increment-write operations allow burst overflows under high concurrency

The only correct solution is distributed rate limiting with a centralized, atomic state store.

Choosing the Right Algorithm for Distributed Rate Limiting

Before touching any code, you need to pick your algorithm. Each has distinct tradeoffs in precision, burst tolerance, and implementation complexity.

1. Fixed Window Counter

The simplest approach. Increment a counter per time window (e.g., per minute). Reset at window boundaries. It's cheap to implement but suffers from the boundary burst problem — a user can send 100 requests at 11:59:59 and another 100 at 12:00:01, effectively sending 200 requests in 2 seconds while technically staying within quota.

2. Sliding Window Log

Store a timestamp for every request in a sorted set. On each new request, remove entries older than the window, count remaining entries, and reject if over quota. Precise, but memory-intensive at scale — storing per-request timestamps for millions of users is expensive.

3. Sliding Window Counter (Hybrid)

The practical sweet spot. Blend the current and previous fixed window counts using a weighted formula based on how far you are into the current window. Precision within ~0.1% of a true sliding window, with O(1) memory per user. This is what most production systems use.

4. Token Bucket

Each user has a "bucket" that fills at a fixed rate (e.g., 10 tokens/second, max 100 tokens). Each request consumes a token. Allows controlled bursting up to the bucket capacity while enforcing a sustained rate. Ideal for APIs where short bursts are acceptable but sustained abuse must be blocked.

5. Leaky Bucket

Requests queue up and are processed at a fixed rate. Smooths traffic into a constant output rate. Great for downstream systems that can't handle bursts, but adds latency as requests wait in queue.

For most distributed rate limiting scenarios on HTTP APIs, the sliding window counter or token bucket gives the best balance. We'll implement both using Redis atomic operations.

The Redis Architecture Behind Distributed Rate Limiting

Redis is the de facto standard for distributed rate limiting state for good reasons: sub-millisecond read/write latency, native atomic operations via Lua scripting, built-in key expiry, and cluster mode for horizontal scaling. A well-configured Redis instance can handle 500,000+ operations per second on commodity hardware.

Key Design Principles

• Atomicity is non-negotiable: Use Lua scripts or Redis transactions (MULTI/EXEC) to prevent race conditions between read and write operations
• Key namespacing: Structure keys as rl:{tenant_id}:{user_id}:{endpoint}:{window} for granular, multi-dimensional limiting
• TTL management: Always set expiry on rate limit keys to prevent memory leaks from inactive users
• Pipeline batching: For high-frequency endpoints, batch Redis calls to reduce round-trip overhead

Implementing Sliding Window Counter in Redis + Node.js

Here's a production-grade implementation using a Lua script to guarantee atomicity. The Lua script runs entirely server-side in Redis, making the check-and-increment operation truly atomic — no other command can interleave.

-- sliding_window_rate_limit.lua
-- Args: KEYS[1] = current window key, KEYS[2] = previous window key
-- ARGV[1] = limit, ARGV[2] = window size (seconds), ARGV[3] = current timestamp

local current_key = KEYS[1]
local previous_key = KEYS[2]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

-- Calculate how far we are into the current window (0.0 to 1.0)
local window_start = math.floor(now / window) * window
local elapsed = now - window_start
local weight = 1 - (elapsed / window)

-- Get counts from current and previous windows
local current_count = tonumber(redis.call('GET', current_key)) or 0
local previous_count = tonumber(redis.call('GET', previous_key)) or 0

-- Weighted estimate of requests in the sliding window
local estimated = math.floor(previous_count * weight + current_count)

if estimated >= limit then
  -- Rate limit exceeded — return remaining TTL and current count
  local ttl = redis.call('TTL', current_key)
  return {0, estimated, ttl}
end

-- Increment current window counter
local new_count = redis.call('INCR', current_key)

-- Set expiry only on first increment (avoids resetting TTL on each request)
if new_count == 1 then
  redis.call('EXPIRE', current_key, window * 2)
end

return {1, estimated + 1, -1}

// rateLimiter.js — Production Node.js wrapper
import Redis from 'ioredis';
import fs from 'fs';
import path from 'path';

const redis = new Redis({
  host: process.env.REDIS_HOST,
  port: 6379,
  enableReadyCheck: true,
  maxRetriesPerRequest: 2,
  lazyConnect: false,
});

// Load and register the Lua script as a Redis script (cached SHA)
const luaScript = fs.readFileSync(
  path.join(__dirname, 'sliding_window_rate_limit.lua'),
  'utf8'
);

// SHA1 hash of script — Redis caches it server-side
let scriptSha = null;

async function loadScript() {
  scriptSha = await redis.script('LOAD', luaScript);
  console.log(`Rate limit Lua script loaded. SHA: ${scriptSha}`);
}

/**
 * Check and enforce distributed rate limiting.
 *
 * @param {string} identifier  - Unique key (e.g., "user:123:POST:/api/messages")
 * @param {number} limit       - Max requests allowed per window
 * @param {number} windowSecs  - Window size in seconds
 * @returns {{ allowed: boolean, count: number, retryAfter: number }}
 */
async function checkRateLimit(identifier, limit, windowSecs) {
  const now = Math.floor(Date.now() / 1000); // Unix timestamp in seconds
  const windowId = Math.floor(now / windowSecs);

  const currentKey = `rl:${identifier}:${windowId}`;
  const previousKey = `rl:${identifier}:${windowId - 1}`;

  try {
    // EVALSHA uses the cached Lua script — avoids re-sending script on every call
    const [allowed, count, retryAfter] = await redis.evalsha(
      scriptSha,
      2,              // number of KEYS
      currentKey,
      previousKey,
      limit,
      windowSecs,
      now
    );

    return {
      allowed: allowed === 1,
      count: count,
      retryAfter: retryAfter > 0 ? retryAfter : 0,
    };
  } catch (err) {
    // If script was evicted (NOSCRIPT error), reload and retry once
    if (err.message.includes('NOSCRIPT')) {
      await loadScript();
      return checkRateLimit(identifier, limit, windowSecs);
    }
    // Fail open on Redis errors — never block legitimate traffic due to infra issues
    console.error('Rate limiter Redis error:', err.message);
    return { allowed: true, count: 0, retryAfter: 0 };
  }
}

// Express middleware factory
export function rateLimitMiddleware({ limit = 100, windowSecs = 60, keyFn }) {
  return async (req, res, next) => {
    const identifier = keyFn ? keyFn(req) : `ip:${req.ip}`;
    const startTime = Date.now();

    const result = await checkRateLimit(identifier, limit, windowSecs);

    const latency = Date.now() - startTime;

    // Expose standard rate limit headers
    res.set('X-RateLimit-Limit', limit);
    res.set('X-RateLimit-Remaining', Math.max(0, limit - result.count));
    res.set('X-RateLimit-Latency-Ms', latency); // observability

    if (!result.allowed) {
      res.set('Retry-After', result.retryAfter);
      return res.status(429).json({
        error: 'Too Many Requests',
        retryAfter: result.retryAfter,
      });
    }

    next();
  };
}

// Initialize on startup
loadScript();

In our benchmarks, this implementation adds 2–4ms of latency per request when Redis is co-located in the same availability zone — well within acceptable overhead for API gateways. The EVALSHA approach means the Lua script is cached server-side by its SHA1 hash, eliminating script transmission overhead on every call.

Token Bucket Implementation for Burst-Tolerant Distributed Rate Limiting

For APIs where you want to allow short bursts (e.g., a user can send 20 messages instantly but not sustain more than 5/second), the token bucket algorithm is superior. Here's a Redis-based implementation:

-- token_bucket.lua
-- KEYS[1] = bucket key
-- ARGV[1] = capacity (max tokens), ARGV[2] = refill rate (tokens/sec)
-- ARGV[3] = tokens requested, ARGV[4] = current timestamp (float)

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])
local now = tonumber(ARGV[4])

-- Read current bucket state: {tokens, last_refill_timestamp}
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now

-- Calculate tokens to add based on elapsed time
local elapsed = math.max(0, now - last_refill)
local refill_amount = elapsed * refill_rate
tokens = math.min(capacity, tokens + refill_amount)

if tokens < requested then
  -- Not enough tokens — calculate wait time
  local wait = (requested - tokens) / refill_rate
  return {0, tokens, math.ceil(wait)}
end

-- Consume tokens
tokens = tokens - requested

-- Persist updated bucket state with TTL
redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)

return {1, tokens, 0}

Multi-Dimensional Rate Limiting: Tenant, User, and Endpoint Tiers

Real SaaS platforms need layered distributed rate limiting — not just "100 requests per user per minute" but a hierarchy of quotas:

1. Global cluster limit: Protect your infrastructure from total overload (e.g., 1M req/min across all tenants)
2. Tenant plan limit: Enforce SaaS tier quotas (Starter: 10k/day, Pro: 100k/day, Enterprise: custom)
3. User