Our API Rate Limiter Was Bypassable With X-Forwarded-For

The $40K bot attack that taught me to never trust HTTP headers again.

I’m gonna tell you about the dumbest security hole I ever shipped to production.

Not because I’m proud of it. Because maybe if I embarrass myself publicly, you won’t make the same mistake.

Spoiler: You probably will. We all do.

The Feature Request That Seemed Simple

“We need rate limiting on the API.”

Sure. Easy. I’d done this before. Redis-based rate limiter, check the IP, limit to 100 requests per minute. Standard stuff.

I wrote it in an afternoon:

@Component
public class RateLimiterFilter implements Filter {
    
    @Autowired
    private StringRedisTemplate redis;
    
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                         FilterChain chain) {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        
        // Get client IP
        String clientIp = httpRequest.getRemoteAddr();
        
        // Check rate limit
        String key = "rate_limit:" + clientIp;
        Long requests = redis.opsForValue().increment(key);
        
        if (requests == 1) {
            redis.expire(key, 60, TimeUnit.SECONDS);
        }
        
        if (requests > 100) {
            ((HttpServletResponse) response).setStatus(429);
            return;
        }
        
        chain.doFilter(request, response);
    }
}
```

Clean. Simple. Deployed on Friday at 3 PM.
Friday at 3 PM. Remember that.
## Friday Night: Everything's Fine
Weekend came. I cracked a beer. Watched some Netflix. Life was good.
My rate limiter was protecting our API from abuse. Users were happy. Product manager was happy. I was happy.
Then Monday morning happened.
## Monday 9:47 AM: The Alerts Start
Slack notification: "API credits depleted."
Wait, what?
We had 100,000 API credits for our third-party service. They cost us $0.40 each. Should last us the whole month.
They were gone. All of them. In one weekend.
I checked the logs. Someone made 100,000 API calls. In 48 hours.
My rate limiter should've stopped this after 100 requests. What the hell happened?
## The $40,000 Lesson
I SSH'd into the server. Checked Redis. 
Found entries like:
```
rate_limit:10.0.0.5 = 87
rate_limit:10.0.0.5 = 92
rate_limit:10.0.0.5 = 95
```
Wait. The rate limiter was working. It was tracking requests. So how did someone make 100,000 calls?
Then I saw it in the access logs:
```
10.0.0.5 - - [GET /api/data] X-Forwarded-For: 1.2.3.4
10.0.0.5 - - [GET /api/data] X-Forwarded-For: 1.2.3.5
10.0.0.5 - - [GET /api/data] X-Forwarded-For: 1.2.3.6
10.0.0.5 - - [GET /api/data] X-Forwarded-For: 1.2.3.7

Same IP (10.0.0.5 – our load balancer). Different X-Forwarded-For headers.

Someone was just… changing the header.

And my brilliant rate limiter was only checking request.getRemoteAddr(), which was always the load balancer’s IP.

They bypassed the entire thing by adding a random X-Forwarded-For header to each request.

Cost: $40,000 in API credits.

Time to exploit: Probably 10 minutes once they figured it out.

My ego: Destroyed.

The Autopsy: How I Fucked Up

Let me break down exactly what went wrong, because this is embarrassingly common.

Mistake #1: Trusting the wrong IP

String clientIp = httpRequest.getRemoteAddr();

In development? This works. You hit the API directly, you get the real client IP.

In production? Everything goes through a load balancer. getRemoteAddr() returns the load balancer’s IP. Always. For every request.

The actual client IP is in the X-Forwarded-For header. But here’s the thing…

Mistake #2: Not understanding how X-Forwarded-For works

X-Forwarded-For is just an HTTP header. Anyone can set it to anything.

curl -H "X-Forwarded-For: 8.8.8.8" https://api.example.com

Boom. Now my API thinks the request came from Google’s DNS server.

The attacker’s script probably looked like this:

import requests
import random

for i in range(100000):
    fake_ip = f"{random.randint(1,255)}.{random.randint(1,255)}.{random.randint(1,255)}.{random.randint(1,255)}"
    headers = {"X-Forwarded-For": fake_ip}
    requests.get("https://api.example.com/data", headers=headers)

Every request got a “different IP.” My rate limiter saw 100,000 different clients. Each one got their own 100 requests.

Game over.

Mistake #3: Not testing behind a real load balancer

In my local environment, getRemoteAddr() worked fine. I tested it. It passed.

But I tested it hitting the app directly. Not through Nginx. Not through CloudFlare. Not through anything that would actually be in production.

Classic “works on my machine” bullshit.

The Fix (That Should’ve Been Obvious)

First, I had to actually get the real client IP. Behind a load balancer, it’s complicated:

private String getClientIp(HttpServletRequest request) {
    // Try X-Forwarded-For first
    String xForwardedFor = request.getHeader("X-Forwarded-For");
    
    if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
        // X-Forwarded-For can be: "client, proxy1, proxy2"
        // We want the leftmost (original client) IP
        String[] ips = xForwardedFor.split(",");
        return ips[0].trim();
    }
    
    // Fallback to remote addr (for direct connections)
    return request.getRemoteAddr();
}

But wait. This is still vulnerable. An attacker can still set X-Forwarded-For to whatever they want.

The real fix depends on your infrastructure:

If you control the load balancer (AWS ALB, Nginx, etc):

Configure it to overwrite the X-Forwarded-For header with the actual client IP. Don’t append, replace.

Nginx example:

proxy_set_header X-Forwarded-For $remote_addr;

Now you can trust X-Forwarded-For because your infrastructure set it, not the client.

If you’re behind CloudFlare:

Use CF-Connecting-IP instead. CloudFlare sets this header and the client can’t fake it.

String clientIp = request.getHeader("CF-Connecting-IP");
if (clientIp == null) {
    clientIp = getClientIp(request); // fallback
}

If you don’t control anything:

You’re kind of screwed. You have to trust headers you can’t verify. Your best bet:

  1. Use multiple headers as a fingerprint
  2. Add additional rate limiting by API key
  3. Require authentication
  4. Accept that IP-based rate limiting behind unknown proxies is fundamentally broken

Here’s what I actually deployed:

@Component
public class RateLimiterFilter implements Filter {
    
    @Autowired
    private StringRedisTemplate redis;
    
    private static final List<String> TRUSTED_PROXIES = Arrays.asList(
        "10.0.0.0/8",  // Our internal network
        "172.16.0.0/12" // AWS VPC range
    );
    
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                         FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        
        String clientIp = extractClientIp(httpRequest);
        String fingerprint = createFingerprint(httpRequest, clientIp);
        
        String key = "rate_limit:" + fingerprint;
        Long requests = redis.opsForValue().increment(key);
        
        if (requests == 1) {
            redis.expire(key, 60, TimeUnit.SECONDS);
        }
        
        if (requests > 100) {
            logSuspiciousActivity(httpRequest, clientIp, requests);
            ((HttpServletResponse) response).setStatus(429);
            ((HttpServletResponse) response).getWriter()
                .write("{"error":"Rate limit exceeded"}");
            return;
        }
        
        chain.doFilter(request, response);
    }
    
    private String extractClientIp(HttpServletRequest request) {
        // Try CloudFlare header first (can't be spoofed)
        String cfIp = request.getHeader("CF-Connecting-IP");
        if (cfIp != null && !cfIp.isEmpty()) {
            return cfIp;
        }
        
        // Try X-Real-IP (set by our Nginx)
        String realIp = request.getHeader("X-Real-IP");
        if (realIp != null && !realIp.isEmpty()) {
            return realIp;
        }
        
        // Parse X-Forwarded-For carefully
        String forwardedFor = request.getHeader("X-Forwarded-For");
        if (forwardedFor != null && !forwardedFor.isEmpty()) {
            // Take the first IP, but validate it's not from our trusted proxies
            String[] ips = forwardedFor.split(",");
            for (String ip : ips) {
                String cleanIp = ip.trim();
                if (!isTrustedProxy(cleanIp)) {
                    return cleanIp;
                }
            }
        }
        
        // Last resort
        return request.getRemoteAddr();
    }
    
    private String createFingerprint(HttpServletRequest request, String ip) {
        // Don't rely solely on IP - create a composite fingerprint
        StringBuilder fingerprint = new StringBuilder(ip);
        
        // Add User-Agent (can be spoofed but adds friction)
        String userAgent = request.getHeader("User-Agent");
        if (userAgent != null) {
            fingerprint.append(":").append(userAgent.hashCode());
        }
        
        // If there's an API key, use that instead
        String apiKey = request.getHeader("X-API-Key");
        if (apiKey != null && !apiKey.isEmpty()) {
            return "api_key:" + apiKey;
        }
        
        return fingerprint.toString();
    }
    
    private boolean isTrustedProxy(String ip) {
        // Check if IP is in our trusted proxy ranges
        // (Actual implementation would use proper CIDR matching)
        return ip.startsWith("10.") || ip.startsWith("172.16.");
    }
    
    private void logSuspiciousActivity(HttpServletRequest request, 
                                      String ip, Long requests) {
        log.warn("Rate limit exceeded - IP: {}, Requests: {}, User-Agent: {}, Path: {}",
                ip, requests,
                request.getHeader("User-Agent"),
                request.getRequestURI());
    }
}

Is it perfect? No. Can determined attackers still get around it? Probably. But it’s 100x better than what I had.

When I was debugging this disaster at 2 AM, I wished I had a runbook. Something that said “here’s how headers work behind proxies, here’s what to check, here’s how to fix it.”

I built that runbook after this incident:

👉 On-Call Survival Kit for Backend Engineers — How to debug production fires when you’re half-asleep

The Other Ways People Bypass Rate Limiters

After this incident, I started researching. Turns out, IP-based rate limiting is hilariously easy to bypass:

Method 1: Rotating IPs

AWS has unlimited IP addresses. Spin up 1000 EC2 instances with different IPs. Boom, 100,000 requests.

Cost to attacker: ~$50 Cost to you: Whatever damage 100,000 requests cause

Method 2: Residential Proxies

Services sell access to millions of real residential IPs. Rotating proxies make every request look like a different person.

Cost: $500/month for unlimited requests Your IP-based rate limiter: Useless

Method 3: Distributed Attack

Compromise 1000 IoT devices (security cameras, routers, etc). Each makes 100 requests. You can’t block them all without blocking legitimate users.

This is basically a DDoS, but targeted at your API credits instead of your uptime.

Method 4: CloudFlare Workers

CloudFlare Workers run on CloudFlare’s IP ranges. If you whitelist CloudFlare (which many APIs do), attackers can abuse Workers to bypass your limits.

Method 5: Just Ask Nicely

If your rate limiter returns a helpful error like:

{
  "error": "Rate limit exceeded. Try again in 42 seconds",
  "retry_after": 42
}

Congratulations, you just told the attacker exactly how long to wait. They’ll just add a 42-second delay and continue.

The Real Solution: Defense in Depth

Here’s what I learned: you can’t rely on one thing.

IP-based rate limiting is layer 1. You need layers 2, 3, 4, and 5:

Layer 1: IP-based rate limiting (what we’ve been discussing)

  • Helps with accidental abuse
  • Stops unsophisticated attacks
  • Easy to bypass

Layer 2: API key rate limiting

String apiKey = request.getHeader("X-API-Key");
String key = "rate_limit:api_key:" + apiKey;
// Rate limit per API key, not per IP

Much harder to bypass. Attackers would need multiple valid API keys.

Layer 3: Endpoint-specific limits

Not all endpoints cost the same. Listing users? Cheap. Generating a PDF report? Expensive.

Map<String, Integer> endpointLimits = Map.of(
    "/api/users", 1000,
    "/api/reports", 10,
    "/api/export", 5
);

int limit = endpointLimits.getOrDefault(request.getRequestURI(), 100);

Layer 4: CAPTCHA for suspicious behavior

If someone hits your limit, don’t just block them. Make them prove they’re human.

if (requests > 100 && requests < 150) {
    response.setStatus(429);
    response.getWriter().write("{"error":"Please solve CAPTCHA"}");
    return;
}

Layer 5: Cost-based throttling

Track actual resource usage, not just request count.

// Charge "credits" based on what the request actually costs
int cost = calculateCost(request);
Long creditsUsed = redis.opsForValue().increment(key, cost);

A request that processes 1000 records costs more than one that returns 10.

At this point, you might be thinking “this is getting complicated.” Yeah. Welcome to production security.

If you want the full playbook on handling these kinds of security incidents:

👉 Production Incident Toolkit — Database & API — Real runbooks for real incidents

The Uncomfortable Questions I Had to Answer

Monday afternoon. Emergency meeting. CTO, product manager, finance person who looked like she wanted to murder me.

“How did this happen?”

I explained the X-Forwarded-For thing. Tried to use technical terms to sound smart. Didn’t work.

“Why didn’t QA catch this?”

QA tested in staging. Staging doesn’t have a load balancer. Everything worked in staging.

Classic staging/production environment mismatch.

“Why did we have so many API credits available?”

We didn’t think anyone would actually burn through them. We were wrong.

“Can we get the money back?”

Nope. The API calls were made. The third-party service processed them. We owe $40,000.

“Can we find who did this?”

All we have are fake IPs and a User-Agent that says “python-requests/2.28.0”. Could be anyone. Could be a researcher testing our API. Could be a competitor. Could be some bored kid.

We’ll never know.

“What’s your plan to make sure this never happens again?”

That’s when I pulled out my hastily written document titled “API Security Improvements” and prayed they wouldn’t fire me on the spot.

The Post-Mortem (That Saved My Job)

I wrote a 5-page post-mortem. Here’s the summary:

What went wrong:

  1. Rate limiting based on untrusted header values
  2. No validation of X-Forwarded-For
  3. No monitoring for unusual API credit usage
  4. No cost-based throttling
  5. Staging environment didn’t match production

Immediate fixes:

  1. Fixed IP extraction to use CloudFlare’s CF-Connecting-IP
  2. Added API key-based rate limiting
  3. Reduced max API credits per hour to 1000
  4. Added alerting when credits hit 50% of hourly limit

Long-term fixes:

  1. Implemented defense-in-depth rate limiting strategy
  2. Added CAPTCHA challenges for suspicious activity
  3. Built proper staging environment with load balancer
  4. Documented how headers work in our infrastructure
  5. Added security review step to deploy checklist

The CTO read it. Nodded. Said “Good. Implement this. Don’t let it happen again.”

I didn’t get fired. But I did learn a $40,000 lesson.

Want to avoid learning these lessons the expensive way? I documented all the database and API failure patterns I’ve seen:

👉 Database Incident Playbook — How Production Databases Actually Fail

What I’d Do Differently (Time Machine Edition)

If I could go back and slap myself before writing that code:

1. Read the CloudFlare docs

They literally have a page titled “Restoring original visitor IP”. I could’ve just followed that.

2. Test with curl and custom headers

# This would've exposed the bug instantly
curl -H "X-Forwarded-For: 1.2.3.4" http://localhost:8080/api/data
curl -H "X-Forwarded-For: 5.6.7.8" http://localhost:8080/api/data
# Check Redis - are these counted as different IPs?

3. Add logging from day one

Log every rate limit decision:

log.info("Rate limit check - IP: {}, Fingerprint: {}, Count: {}/{}", 
         clientIp, fingerprint, requests, limit);

When the attack happened, I had no idea what was going on because I had no logs.

4. Start with API key auth

IP-based rate limiting without authentication is security theater. If requests need to be rate limited, they probably need to be authenticated too.

5. Monitor credit usage

Don’t wait for credits to run out. Alert when usage is abnormal.

// Compare current hour to average of previous hours
long currentHourUsage = getCreditsThisHour();
long avgHourlyUsage = getAvgHourlyUsage();

if (currentHourUsage > avgHourlyUsage * 3) {
    sendAlert("API credit usage 3x higher than normal!");
}

6. Read production horror stories

You know what’s cheaper than a $40K mistake? Reading about someone else’s $40K mistake.

I should’ve Googled “X-Forwarded-For security issues” before deploying. First result would’ve warned me.

The Things Nobody Tells You About Rate Limiting

After fixing this disaster and researching for weeks, here’s what I learned:

Truth #1: Every rate limiting strategy sucks

  • IP-based? Easily bypassed.
  • API key? Users share keys.
  • Cookie-based? Cleared in incognito mode.
  • Device fingerprinting? Privacy nightmare and easily spoofed.
  • CAPTCHA? Farms solve them for $1 per 1000.

There’s no perfect solution. You layer imperfect solutions until it’s annoying enough for attackers to give up.

Truth #2: Rate limiting conflicts with legitimate use cases

Office networks share one IP. What if 100 employees all hit your API at once? You just rate limited an entire company.

Mobile users on carrier-grade NAT share IPs with thousands of people. Rate limit by IP, you block innocent users.

Truth #3: Attackers will find the expensive endpoint

We rate limited all endpoints equally. Then attackers found /api/generate-pdf, which costs 10x more than other endpoints.

They specifically targeted that one. Smart.

Truth #4: Good DDoS protection costs money

CloudFlare’s free tier is amazing. But rate limiting? That’s a paid feature.

AWS WAF? Costs money per rule and per request evaluated.

You can build your own, but then you’re maintaining security infrastructure instead of building features.

Truth #5: Documentation lies

Spring Boot docs say “use request.getRemoteAddr()”. That’s technically correct but practically useless behind a load balancer.

Every framework has tips that work in development and fail in production.

The Security Mindset I Should’ve Had

Before this incident, I thought about features. After, I think about attacks.

Old me: “Users need to call our API. Let’s add rate limiting so the server doesn’t crash.”

New me: “Attackers will try to abuse our API. How would I bypass this rate limiter if I were them?”

The mindset shift:

  • Assume all input is malicious
  • Assume headers can be faked
  • Assume users will do the worst possible thing
  • Assume your code has bugs
  • Assume production is different from staging
  • Assume documentation is wrong

Paranoid? Maybe. But paranoid engineers don’t lose $40,000 on a weekend.

Want to Go Deeper?

If you found this useful, I’ve packaged these lessons into focused bundles:

For building reliable backend systems: 👉 Spring Boot Production Bundle

For leveling up your career: 👉 Senior Engineer Interview & Career Bundle

For my complete production playbook: 👉 Production Engineering Master Bundle — V2

You can find all of them here:

Subscribe to Devrim Ozcay on Gumroad

If you care about writing safer, faster, and more production-ready code — these bundles are built for you.

The Bottom Line

Rate limiting by IP is broken by design.

Not because IP-based identification is bad (though it is). But because the web wasn’t built for it.

We have:

  • Load balancers that hide client IPs
  • Proxies that can fake any IP
  • NAT that puts thousands of users behind one IP
  • Headers that anyone can set to anything

And we’re trying to build security on top of this foundation of lies.

It’s like building a house on quicksand and wondering why it keeps sinking.

The fix isn’t better rate limiting. The fix is assuming rate limiting will fail and building layers of defense.

IP-based rate limiting is your bicycle lock. It stops casual theft. But if someone really wants your bike, they’ll bring bolt cutters.

So you add:

  • A better lock (API keys)
  • A camera (logging)
  • A GPS tracker (monitoring)
  • Insurance (cost limits)

You don’t stop at one lock and call it secure.

One Last Thing

I’m actively talking to teams dealing with:

  • Security holes they didn’t know existed
  • Rate limiting that doesn’t actually limit
  • API abuse burning through budgets
  • Incidents that keep repeating for unclear reasons

If any of this sounds familiar, I’d genuinely love to hear what you’re dealing with.

I’m trying to understand where teams are struggling most so I can build better tools and practices around it.

For deeper technical insights and weekly stories from production:

👉 Subscribe to my Substack:

Devrim’s Engineering Notes | Substack

This article is for educational purposes, sharing real production experiences to help other engineers avoid similar security pitfalls.

Now go check your rate limiter. I’ll wait.

Actually, don’t wait. Check it now. Before someone costs you $40,000.

And if you find X-Forwarded-For in your code… yeah. Fix that.

You’re welcome.

Our API Rate Limiter Was Bypassable With X-Forwarded-For was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post first appeared on Read More

Related Posts