Rate Limiting with Nginx Behind AWS ELB

By | 2014/11/20

To prevent abuse or limit the number of requests to a site or url, nginx works great. However behind an AWS ELB, one small change is needed to make this work. Check it out!


The nginx docs for reference are here:

http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

Instead of using $binary_remote_addr as in those docs, use $http_x_forwarded_for when behind an ELB.

limit_req_zone $http_x_forwarded_for zone=search:10m rate=5r/s;


Below is an example snipit to limit requests on the url /search.

For the url I want to rate limit (/search in this example) I copied my entire location block and added the limit_req line. This way persons hitting /search will be limited and temporarily see an HTTP 503 until their attempts slow down or stop.


limit_req_zone $http_x_forwarded_for zone=search:10m rate=5r/s;


server {

...

    location /search {
    
        limit_req zone=search burst=10;

        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Protocol https;
        proxy_read_timeout 90;
    }

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $my_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Protocol https;
        proxy_read_timeout 90;
    }

...

}


When rate limited, the nginx error logs will produce output similar to the following:

2014/11/20 17:28:46 [error] 30347#0: *55 limiting requests, excess: 5.658 by zone "search", client: 10.170.2.23, server: www.example.com, request: "GET /search/results/?keyword= HTTP/1.1", host: "test-site-www-2014112001.example.com", referrer: "https://test-site-www-2014112001.example.com/search/results/?keyword=" 
2014/11/20 17:28:46 [error] 30347#0: *55 limiting requests, excess: 5.273 by zone "search", client: 10.170.2.23, server: www.example.com, request: "GET /search/results/?keyword= HTTP/1.1", host: "test-site-www-2014112001.example.com", referrer: "https://test-site-www-2014112001.example.com/search/results/?keyword=" 
2014/11/20 17:28:47 [error] 30347#0: *55 limiting requests, excess: 5.508 by zone "search", client: 10.170.2.23, server: www.example.com, request: "GET /search/results/?keyword= HTTP/1.1", host: "test-site-www-2014112001.example.com", referrer: "https://test-site-www-2014112001.example.com/search/results/?keyword=" 
2014/11/20 17:28:47 [error] 30347#0: *55 limiting requests, excess: 5.200 by zone "search", client: 10.170.2.23, server: www.example.com, request: "GET /search/results/?keyword= HTTP/1.1", host: "test-site-www-2014112001.example.com", referrer: "https://test-site-www-2014112001.example.com/search/results/?keyword=" 
2014/11/20 17:28:48 [error] 30347#0: *55 limiting requests, excess: 5.567 by zone "search", client: 10.170.2.23, server: www.example.com, request: "GET /search/results/?keyword= HTTP/1.1", host: "test-site-www-2014112001.example.com", referrer: "https://test-site-www-2014112001.example.com/search/results/?keyword=" 


The default nginx 503 page as seen by the user when rate limited appears as below. You can of course style a custom 503 page like many sites do.

nginx-rate-limit

9 thoughts on “Rate Limiting with Nginx Behind AWS ELB

  1. ttt980

    In my nginx behind AWS ELB, I’m not seeing any differnce on using $binary_remote_addr, or $http_x_forwarded_for.
    The client ip limited, is always ELB private ip (as on your logs). Is this correct?
    I think the idea on using $http_x_forwarded_for on limitinig, is to prevent abuse, so to limit requests form certain real client ip (badbots, crawlers)

    Reply
    1. Scott Miller Post author

      To test it out, hold down F5 in a browser on the page you want to test (if you first set a low value). That should temporarily give the 503 page just for your client IP.

      Other users of the site with different originating IPs will not be effected.

      Reply
  2. todd

    Wouldn’t you want to use $proxy_protocol_addr when behind an ELB?

    Reply
  3. Marty Woodlee

    Do you have multiple nginxes running behind your ELB? (We’re considering doing that where I work mostly just for high availability purposes.) If so, I wonder if you’ve encountered any oddness since each nginx maintains its own memory zone for tracking request rates. I presume that if ELB is doing round-robin balancing, it should roughly work if you set the max rate on each node to [the desired overall max rate / the number of nodes], provided the number of nodes is small. But I’d love to hear any experiences you have with that if any.

    Reply
    1. Scott Miller Post author

      Thanks. Yes I have an autoscaling stack with 2-8 EC2 instances (nginx / gunicorn /django). You are correct the individual nginx instances would each have their own memory for this.

      That does not seem to make any issues for me and all works great.

      Depending on the size of your stack you may wish to alter the rate limiting values.

      Reply
  4. Michael Averto

    Scott, if you are running behind a LB, doesn’t only each machine know about the request sent to it? So essentially you are rate limiting the IP per machine, not as a whole. So if you had a rate limit of 10, and 8 machines running, I could in theory make 80 requests before being rate limited my each machine right?

    With this approach it seems you would slow down someone who is abusing the system as long as the LB sends their request to each server but there is no database to rate limit overall right with NGINX?

    Reply
  5. Umair

    What does 5.200 indicating in following error log?
    2014/11/20 17:28:47 [error] 30347#0: *55 limiting requests, excess: 5.200 by zone “search”, client: 10.170.2.23, server: http://www.example.com, request: “GET /search/results/?keyword= HTTP/1.1”, host: …

    Reply
  6. zeming

    Test Aws-Cn results on this side
    ELB + NGINX + need setup module –with-http_realip_module

    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;

    Or $http_x_forwarded_for get less than real user IP, intelligent access ELB IP network
    Hope to be able to help people have a problem

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.