forgejo/infrastructure-documentation
1
0
Fork 0
mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-25 20:11:11 +00:00
Code Issues Activity

nginx configuration for rate limiting crawlers

Browse source 
Fixes: #8
This commit is contained in:
Earl Warren 2024-09-18 15:50:50 +02:00
parent 891af11fd5
commit 52d46196dd
No known key found for this signature in database
GPG key ID: 0579CB2928A78A00
1 changed files with 25 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
README.md
View file

@ -101,6 +101,31 @@ Forwarding TCP streams (useful for ssh) requires installing the module:
sudo apt-get install libnginx-mod-stream sudo apt-get install libnginx-mod-stream
``` ```
Rate limiting crawlers is done by adding the following to `/etc/nginx/conf.d/limit.conf`:
```
# http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
# https://blog.nginx.org/blog/rate-limiting-nginx
map $http_user_agent $isbot_ua {
default 0;
~*(GoogleBot|GoogleOther|bingbot|YandexBot) 1;
}
map $isbot_ua $limit_bot {
0 "";
1 $binary_remote_addr;
}
limit_req_zone $limit_bot zone=bots:10m rate=1r/m;
limit_req_status 429;
```
and the following in the location to be rate limited:
```
location / {
limit_req zone=bots burst=2 nodelay;
...
```
## Host wakeup-on-logs ## Host wakeup-on-logs
https://code.forgejo.org/infrastructure/wakeup-on-logs https://code.forgejo.org/infrastructure/wakeup-on-logs