forgejo/infrastructure-documentation
1
0
Fork 0
mirror of https://code.forgejo.org/infrastructure/documentation synced 2024-11-15 01:21:53 +00:00
Code Issues Activity

Merge pull request 'nginx configuration for rate limiting crawlers' (#9) from earl-warren/documentation:wip-rate into main

Browse source 
Reviewed-on: https://code.forgejo.org/infrastructure/documentation/pulls/9
This commit is contained in:
earl-warren 2024-09-19 05:43:12 +00:00
parent 891af11fd5 52d46196dd
commit 743251c733
1 changed files with 25 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
README.md
View file

@ -101,6 +101,31 @@ Forwarding TCP streams (useful for ssh) requires installing the module:
sudo apt-get install libnginx-mod-stream
```
Rate limiting crawlers is done by adding the following to `/etc/nginx/conf.d/limit.conf`:
```
# http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
# https://blog.nginx.org/blog/rate-limiting-nginx
map $http_user_agent $isbot_ua {
default 0;
~*(GoogleBot|GoogleOther|bingbot|YandexBot) 1;
}
map $isbot_ua $limit_bot {
0 "";
1 $binary_remote_addr;
}
limit_req_zone $limit_bot zone=bots:10m rate=1r/m;
limit_req_status 429;
```
and the following in the location to be rate limited:
```
location / {
limit_req zone=bots burst=2 nodelay;
...
```
## Host wakeup-on-logs
https://code.forgejo.org/infrastructure/wakeup-on-logs