Merge pull request 'nginx configuration for rate limiting crawlers' (#9) from earl-warren/documentation:wip-rate into main

Reviewed-on: https://code.forgejo.org/infrastructure/documentation/pulls/9

README.md

@@ -101,6 +101,31 @@ Forwarding TCP streams (useful for ssh) requires installing the module:
 sudo apt-get install libnginx-mod-stream
 ```
 
+Rate limiting crawlers is done by adding the following to `/etc/nginx/conf.d/limit.conf`:
+
+```
+# http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
+# https://blog.nginx.org/blog/rate-limiting-nginx
+map $http_user_agent $isbot_ua {
+        default 0;
+        ~*(GoogleBot|GoogleOther|bingbot|YandexBot) 1;
+}
+map $isbot_ua $limit_bot {
+        0       "";
+        1       $binary_remote_addr;
+}
+limit_req_zone $limit_bot zone=bots:10m rate=1r/m;
+limit_req_status 429;
+```
+
+and the following in the location to be rate limited:
+
+```
+    location / {
+        limit_req zone=bots burst=2 nodelay;
+		...
+```
+
 ## Host wakeup-on-logs
 
 https://code.forgejo.org/infrastructure/wakeup-on-logs