teutat3s/elk
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

fix: disallow all indexing on canary site and indexing of auth-only pages in prod (#1175)

Browse source
This commit is contained in:
Daniel Roe 2023-01-15 14:34:23 +00:00 committed by GitHub
parent 5da5f9ac7d
commit 5a6248a08f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 23 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
public-staging/robots.txt Normal file
View file

@ -0,0 +1,2 @@
User-agent: *
Disallow: /

21
public/robots.txt
View file

@ -1,2 +1,23 @@
User-agent: * User-agent: *
Allow: / Allow: /
# Disallow authenticated pages
Disallow /intent
Disallow /settings
Disallow /blocks
Disallow /bookmarks
Disallow /compose
Disallow /conversations
Disallow /domain_blocks
Disallow /favourites
Disallow /home
Disallow /mutes
Disallow /notifications
Disallow /pinned
Disallow /search
Disallow /settings
Disallow /share-target
# Wait 1 second between successive requests.
Crawl-delay: 1