Pagwin/new_blog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

added search engine to robots.txt

Browse source
This commit is contained in:
Pagwin 2025-01-23 12:49:34 -05:00
parent f3eb133d0e
commit 5ec01a63ed
No known key found for this signature in database
GPG key ID: 81137023740CA260
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
robots.txt
View file

@ -50,6 +50,9 @@ Disallow: /
User-agent: YouBot User-agent: YouBot
Disallow: / Disallow: /
# https://marginalia-search.com/ it's a search engine so it's fine
User-Agent: search.marginalia.nu
Allow: /
# generic allowance for search engines and what not # generic allowance for search engines and what not
User-Agent: * User-Agent: *
Allow: / Allow: /