Recommended robots.txt Xenforo 2.2

[juabo10]

Hello!

I've wanted ask, what nowadays is recommended to setup robots.txt for XF2.2

My site is flooded with bots like:
Petal Search
Ahrefs
BLEXbot
SEMRush

etc

Any recommendations are very appreciated, thanks!

 

[DareSec]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

 

[BattleKing]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

DareSecThe one from here: https://xenforo.com/community/threads/recommended-robots-txt.209086/

 

[juabo10]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

DareSecthank you

 

[lbebulk]

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemRush
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /members/
Disallow: /attachments/
Disallow: /goto/
Disallow: /help/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

Sitemap: https://yourforum.com/sitemap.xml

DareSec@DareSec Thank You Very Mush

 

[UNKNOWN PH]

does robot.txt affect google adsense?

@DareSec @BattleKing

 

[DareSec]

does robot.txt affect google adsense?

@DareSec @BattleKing

UNKNOWN PHno

 

[UNKNOWN PH]

no

DareSeci want to know what is the purpose of robot.txt?

 

[DareSec]

i want to know what is the purpose of robot.txt?

UNKNOWN PHto not crawl certain pages