xF2 Add-on AndyB Robots 2.6

[ENXF NET]

LQD submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Robots link in Visitors tab)
View attachment 993
(Example of Robots page)
View attachment 994
(Example of Options page)
View attachment 995
(Example of User group permissions)
View attachment 996
Features:

• All phrases start with robots_ for your convenience.

Installation:

1. Download Andy-Robots-2.3.zip and unzip it.
2. Copy the...

Read more about this resource...

 

[ENXF NET]

ENXF NET updated Robots with a new update entry:

See description

Robots v2.4 changes:

Updated setup to use schema manager.

Read the rest of this update entry...

 

[ENXF NET]

ENXF NET updated AndyB Robots with a new update entry:

See description

Robots v2.5 changes:

Now using finder for performing database queries.​

Read the rest of this update entry...

 

[ENXF NET]

ENXF NET updated AndyB Robots with a new update entry:

See description

Robots v2.6 changes:

Fixed issue with Cron entry not deleting old data.

Read the rest of this update entry...

 

[CSAdmin]

Andy pulled this add-on.

Oct 11, 2022 - 1 Posts - Andy
The Robots v2.6 add-on is no longer available. I suggest using .htaccess instead to block bad bots. Here are instructions: https://www.xf2addons.com/threads/using-htaccess-to-block-bad-bots.1135/

 

[BattleKing]

Using .htaccess to block bad bots​

If you would like to add additional bad bots, you add them on this line. Keep in mind that by having "bot" already entered, that will cover any bot with the work "bot" in the user agent.

SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect" bad_bot

Just add the | symbol followed by the name of the bad bot.

If you would like to add good bots, you add them on this line.

SetEnvIfNoCase User-Agent "bingbot|duckduckgo|googlebot|yahoo" good_bot

Just add the | symbol followed by the name of the good bot. Note that I prefer to remove the bingbot as I consider it a bad bot.

    #   Deny and Allow bots by User-Agent
    SetEnvIfNoCase User-Agent "bot|crawler|fetcher|headlesschrome|inspect|search|spider" bad_bot
    SetEnvIfNoCase User-Agent "duckduckgo|googlebot|yahoo" good_bot
    Deny from env=bad_bot
    Allow from env=good_bot

I suggest installing the following two add-ons. With them you can easily identify bad bots which might not be identified with the directives above. Once identified you can update the .htaccess bad bots directives.

• Access Log
• Broken Links