Scraping Xenforo Posts

[chronoshift]

Does anyone have experience with scraping an entire forum? I can pay for help with this

 

[Thanos Crax]

Use rss feeds
Though its limited to 25 latest threads.

Scraping a whole forum is hard. You need a good hold of selenium bots and maybe possible then

 

[BattleKing]

Take a look into these tools maybe they could help.

1. https://github.com/TUVIMEN/xenforo
2. https://github.com/qjroberts/xenforo-scraper

If you test them, please let some feedback here.

 

[chronoshift]

Take a look into these tools maybe they could help.

1. https://github.com/TUVIMEN/xenforo
2. https://github.com/qjroberts/xenforo-scraper

If you test them, please let some feedback here.

BattleKingThanks for the links, I actually have a custom script that someone made to scrape a forum, it works really well, however the account used for scraping keeps getting banned on the forum (even though it's not posting anything, only scraping)... I can't figure out how they are detecting the scraper and banning the user account being used to do the scraping

 

[BattleKing]

Thanks for the links, I actually have a custom script that someone made to scrape a forum, it works really well, however the account used for scraping keeps getting banned on the forum (even though it's not posting anything, only scraping)... I can't figure out how they are detecting the scraper and banning the user account being used to do the scraping

chronoshiftCould you sent me the script you have used, please

 

[Mafia]

Could you sent me the script you have used, please

BattleKingi can fix your scrapping script your account will not banned but you will need to wait better is scrap via RDP

 

[BattleKing]

i can fix your scrapping script your account will not banned but you will need to wait better is scrap via RDP

MafiaThanks @Mafia but it is not my script, I wanted to take a look into the script as well. I guess the user will be banned, because of to fast requests.

 

[ShieldX]

Thanks for the links, I actually have a custom script that someone made to scrape a forum, it works really well, however the account used for scraping keeps getting banned on the forum (even though it's not posting anything, only scraping)... I can't figure out how they are detecting the scraper and banning the user account being used to do the scraping

chronoshiftcan you tell me that site name who keep making you banned ?

 

[BattleKing]

Take a look into these tools maybe they could help.

1. https://github.com/TUVIMEN/xenforo
2. https://github.com/qjroberts/xenforo-scraper

If you test them, please let some feedback here.

 

[Mafia]

i am pythong developer not nodejs and python is best for data scrapping

 

[BattleKing]

i am pythong developer not nodejs and python is best for data scrapping

MafiaAnd you got a script to scrap a complete forum, threads posts images ...

 

[ShieldX]

Does anyone have experience with scraping an entire forum? I can pay for help with this

chronoshiftcontact me , i can scrape anything and post in your forum , whether its a website or forum , i can scrape and do it for you.