Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Server Farms 2024 Continuing discussion of hosting and data center IP ranges |
5 | not2easy | 7:07 pm Jan 19, 2024 | ||||
| AI UAs - Identifying the crawlers AI uses |
1 | not2easy | 5:38 pm Oct 18, 2023 | ||||
| At Home Without the Robots: 2023 edition |
5 | lucy24 | 11:39 am Jul 29, 2023 | ||||
| At Home with the Law-Abiding Robots (2021) |
2 | lucy24 | 8:32 pm Sep 6, 2021 | ||||
| At Home with the Robots: 2020 edition |
11 | lucy24 | 2:48 pm Aug 13, 2020 | ||||
| At Home with the Robots: 2019 edition |
3 | lucy24 | 3:52 am Mar 13, 2019 | ||||
| At Home with the Robots: 2017 edition Critique of the Year's Active User Agents |
15 | lucy24 | 3:52 pm May 18, 2017 | ||||
| User Agents of Programming Libraries and Command Line Tools Common user agents |
14 | incrediBILL | 8:54 pm Apr 29, 2008 | ||||
| Digital Ocean has new IP's |
6 | SumGuy | 2:55 pm Apr 14, 2024 | ||||
| Seeing binance dot com showing up as a referrer UA is always Chrome/90.0.4430.85 |
2 | SumGuy | 2:01 am Apr 12, 2024 | ||||
| Anyone else seeing a lot more of Applebot? Since the beginning of 2024, Applebot has been crawling my sites more often |
17 | sudo | 11:08 pm Apr 5, 2024 | ||||
| Why Is Host Europe (HEG) Suddenly Crawling Me? |
3 | RedBar | 11:20 pm Mar 31, 2024 | ||||
| Google-PageRenderer / developers.google.com/+/web/snippet/ From a google-proxy IP |
1 | SumGuy | 10:47 pm Mar 27, 2024 | ||||
| OpenWebSearch.eu Crawler OWLer |
4 | engine | 6:36 pm Mar 25, 2024 | ||||
| What exactly is YaBrowser and Yowser ? Seeing both in the same UA |
1 | SumGuy | 3:49 am Mar 25, 2024 | ||||
| Has anyone ever seen instagram as a referrer? And also instagram as a user-agent? |
1 | SumGuy | 3:24 am Mar 9, 2024 | ||||
| zstd in Accept-Encoding request header Is this new? Anyone else seeing it? |
1 | SumGuy | 11:02 pm Feb 25, 2024 | ||||
| VsuSearchSpider/1.0 From a residential Deutsche Telekom IP |
4 | SumGuy | 4:01 am Feb 23, 2024 | ||||
| Speaking of blocking VPN's. User-Agent Workstream? |
3 | SumGuy | 12:51 pm Feb 20, 2024 | ||||
| do you block IPXO ranges? |
4 | LifeinAsia | 2:59 am Feb 15, 2024 | ||||
| Timpibot/0.9 |
2 | tangor | 6:55 pm Feb 7, 2024 | ||||
| SiteOne-Crawler Is this a known bot/crawler? |
2 | SumGuy | 4:32 am Jan 25, 2024 | ||||
| Quantbot search engine based in France |
6 | SumGuy | 4:26 am Jan 25, 2024 | ||||
| Interesting hit from a VPN with python UA ?checkstatus=areyouok |
3 | SumGuy | 10:38 pm Jan 22, 2024 | ||||
| fasthttp bots |
2 | not2easy | 1:15 am Jan 21, 2024 |