AI Definitions: Stealth Crawler
/Stealth Crawler (AI crawlers) – This type of web crawler scrapes content from sites, often ignoring whether a site wants them to do so or not. Stealth crawlers may disguise themselves as ordinary visitors, making it hard for publishers to identify and stop them. Since stealth crawlers don’t identify themselves, the traffic could come from an AI company or a company mining data it can resell. Publishers are asking for laws that would require AI crawlers to disclose their identity and purpose. Critics say these laws are too broad—that crawlers often benefit the public by facilitating academic research, investigative reporting, etc. For example, Meta stopped researchers from using automated tools that studied misinformation on Facebook. Observers point out that blocking AI crawlers can make sites less visible—preventing them from showing up in AI summaries and answers may be similar to blocking Google crawlers, which would keep a site from appearing in Google search results. Some observers believe scraping has become a billion-dollar industry.