Publishers are shutting the door on the Internet Archive to starve AI models
Table of Contents
A preemptive strike against the scrapers
For decades, the Internet Archive has acted as the web’s definitive safety net. Through its Wayback Machine, the non-profit has meticulously cached billions of pages, ensuring that when a site goes dark or a story is edited into oblivion, a record remains. But for a growing number of the world’s largest news publishers, this digital preservation is starting to look like a liability.
Major media organizations—including The New York Times, The Guardian, and USA Today Co.—have begun implementing blocks to prevent the Internet Archive from crawling and indexing their content. The move isn’t a sudden dispute over copyright or a disagreement with the Archive’s mission, but rather a strategic defensive maneuver in the ongoing war over generative AI training data.
The core of the conflict lies in how LLMs (Large Language Models) are built. AI companies like OpenAI and Perplexity require astronomical amounts of high-quality, factually grounded text to refine their models. While many publishers have already updated their robots.txt files to tell AI bots to stay away, those blocks only apply to the live web. The Wayback Machine, however, represents a massive, pre-compiled library of historical data that AI developers can potentially leverage to bypass current restrictions.
The ‘Ghost’ Data Problem
The irony of the situation is that the publishers aren’t necessarily fighting the Internet Archive itself. Instead, they are treating the Archive as a potential conduit for third-party AI firms. By blocking the Archive’s crawlers, publishers are attempting to close a loophole that would allow AI companies to scrape years of journalistic work from a secondary source rather than the primary site.
Interestingly, there is little evidence that this is happening on a massive scale yet. To date, no major publisher has publicly confirmed a specific instance where an AI company scraped their archives specifically via the Wayback Machine. Yet, the number of sites implementing these blocks has grown steadily over the last few months. It is a preemptive strike—a digital scorched-earth policy intended to make their archives less useful for machine learning before the practice becomes systemic.
This creates a tension between the immediate business needs of the press and the long-term goal of historical preservation. When a news organization blocks the Internet Archive, they aren’t just stopping a bot from Google or OpenAI; they are effectively erasing their future historical footprint from one of the only neutral repositories left on the internet.
The legal grey zone
The industry is currently operating in a legal vacuum. While the Internet Archive has fought high-profile battles with book publishers over digital lending, the scraping of news content for AI training is a different beast entirely. Recent lawsuits from publishers against AI companies argue that the training process constitutes copyright infringement on a scale previously unimaginable.
By restricting access to the Wayback Machine, publishers are attempting to exert a form of “data sovereignty.” If the content is not available in a crawlable, archived format, it becomes significantly harder for AI companies to claim that the data was “publicly available” in a way that permits fair use.
For the public, the result is a fraying of the digital record. As more prestige outlets move behind paywalls or block archival bots, the transparency of the internet diminishes. The ability to hold a politician or a corporation accountable for a statement made five years ago—by pulling up a cached version of a deleted article—is becoming a luxury of the few who happened to save PDFs or screenshots.
The Internet Archive continues to advocate for the open web, but as the financial stakes of AI training reach billions of dollars, the concept of a “universal library” is clashing violently with the reality of intellectual property in the age of automation.
