Here's How Challenges In AI Data Accessibility Pose Threats To Future Developments
AI developers have long relied on vast amounts of text, images, and videos from the internet to train their models. However, this data is becoming less accessible.
Over the past year, many key web sources for AI training have restricted data usage. A recent study by the Data Provenance Initiative, led by MIT, highlights this trend.
Emerging Crisis in Consent
The study examined 14,000 web domains included in three popular AI training data sets. It found an "emerging crisis in consent" as publishers and platforms are increasingly blocking data harvesting.
Researchers estimate that 5% of all data and 25% of high-quality data in the C4, RefinedWeb, and Dolma sets have been restricted. These restrictions are implemented through the Robots Exclusion Protocol using a file called robots.txt.
Shayne Longpre, the study's lead author, stated, "We're seeing a rapid decline in consent to use data across the web that will have ramifications not just for AI companies, but for researchers, academics and noncommercial entities."
Impact on Generative AI Systems
Data is crucial for generative AI systems like OpenAI's ChatGPT and Google's Gemini. These systems learn from billions of text, image, and video examples scraped from public websites.
The quality of AI outputs improves with better data. However, as more websites restrict access or change terms of service, obtaining high-quality data becomes challenging.
For years, AI developers could easily gather data. But recent tensions with data owners have led to paywalls and legal actions. Some publishers now charge for access or block automated crawlers used by companies like OpenAI and Google.
Legal Actions and Agreements
Sites like Reddit and StackOverflow now charge AI companies for data access. The New York Times sued OpenAI and Microsoft last year for copyright infringement over using news articles without permission.
In response to these challenges, some AI companies have made deals with publishers. For example, The Associated Press and News Corp have agreements allowing ongoing access to their content.
This shift in data availability poses significant challenges for AI development. As restrictions increase, finding alternative high-quality data sources becomes essential for maintaining progress in AI technologies.
The evolving landscape of data accessibility will continue to impact researchers and developers. Adapting to these changes is crucial for future advancements in artificial intelligence.
Sign in with Google