Breaking the Paywall Black Hole: How AI Could Finally Democratize Real News Access
Every morning I open my phone expecting to catch up on the day's business and economic pulse—only to slam into paywalls. The Economic Times (ET) has become particularly brazen: 80–90% of their stories are locked behind the ET Prime paywall. Worse, they even wall off stories that are already public—PIB releases, government announcements, even widely shared updates on X. It is creating a growing black hole in accessible information. This isn't just an irritation for readers. It's a systemic problem for AI too. Most large language models (LLMs) were trained on broad web crawls from years ago, before paywalls hardened and publishers started aggressively blocking crawlers. Newer models increasingly miss high-quality, exclusive content, falling back on open snippets, social media noise, aggregated summaries, or lower-effort reposts. The result? AI knowledge on current events gets shallower over time—especially for nuanced stories from countries like India. I’ve been turning this ...