The Coming Intranet

I'm old enough to remember when "being online" meant dialing into a company. You logged into AOL or CompuServe, and what you could read was what they had decided to put inside. The web felt like an escape from that. You could link to anything, anyone could publish, and nobody had to sign in to read a page.

I think we're quietly heading back inside.

Yesterday I wrote about why the schema is the skeleton and the network is the bloodstream: the winning formula, as I see it, is modular, well-structured data plus an established business network that carries it. This post is about what comes next. Once a company holds clean data and a loyal network, the logical next move is to circle the wagons around both. When enough companies do that at once, you don't get one walled garden. You get thousands of small ones, and the open web between them starts to thin out.

I'm calling that the coming intranet. Not one network behind one login, but a web where more and more of what's worth reading lives behind someone's gate.

The walls are going up in the plumbing#

The clearest evidence isn't in headlines. It's in robots.txt files.

The Data Provenance Initiative's Consent in Crisis study audited around 14,000 web domains used to build AI training sets. Between April 2023 and April 2024, more than 5% of all tokens in C4, a widely used training corpus, and over 28% of tokens from its most actively maintained, critical sources became fully restricted by robots.txt. Counting terms of service as well, about 45% of C4 was restricted. That happened in one year, and it was publishers putting up fences, one file at a time.

Then the infrastructure joined in. In July 2025, Cloudflare announced that it would block AI crawlers by default, asking every new domain up front whether to let them in. Cloudflare sits in front of about 20% of the web, and it said more than a million customers had already turned on its one-click AI crawler blocking. It also launched pay per crawl, which lets site owners allow, charge or block each crawler, using the long-dormant HTTP 402 "Payment Required" status code.

I don't blame anyone for this. If a machine is reading your work and answering questions with it so nobody visits you, a fence is a rational response. But a lot of rational fences add up to a different web.

Platforms closed the side doors#

The big platforms moved first, and more bluntly.

In 2023, Reddit announced API pricing of $0.24 per 1,000 calls. The developer of Apollo, a popular third-party app, said that would cost him about $20 million a year, as Ars Technica reported. Apollo shut down. Twitter's new API tiers, per TechCrunch, left the free tier write-only, with Basic at $100 a month and enterprise access starting at $42,000 a month. Wired pointed out that the $42,000 tier bought 50 million tweets, about 0.3% of what the platform produced.

That June, Twitter started blocking logged-out visitors from viewing tweets at all. Elon Musk called it a temporary emergency measure against "data pillaging." The direction was clear anyway: public posts became members-only reading.

The doors didn't close for everyone, though. They closed for people who wouldn't pay. In 2024, Reddit struck an AI data licensing deal with Google, reportedly worth about $60 million a year, as The Verge summarised. News Corp signed a content deal with OpenAI that could be worth more than $250 million over five years in cash and credits, according to reporting cited by The Verge. That is the intranet model in its pure form: the data is still flowing, but along private pipes between parties who have a contract.

The open commons is losing traffic and memory#

Gates change behaviour on both sides. Readers find less on the open web, so they visit it less, so there's less reason to publish there.

SparkToro's 2024 zero-click study estimated that 58.5% of US Google searches and 59.7% of EU searches ended without a click. For every 1,000 searches, only 374 clicks in the US and 360 in the EU went to the open web; the rest stayed on Google's properties or went nowhere. SparkToro now flags that page as out of date, and AI answers have arrived since. I doubt the newer numbers are kinder.

Some of the public commons is visibly thinning. Stack Overflow is the example everyone reaches for. A discussion on Meta Stack Overflow puts monthly new questions at 6,866 in December 2025 and 6,309 in January 2026. By mid-2025, monthly questions had already fallen to roughly where they were when the site launched in 2009, according to The Pragmatic Engineer. My read is that a lot of questions that used to be asked in public are now asked privately, to a chatbot, and the answers live in someone's chat history.

And the open web is forgetting. Pew Research found in 2024 that 38% of web pages that existed in 2013 were no longer accessible a decade later, and a quarter of all pages from 2013 to 2023 were gone. 23% of news pages and 21% of government pages contained at least one broken link, and 54% of Wikipedia pages had a reference link that no longer worked. Content that isn't actively maintained and openly published simply disappears.

Governments are building walls too. The ITIF counted 62 countries with 144 data-localization restrictions in 2021, up from 35 countries and 67 restrictions in 2017. That report is a few years old now, but the direction it describes hasn't reversed.

The data that complicates my story#

I want to be honest about where this thesis is weaker than it sounds.

Stack Overflow was declining before ChatGPT. Gergely Orosz showed that question volume started falling around 2014, when moderation got stricter, and began dropping faster in mid-2020, more than two years before ChatGPT launched. AI accelerated a slide; it didn't start one.

Open models are booming. Hugging Face's spring 2026 report counts 13 million users, more than 2 million public models and over 500,000 public datasets, with more than 30% of the Fortune 500 holding verified accounts. That's a large, open commons growing quickly. Even there, concentration shows: the top 200 models account for about half of all downloads.

Paywalls aren't winning readers over. The Reuters Institute's Digital News Report 2025 found that only 18% of people across 20 richer countries pay for online news, and that subscription levels look to have hit a ceiling. Most people are happy with free offerings. Gating doesn't automatically make people pay. Often they just go without.

The alternatives are small. The same report found that newer networks like Threads, Bluesky and Mastodon reach 2% or less of people for news. Meanwhile, 7% use AI chatbots for news weekly, rising to 15% of under-25s. The exits from the old platforms aren't leading back to the open web so much as into new, private interfaces.

So it isn't a clean story of walls going up everywhere. Some parts of the commons are thriving. But the parts that are thriving are mostly the ones where people chose, deliberately, to publish in the open.

We have seen this movie#

AOL and CompuServe were intranets with marketing budgets. They curated, they charged, and they were very convenient. Then the open web came along and was more interesting than anything inside the walls, because anyone could add to it and anything could link to anything.

Platforms rebuilt the walls in a softer shape. You could still read Facebook pages or tweets without an account, for a while. Search engines indexed them. It felt open while it suited the platforms.

What's different now is that the walls aren't only around consumer platforms. They're around data itself. Every company that has done the work I described yesterday, clean schema and trusted network, now has something worth protecting, and the cheapest protection is a gate. Robots.txt, login walls, paid APIs, private licensing. Each one is defensible. Together, they look like the old dial-up world again, just with better design.

What this means if you build#

Own your corpus. Keep the canonical copy of everything you make on infrastructure you control, in formats you can move. A platform's export button is not a backup plan.

Publish openly on purpose. If you want your work to be found, cited and remembered, put it on the open web deliberately, with stable URLs, and maintain it. Pew's numbers are what happens when nobody does. That's what this site is for.

Treat access as the product. Decide explicitly who can read what, and on which terms: open, logged in, paid, licensed. Then design your schema and your network around that decision instead of discovering it later.

Make gates legible. If you do close something, say so clearly and offer a way in. A transparent gate with a fair price is better than a quietly broken link.

What this means if you read#

Support the open sources you rely on, with money or attention or both. Save what matters to you somewhere you control. Notice when an answer arrives with no source behind it. And when you find something good on the open web, link to it. Links are how the commons stays connected.

The takeaway#

The web isn't closing all at once. It's being fenced one reasonable decision at a time, by publishers, platforms, infrastructure providers and governments, each protecting something they built. I understand every one of those decisions. I've made some of them.

But the sum is an intranet: a web where the best material lives inside someone's walls and the space between gets thinner and more forgetful. I'd like this site to be a small piece of the other thing. Open, linkable and still here in ten years.