Why Major LLM Providers Will End Up Building Their Own Search Index

Newspapers used to have two ways to fill a page. Send a reporter, which was slow and expensive and yours. Or run wire copy from AP, which was cheap, fast, and word for word what every other paper in the country printed that morning.

Most papers ran mostly wire. It was the sensible choice, since you can’t fund a Delhi bureau on a small paper’s budget. But the papers that ended up mattering all did the same thing eventually. They hired reporters.

AI assistants started on wire copy. ChatGPT asked Bing. Perplexity asked Bing. Everyone pulled from the same source, formatted it differently, and shipped it as a product. Perplexity got off it years ago and built its own crawler once it had the traffic to justify one.

Gemini never had this problem. It was born inside the newsroom. Google spent twenty years building the index everyone else was renting, so when it came time to ship an assistant, the reporters were already on staff. No licensing deal, no crawler to build. It’s why AI Overviews needed no new plumbing, and why Gemini is the only assistant in this story that never had to make a decision about any of it.

ChatGPT is the newest one to hire reporters.

The clearest example so far

Peec AI’s investigation, led by Tomek Rudzki, found that ChatGPT runs its own index internally called Labrador. Labrador is really a newsroom with desks: general web, PDFs, YouTube, news split by how fresh it is, arXiv, Wikipedia, local, finance, legal, medical, shopping, images. Each one stores full page content, crawl date, publish date.

ChatGPT still calls Google, Microsoft’s Web IQ, Yelp, and TripAdvisor for some things. Every newsroom keeps a wire subscription. Hiring reporters just means the wire stops being the whole paper.

OpenAI’s job postings describe teams building indexing and retrieval infrastructure at exabyte scale. Sworn testimony from Google’s antitrust trial has ChatGPT’s head, Nick Turley, saying the licensed data had real quality problems. He also says OpenAI asked Google to sell it access directly, and Google refused. He puts the decision to build in-house at 2023. Peec caught live A/B tests too, weighing Labrador’s results against scraped ones in shopping specifically, which is the beat you’d double-check before trusting your own reporter over the wire.

Their piece is worth reading in full. What I want to get at is the pattern behind it.

Why this keeps happening

The obvious answer is cost. Licensed data is slow and gets expensive fast, and once you’re big enough the bill starts looking worse than the engineering.

But cost only explains the timing. The real problem with wire copy was never quality. AP was good. AP was good for everyone. If your paper and the paper across town ran the same story, filed by the same correspondent, the only thing either of you controlled was the headline. You were competing on formatting.

That’s the position every assistant querying Bing was in. You can’t differentiate on a source you share with your competitor.

Google is the other half of this. It isn’t a neutral wire service that happens to also run a paper. It’s the paper, and it owns the wire, and it gets to decide who else is allowed to print. Turley’s testimony is the clearest version of that: OpenAI went and asked to license Google’s data directly, and Google said no. Of course it did. Selling your index to the company trying to replace you means funding the replacement and handing over the one advantage nobody can copy. Refusing was the cheapest thing Google did that year.

That refusal is what sets the terms for everyone else. Gemini gets the index for free because it’s in-house. Everyone else either rents something worse or builds their own, and eventually the second option stops looking optional. At that point hiring reporters is just the only door left.

I don’t think ChatGPT is the last one to make this move. It’s just the one someone caught mid-hire, thanks to a field in its own server data that gave the whole thing away for two months before it got cleaned up. I doubt anyone still on wire stays there once they can afford a newsroom.

What this breaks in how GEO has been done

The old model was a wire pitch. Land your story on AP and every paper in the country might run it. Rank in Google, show up in Bing, and the assistants downstream would eventually reflect it.

That worked because there was one wire. It doesn’t work now that every paper has reporters.

Being visible to Google tells you nothing about whether GPTBot has crawled you, or whether Perplexity holds a current copy of your page. It also tells you nothing about which desk you landed on. Labrador having separate shopping, local, and finance indexes is the detail I keep coming back to. Getting picked up by the general news desk doesn’t get you into the business section.

There’s a freshness problem under all of this too. If an assistant answers from a stored copy of your page instead of fetching it live, how often that copy gets refreshed matters as much as whether the page ranks. An archive is only as good as its last update, and most GEO tools I’ve used can’t tell you when that was.

Where I’d put the effort now

I’m not throwing out the fundamentals. Clear, current pages still win everywhere, because all of these systems reward some version of what Google always rewarded.

What’s gone is the single pitch. I’m checking server logs for GPTBot and PerplexityBot by name instead of assuming Google speaks for everyone, and thinking about which desk a page belongs on rather than whether it’s indexed at all. ChatGPT’s lockdown mode helps here. It can’t browse live, so anything it still returns has to be coming from a stored copy, which makes it a rough way to check whether your page has actually been picked up.

There’s no single funnel left

The uncomfortable part is that nobody announces any of this. We only know about Labrador because a field in ChatGPT’s own server data slipped out for two months. There’s no reason to expect the next one to surface that cleanly.

There is no wire left to pitch. There are only editors.