TIME Is Serving AI Bots a Different Website, With Ads Built In
TIME is now serving two different versions of its website. Humans get the magazine. AI crawlers get a stripped down markdown copy…
Categories
Google Now Defaults to Not Indexing Your Content
Picture this: It's ten years ago, and you've just launched a new WordPress blog. Within hours, sometimes even minutes, your content is indexed by Google. You search for a unique sentence from your latest post, and there it is, right on the first page of Google. It felt like magic, didn't it?
This was Google living up to its mission: "to organize the world's information and make it universally accessible." For users, it meant that if the information existed somewhere on the web, a bit of clever searching would unearth it. For content creators, it was a golden opportunity: create content, and Google will make sure it's found.
The SEO Arms Race and Google's Response
But where there's opportunity, there's also exploitation:
- Low-quality content flooding the web
- Link buying and selling
- Lots of grey and black hat SEO stuff
Google introduced a series of algorithm updates, each more sophisticated than the last. Panda, Penguin, and their successors aimed to clean up what Google saw as manipulative SEO practices.
This wasn't a one-time event, but more like a two-decade-long dance between SEOs and Google. Each side continually adapted, with Google rolling out updates and SEOs finding new ways to optimize within (and sometimes outside) the guidelines.
The Rise and Fall of "Content is King"
Amid this ongoing battle between SEOs and Google, a new mantra emerged in the industry: "Content is King". This concept had two main aspects:
- Good content will naturally attract links and possibly generate positive user interaction signals, which could improve rankings.
- Google's algorithms would inherently value and rank high-quality content.
The idea was that by focusing on creating great content, websites could improve their SEO without resorting to more technical or potentially manipulative tactics.
However, this concept was never fully true. Many creators of genuinely good content never achieved the rankings they felt they deserved, showing that quality alone wasn't enough. At the same time, it wasn't entirely false either - good content did often perform well in search results.
One of the fundamental issues was that neither Google nor anyone else could algorithmically determine "good" content.
The mantra served as a useful simplification for SEOs and content creators, giving them a clear goal to aim for. It was also a convenient explanation for Google when SEOs delved too deeply into technical details - they could always fall back on "just create good content".
The AI Content Revolution
Fast forward to 2018. OpenAI releases GPT-1, and suddenly, the future of content creation becomes crystal clear. AI-generated content, indistinguishable from human-written text, is on the horizon. The "content is king" era is coming to an end.
Google's Existential Crisis
Now, put yourself in Google's shoes. You're facing a future where AI can generate infinite amounts of human-like content. What do you do?
Google's response was twofold:
-
Promote the vague concept of E-A-T (Expertise, Authoritativeness, Trustworthiness). In practice, this translates to favoring well-known brands and established websites.
-
Abandon the mission of indexing everything. Instead, become selective. Very selective.
The New Reality: Selective Indexing
This brings us to the current state of affairs: Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.
From my experience, Google now seems to operate on a "default to not index" basis. It only includes content in its index when it perceives a genuine need. This decision appears to be based on various factors:
- Extreme content uniqueness: It's not enough to write about something that isn't extensively covered. Google seems to require content to be genuinely novel or fill a significant gap in its index.
- Perceived authority: Sites that Google considers highly authoritative in their niche may have more content indexed, but even then, it's not guaranteed.
- Brand recognition: Well-known brands often see most of their content indexed, while small or unknown bloggers face much stricter selectivity.
- Temporary indexing and de-indexing: In practice, Google often indexes new content quite quickly, likely to avoid missing out on breaking news or important updates. Soon after, Google may de-index the content, and it remains de-indexed thereafter. So getting initially indexed isn't necessarily a sign that Google considers your content valuable.
I've observed this shift firsthand. In the past, when I set up a new domain, it would be indexed within an hour or faster, sometimes in seconds. This was true even for brand new domains with no mentions anywhere and no backlinks. When I searched for the title of one of those brand new blog posts or some unique sentence from the article, it would be right there on the first Google page.
Now, for each piece of content, Google decides if it's worth indexing, and more often than not, the answer seems to be "no." They might index content they perceive as truly unique or on topics that aren't covered at all. But if you write about a topic that Google considers even remotely addressed elsewhere, they likely won't index it. This can happen even if you're a well-respected writer with a substantial readership.
Interestingly, I've noticed that when content does manage to get indexed, it often ranks surprisingly well. It's as if the hurdle of getting indexed has become so high that once you clear it, you're already most of the way to ranking. However, getting to that point has become exponentially more difficult.
Importantly, this extreme selectivity isn't applied equally. Big, recognized brands often see most of their content indexed quickly, while small bloggers or niche websites face a much higher bar for inclusion. For these smaller players, it's not just about creating good content anymore – it's about convincing Google that your content is absolutely necessary for their index.
The Consequences
Google has transformed from a comprehensive search engine into something more akin to an exclusive catalog.
For users, it means that the information they're looking for might exist but remain undiscoverable through Google.
I'm sure that a vast amount of valuable content is being overlooked. Information that you might search for may never appear in Google's results. Not because it doesn't exist, but because Google has chosen not to include it.
For content creators, it presents a significant challenge: how do you gain visibility if Google refuses to index most of your content?
What I'm building
Delegate tasks. Get software.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com
Thank you. This is one of the saddest articles I’ve read all year.
I wonder if the folks at Google realize that big, established brands are quite frequently resorting to AI content, or otherwise using deceptive practices to lean on their prior expertise and trustworthiness to generate ad revenues through clickbait. Without naming names, I’ll refer to the post at HouseFresh.com about Google killing independent sites.
As a content creator, I have been trying to pay attention to listings on other search engines (like DuckDuckGo) since the writing seems to be on the wall for Google’s eventual demise as the king of search.
Super interesting read, Vincent! Thank you for sharing. Would you mind popping a link to some of the references you used to write this? I’m trying to find the documentation to back this up on Google but seem to be having a hard time finding more info. Thanks in advance, and thanks again for your insight!
> how do you gain visibility if Google refuses to index most of your content?
Nothing, you do nothing, you just accept it. You can’t convince a titan to care about a flea. The only viable option is to move to a customer base that doesn’t require google for you to be accessed. If you move platforms you face the same problem, if not now then in the near future. As long as the platforms keep the popular people happy there is nothing you can do.
The dystopian future exists here today, it is just hidden behind attractive CSS themed html.
From my view the solution is right there in your article.
Stop caring about if and how you’re indexed by Google. As long as your “content” seems to be there just to be indexed it has no actual value.
The problem is that there are “creators” that “make content” to gain “visibility”. Not knowledgable people that want to share their knowledge with others.
It’s likely that your post will be indexed by Google 🙂
I find I need Bing to find unique content.
Indeed. I just tried this out with my own website. It’s not advertised anywhere and barely linked to from anyone, if at all. Bing will find my posts with a targeted search, but for Google my site simply doesn’t t exist.
So this means if I use the Google Search console indexing tool to submit a page to the index, it will just be ignored if G believes there is no need for that info?
It’s called censorship people. Wake up.
Censorship is about the gov using their power to shut one up.
A company or human being, no matter how powerful, reserves the right to not associate with stuff they don’t want to be associated with. That includes Google, their index and anyone’s content. If you don’t like that, you need to vote into power people that will classify Google as a sort of common good and force it to index all content. Good luck with that!
The government has no power over Google??? That’s news!
Are you suggesting that this was instigated by the government and is thus censorship?
I see the traditional method of preparing for a “pay to index” subscription service. DuckDuckGo.com might be getting a boast to their site
Funnily, I read your post in Flipboard. There are still places to still index small but valuable content😜
Funny you mention Flipboard: I have just discovered today that — after over a decade of being able to tell anyone interested (including students) that they can find my commented public collections under 50 topics by a Google search for “Flipboard” and my username (unsecurity), or the name of a collection — this suddenly does not work, despite 170,000 followers.
This also includes Googling my Flipboard account’s URL, its username prefixed by “@”, a fairly uncommon collection name (uberveillance — this finds a few secondary references on other sites but not it itself), or other variants.
Gone, nada, null. No longer indexed.
Bing happily serves up my main Flipboard page, plus a sampling of the more popular collections.
I am a bit shocked that Google’s widely reputed enshittification has suddenly broken what had been a reliable omnivorous infrastructure service for nearly 15 years.
This explains so much. Google has de-listed pages on my tiny microblogging site that are ten years old, leaving me previously very confused. On the other hand, last month I got one hit from Bing, but otherwise all search traffic was Google. This, even though Bing scarfs up 5X the MB every month that Google does.
Well, at least the content is training AIs to replace us all, so some good will come of it.
Interesting read, thanks, Vincent. I’ve heard from a couple of respected designers in recent weeks who’ve said similar things about their blog traffic, that previously huge monthly site visits have fallen off a cliff.
Perhaps it’ll push more people into increased “social media” use and away from more time spent on their own domains. I hope not.
I believe this has been the case for some time. I run a [very old, nearly pre-Google] site with about 9 million pages of entirely user-generated content, and it used to be the case that a good 80-90% of the site — still several million pages at the time — was indexed. I don’t participate in SEO or other shenanigans, just simple design and real content. For (5+) years now I’ve barely managed to convince them to index 240k pages, or about 2.5% of them. This coincides with the collapse in the value of web advertising, which already follows the loss of contextual ads, making running a site funded by passive content (content no one’s ever going to see) monetization less and less possible. Gone are the years of internet services that provided value for normal people without turning them into marketing segments.
Thank you!
The harder they try, they mess up things even further. Also they make it very challenging for new sites to populate. If a doctor is starting health blog & he has no knowledge of SEO, he may suffer to even get the site indexed, despite the uniqueness of his content. Give everyone a fair chance to compete at least.
If you see the search results, they are not improved at all. You have a high authority site, post anything and its most likely to rank, recently many affiliate exploited this to rank their affiliate keywords on high authority branded websites which are accepting sponsored content.
However, I noticed the content which does not appear on google has often a valid reason, and I am not sure how deindexing outdated content or old news will improve the research result, If I want to check details of a crime happened in 2007 & google find that page outdated & deindex it, its not good for users.
The irony here is that Google is still organizing the world’s information, it’s just that the world’s web data is slop. Google’s own AIs can piece together the same knowledge as any content creator could (at least those relying on the web for knowledge). Basically, if you’re not truly contributing new info to their knowledge graph AND doing so with authority, don’t expect to be in the index.
That looks like a truly Panglossian take, Tyler M.. Did a Google AI write it?
In this explanation, is it actually impossible for a human to add to the knowledge graph? If not, how does Google determine if something is an addition? The claim seems to be like the joke about the economist not picking up a dollar bill because they know it’s not there because someone else would have picked it up already. I’m fairly sure that Google’s incentives are not dissimilar to Alex Jones’s and eventually the results of their actions may be similar too. I’m not sure that amounts to a knowledge graph.
Another interesting thing that I observed is that when I search for certain text in my article, Google returns 0 results. When I include the site name, then it appears in the search results. Also, GSC says “URL is on Google” for that article. Weird times.
Thank you for explaining why my google searches for a while now were giving poor results. Time to look for another search engine.
Great informative article, but where’s the solution? Throughout my career, I’ve operated under the idea that all things have a solution.
As I navigate the complexities of web indexing on Google Search Console for my clients, I often find myself manually requesting Google to index pages that I’ve verified as active. Although time-consuming, adapting to change is a process forward until innovations come on the scene to solve such problems.
“It is not the strongest species that survive, nor the most intelligent, but the most adaptable to change.”
-Charles Darwin
Pay for Google Ads, that’s what this is all about.
I don’t know enough about the current state of indexation to comment on these conclusions.
On the ranking front, this discussion is missing the most important point. For the last decade or so, search ranking (not indexing) on google has largely been dominated by their AI system. While it is certainly not the only ranking factor, nor possibly the “most important”, it has been so built into the DNA of their system that its effects are huge.
The biggest training set for that AI, the one they trust the most, is user INPUT into the search box followed by behavior on search results for specific queries. Google insists they don’t favors “brands” but they reward navigational search, where users are indicating they are looking for a specific thing (domain name, brand, brand + keyword, or terms unique to a specific brand ) which result in very high click through (85%+) for the top result. These kinds of queries are crack cocaine for the algorithm and help sites rank for related terms. If enough users search for Bellagio Los Vegas then Bellagio also ranks well for Vegas Hotel.
Users also tend to click on familiar brands when they see them in search results. Search for “best pilates roller” and you will see results from The New York Times, Health.com, and Amazon followed by Pilates.com, New York Magazine and The New York Post. Everyone one of these results except Amazon is a site which uses some form of affiliate monetization. The publications create thinly veiled advertorial because image and text ads do not pay enough to support the business model.
The solution may be to consider what would make for a better search experience than Google is currently providing. Some niche directories still exist – in my area, they’re mostly the ones provided by local government in order to boost tourism or improve local health. Now would be a good time to seek them out and bookmark them, and tell your friends about them.
It takes a few things for a directory-style solution to be worthwhile in a niche. A certain critical mass, actual human curation, regular review of the links within, and a USP which means you’re not merely dealing with a boring list of links, but something else that makes the site a decent destination too. It’s not an easy call, nor is it cheap, and there’s a good reason most directories on the Open Directory model disappeared in the 2000s.
So that could be part of the answer, but I don’t think it’s going to get us back to where we were before AI poisoned the well. The web has always been pay to play, but for a brief period at the beginning of this century it felt like anyone who could afford a website and who had something useful to say could do so, and that felt like a democratisation. Now, not so much, and the trajectory is clear.
Maybe we are missing the point here. It was reasonable to expect there to be a content index to the entire Internet in its youth. But now, with its size, growth rate and automated content generation, it’s too large and containing too much ‘noise’ for indexes to be of value. In the early days individuals would curate index / reference sites in their personal areas of interest or expertise. Maybe we need to return to that?
Are there more than anecdotal evidence about this?
Interested on it too!
“Information that you might search for may never appear in Google’s results.”
This is the most bizarre part, because even if we try to treat this shift as logical or whatever, pretty much any search query now trails off into totally irrelevant results after page 2. So it doesn’t make any sense. Like okay, make Forbes and New York Times outrank everyone if you think that’s best… but why are all the results after page 1-2 completely irrelevant now? Like not even remotely helpful to the query and not even tangentially related… just pages and pages of totally random and bizarre results now that just happen to be from a high authority domain or whatever. Makes no sense at all.
Thank you, interesting.
But I believe in the power of competition.
If Google did not index any more than this is the chance for competitors (Bing + ChatGPT, Yahoo, DuckDuckGo etc.).
Did You try to find your blogposts in ChatGPT ? Google is in still in fear of KI-Tools will attract their customers leaving them ruined like Nokia against Apples Smartphones
AI generates a lot of content. Is it possible that the database cannot hold it all?
That’s a very interesting reed. TBH with all the newly generated AI content it’s not like the search engines have much choice or perhaps they do? If so, are we finally going to see some competition in the search space ?
It’s a big blow to content creators and might reduce the diversity of search results. The ‘default to not index’ policy will reshape the SEO landscape and content creation strategies.