Google: From Search Engine to Information Gatekeeper

Google's initial goal was simple: "to organize the world's information and make it universally accessible and useful." But the landscape of search has changed over the years, and many users feel that Google’s trajectory has shifted from that original vision.

1. Immediate Indexing to Selective Showcasing
   • Past: Launch a blog, and within a short span, it would be indexed and discoverable on Google.
   • Present: Today, even if a website is crawled, it isn't necessarily indexed. Google's Webmaster Central often displays a status like "crawled but not indexed" - a limbo state with no clarity on future indexing.
2. Depth vs. Surface Scratching
   • Past: Users could dive deep into search results, going through ~100 pages to find what they wanted.
   • Present: Now, even for broad queries, the results cap at around 150, even if millions are technically available.
3. AI Influence
   • There was a point when Google was transparent about not wanting to use AI in their ranking algorithms, deeming it a "black box." However, in 2016 the baton has been passed to individuals from AI backgrounds, hinting at the substantial role AI now plays in determining search outcomes.
   • I have a feeling that nowadays some AI decides what should be indexed and what shouldn't, making Google a Gatekeeper rather than a Search Engine.
4. The Ad-Infused Ecosystem
   • As Google grew, so did its commercial interests. Today, ads prominently feature in search results, often making genuine organic searches harder to spot.
   • Google keeps repeating that their ads business has no direct influence on search engine algorithms. Google is very good at clever wording. I'm sure that while there may not be any direct influence, there is certainly some kind of feedback loop that indirectly influences the search algorithms in a way that ends up generating greater advertising revenue.
   • One example is all those features like Local Pack, People Also Ask, etc. which might be useful for some users. But most importantly, they push down the organic search results and thus bring the ads up.
5. The Challenge of Depth
   • A common sentiment is that no matter how specific or refined a query, Google tends to present the same superficial results. It often feels as if there's a default set of information for a topic, with deeper dives becoming increasingly challenging.

Google has evolved from being a comprehensive search engine to a more curated, AI-driven information provider. It's a shift from "everything on the web" to "what we think is relevant from the web."

Why this is bad

• Loss of the Deep Dive: The beauty of the internet was always the depth and diversity of content. With constraints on search depth and breadth, we are losing out on the richness of the web.
• Power Dynamics: When a single entity becomes the primary gatekeeper of information, there are genuine concerns about biases, both unintentional (algorithmic) and deliberate (commercial).
• The Need for Alternatives: The changes in Google's approach emphasize the need for alternative search platforms that cater to different user needs, especially those seeking in-depth or niche content.

Bing and other Alternatives

Bing stands as a noteworthy competitor, but it appears to model itself closely after Google's design and functional choices. In doing so, it often feels less like an alternative and more like a watered-down version of Google.

One notable area where Bing is now better than Google is indexing. From my observations Bing seems to be better at indexing sites compared to Google.

There are emerging players like Brave Search and Mojeek. But building a comprehensive search engine is no small task. As of now, these smaller engines struggle to compete on the same scale.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com