Meta Develops Independent Search Index for AI Chatbot

Meta is reportedly in the process of establishing a dedicated search index for its AI chatbot, aiming to provide AI-generated summaries while decreasing its reliance on Google.

This new initiative indicates a significant advancement for Meta AI, positioning it closer to functioning as an autonomous search engine.

Bot Activity and Internet Crawling

Since last summer, Meta has utilized a user agent named Meta-ExternalAgent to crawl the web extensively. Numerous users on various forums have noted an unusually high level of crawling activity, with one individual on Hacker News claiming to have received 50,000 hits from the bot. Discussions in the WebmasterWorld bot crawling forum highlight that, despite the Meta-ExternalAgent’s documentation stating it adheres to robots.txt protocols, it appears not to have accessed this file during its activities.

It’s possible that the bot was not fully operational earlier in the year, and its erratic behavior may have since improved.

The primary function of this bot is to compile search summaries, which will help Meta rely less on Google and Bing for information retrieval.

A Potential Competitor to Google?

This development might signify Meta’s intent to challenge Google and other search engines in the AI search space. Current evidence suggests that Meta is focused on building a search index that would enhance its Meta AI capabilities. According to a report from The Verge, Meta is gathering data from websites to generate search summaries for its AI chatbot:

“The search engine is expected to deliver AI-generated summaries of current events within the Meta AI interface.”

The Meta AI chatbot resembles a search engine but still seems to depend on Google’s search index for its outputs. For instance, a query regarding the recent fourth game of the World Series resulted in a summary that not only provided accurate information but also included a link directing users to Google.