Multiple developers have seen increased activity from Apple's web crawler Applebot in their website logs recently, reigniting speculation that Apple could be planning to launch a full-fledged search engine eventually. However, it's more likely that any possible uptick relates to Apple's efforts to improve Siri and Spotlight search results.
Jon Henshaw, founder of digital marketing insights firm Coywolf, kicked off the speculation with a blog post this week in which he said that Applebot had begun regularly crawling his websites on a daily basis, something he had not noticed previously. And on Twitter, Stack Overflow engineer Nick Craver and digital marketing consultant Michael James Field also noted spikes in Applebot crawling on websites they oversee in recent days.
Well, that's the first time Applebot went crazy crawling us. I always forget they have a crawler... — Nick Craver (@Nick_Craver) August 18, 2020
This is very interesting. We're also seeing massive spike in crawls from applebot too on our sites (From 0 to 6k-7k over last few days) https://t.co/nUMAMty7Du — Michael James Field (Mikuss) (@Mikuss) August 27, 2020
Like other web crawlers, Applebot scans the web to help determine how search results should be ranked based on several factors, including user engagement, the relevancy and matching of search terms to a page's topics and content, the number of links that a page has received from other websites, and a page's design characteristics.
As noted by Henshaw, Apple updated its Applebot support document in July with new details:
• Added how to verify traffic from Applebot
• Expanded details on the Applebot user agent, including differences between its desktop and mobile version
• Expanded robots.txt rules
• Added a section stating that they don't just crawl HTML, but also render pages similar to Google
• Added a section on search rankings and the factors that affect how it ranks web search results
Given that Apple promotes itself as a privacy-focused company without an ad- or data-driven business model like Google or Facebook, it is uncertain if it would ever want to go down the avenue of launching a full-fledged search engine, although DuckDuckGo has at least shown that it can be accomplished with privacy in mind.
The idea of an Apple search engine has been speculated since at least 2015, when Apple first confirmed its Applebot and posted a series of search-related job listings.
For now, Applebot likely remains dedicated to improving Siri and Spotlight search results, as Apple states in its support document. During its WWDC keynote in June, for example, Apple said that Siri can provide over 20 times as many facts as three years prior.
