Welcome Guest. | Log In| Register | Membership Benefits
  • Email this page E-mail
  • |  Print Print
  • |   Bookmark and Share
  • icon

Yahoo Launches Upgraded Search Crawler


As the rollout of Slurp 3.0 begins, sites using IP-based recognition of crawlers might see a drop in Yahoo activity, the company said.



Yahoo has launched the latest version of its search engine, and notified Web site administrators of some infrastructure changes.

The rollout of Slurp 3.0 officially began Monday and is expected to take several weeks. Among the changes caused by the upgrade would be crawling from a different and much smaller set of IP addresses, but still from the crawl.yahoo.net domain.

As a result, sites using IP-based recognition of crawlers might see a drop in Yahoo activity. "We strongly recommend that you move to reverse DNS-based identification of Yahoo Slurp if you're using any other method to avoid this problem," Yahoo developers said in the portal's search blog. "The current set of IPs will disappear from your Web logs in the next several weeks."

Yahoo said the crawlers also would publish a new user-agent, Yahoo Slurp/3.0. Existing robots.txt directives for "Slurp" or "Yahoo Slurp" would still work, but the new crawler would not recognize directives specific to "Slurp/2.0." However, Yahoo said usage of the latter user-agent is "very rare on the Web, so you won't likely be affected."

Yahoo has set up a help page for more details. The changes are expected to affect the main Yahoo Web Search crawlers.

Research from Web analysis firm Hitwise show that users of Yahoo search are often younger and spend less money online than people who prefer Google. Web searchers who use the latter are most likely to have spent more than $500 online, according to Hitwise.

Yahoo is battling a $40-plus billion takeover bid by Microsoft. Yahoo has recently taken actions that make it more difficult for Microsoft to succeed. The software maker, however, has said it is prepared to launch a hostile takeover.


Subscribe to RSS


Advertisement






Get InformationWeek in Print

Apply for a free 52-week subscription to InformationWeek (a $199 value)



NOTE: Offer valid for U.S., U.S. possessions, & Canada only.