Google Introduces Search Crawler Caller

The software's control panel allows companies using Google's service to index their Web sites on demand.

Thomas Claburn, Editor at Large, Enterprise Mobility

November 12, 2008

2 Min Read

Google on Thursday plans to add a button to its Google Site Search control panel that allows companies using the service to index their Web sites on demand.

Google Site Search is Google's $100-plus/year search-as-a-service offering for public Web sites, not to be confused with the Google Search Appliance, which tends to be deployed behind corporate firewalls to index internal Web sites.

Previously, Site Search customers had no choice but to wait for Google's Web crawler to re-index their sites. This can take days. The exact amount of time depends on an algorithm that Google uses to calculate the interval between indexing sessions. Nitin Mangtani, lead product manager for Google Enterprise Search products, declined to provide details about how that interval is determined.

When new content is added to a Web site, it's effectively invisible to Google Site Search until it gets indexed. Waiting several days for this to happen may not be desirable. That's why Google is giving its Site Search customers the leash to its spider.

"We re-index the Web sites when they change but we don't give the end-user any control over when they want them re-indexed," said Mangtani. "With this release, that's what we're offering them."

Adobe uses Google Site Search on its Web site and the company recently tested on-demand indexing for the launch of Adobe Creative Suite 4.

"Google Site Search made it easy to implement search across our Creative Suite product line and online sites, and we are now able to index thousands of new pages and make them available to millions of users worldwide within hours," said Tanya Wendling, senior director for Learning Resources at Adobe, in a statement.

Google Site Search indexing has no impact on the information in the main Google Search index. "Any special indexing we do for business customers does not impact," said Mangtani. This also applies to the secondary search box that appears beneath some listings on Google search results pages. Though the secondary search box allows searches restricted to a specific site, it queries the main Google index rather than a Google Site Search index. (Secondary search boxes merely provide an alternate format to submit a query using the site: search operator.)

Mangtani acknowledged that some news sites may not need on-demand indexing because Google indexes high-volume producers of content more frequently than the average corporate Web site. But business or government Web sites, which aren't typically indexed several times daily, are more likely to appreciate a button to summon Google's search crawler, he said.

About the Author(s)

Thomas Claburn

Editor at Large, Enterprise Mobility

Thomas Claburn has been writing about business and technology since 1996, for publications such as New Architect, PC Computing, InformationWeek, Salon, Wired, and Ziff Davis Smart Business. Before that, he worked in film and television, having earned a not particularly useful master's degree in film production. He wrote the original treatment for 3DO's Killing Time, a short story that appeared in On Spec, and the screenplay for an independent film called The Hanged Man, which he would later direct. He's the author of a science fiction novel, Reflecting Fires, and a sadly neglected blog, Lot 49. His iPhone game, Blocfall, is available through the iTunes App Store. His wife is a talented jazz singer; he does not sing, which is for the best.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like

More Insights