Google Introduces Search Crawler Caller

Google on Thursday plans to add a button to its Google Site Search control panel that allows companies using the service to index their Web sites on demand.

Google Site Search is Google's $100-plus/year search-as-a-service offering for public Web sites, not to be confused with the Google Search Appliance, which tends to be deployed behind corporate firewalls to index internal Web sites.

Previously, Site Search customers had no choice but to wait for Google's Web crawler to re-index their sites. This can take days. The exact amount of time depends on an algorithm that Google uses to calculate the interval between indexing sessions. Nitin Mangtani, lead product manager for Google Enterprise Search products, declined to provide details about how that interval is determined.

When new content is added to a Web site, it's effectively invisible to Google Site Search until it gets indexed. Waiting several days for this to happen may not be desirable. That's why Google is giving its Site Search customers the leash to its spider.

"We re-index the Web sites when they change but we don't give the end-user any control over when they want them re-indexed," said Mangtani. "With this release, that's what we're offering them."

Adobe uses Google Site Search on its Web site and the company recently tested on-demand indexing for the launch of Adobe Creative Suite 4.

"Google Site Search made it easy to implement search across our Creative Suite product line and online sites, and we are now able to index thousands of new pages and make them available to millions of users worldwide within hours," said Tanya Wendling, senior director for Learning Resources at Adobe, in a statement.

Google Site Search indexing has no impact on the information in the main Google Search index. "Any special indexing we do for business customers does not impact Google.com," said Mangtani. This also applies to the secondary search box that appears beneath some listings on Google search results pages. Though the secondary search box allows searches restricted to a specific site, it queries the main Google index rather than a Google Site Search index. (Secondary search boxes merely provide an alternate format to submit a query using the site: search operator.)

Mangtani acknowledged that some news sites may not need on-demand indexing because Google indexes high-volume producers of content more frequently than the average corporate Web site. But business or government Web sites, which aren't typically indexed several times daily, are more likely to appreciate a button to summon Google's search crawler, he said.