Google Introduces Search Crawler Caller - InformationWeek
Data Management // Big Data Analytics
06:31 PM
Connect Directly

Google Introduces Search Crawler Caller

The software's control panel allows companies using Google's service to index their Web sites on demand.

Google on Thursday plans to add a button to its Google Site Search control panel that allows companies using the service to index their Web sites on demand.

Google Site Search is Google's $100-plus/year search-as-a-service offering for public Web sites, not to be confused with the Google Search Appliance, which tends to be deployed behind corporate firewalls to index internal Web sites.

Previously, Site Search customers had no choice but to wait for Google's Web crawler to re-index their sites. This can take days. The exact amount of time depends on an algorithm that Google uses to calculate the interval between indexing sessions. Nitin Mangtani, lead product manager for Google Enterprise Search products, declined to provide details about how that interval is determined.

When new content is added to a Web site, it's effectively invisible to Google Site Search until it gets indexed. Waiting several days for this to happen may not be desirable. That's why Google is giving its Site Search customers the leash to its spider.

"We re-index the Web sites when they change but we don't give the end-user any control over when they want them re-indexed," said Mangtani. "With this release, that's what we're offering them."

Adobe uses Google Site Search on its Web site and the company recently tested on-demand indexing for the launch of Adobe Creative Suite 4.

"Google Site Search made it easy to implement search across our Creative Suite product line and online sites, and we are now able to index thousands of new pages and make them available to millions of users worldwide within hours," said Tanya Wendling, senior director for Learning Resources at Adobe, in a statement.

Google Site Search indexing has no impact on the information in the main Google Search index. "Any special indexing we do for business customers does not impact," said Mangtani. This also applies to the secondary search box that appears beneath some listings on Google search results pages. Though the secondary search box allows searches restricted to a specific site, it queries the main Google index rather than a Google Site Search index. (Secondary search boxes merely provide an alternate format to submit a query using the site: search operator.)

Mangtani acknowledged that some news sites may not need on-demand indexing because Google indexes high-volume producers of content more frequently than the average corporate Web site. But business or government Web sites, which aren't typically indexed several times daily, are more likely to appreciate a button to summon Google's search crawler, he said.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Digital Transformation Myths & Truths
Transformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll