Big Data // Big Data Analytics
News
11/12/2008
06:31 PM
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

Google Introduces Search Crawler Caller

The software's control panel allows companies using Google's service to index their Web sites on demand.

Google on Thursday plans to add a button to its Google Site Search control panel that allows companies using the service to index their Web sites on demand.

Google Site Search is Google's $100-plus/year search-as-a-service offering for public Web sites, not to be confused with the Google Search Appliance, which tends to be deployed behind corporate firewalls to index internal Web sites.

Previously, Site Search customers had no choice but to wait for Google's Web crawler to re-index their sites. This can take days. The exact amount of time depends on an algorithm that Google uses to calculate the interval between indexing sessions. Nitin Mangtani, lead product manager for Google Enterprise Search products, declined to provide details about how that interval is determined.

When new content is added to a Web site, it's effectively invisible to Google Site Search until it gets indexed. Waiting several days for this to happen may not be desirable. That's why Google is giving its Site Search customers the leash to its spider.

"We re-index the Web sites when they change but we don't give the end-user any control over when they want them re-indexed," said Mangtani. "With this release, that's what we're offering them."

Adobe uses Google Site Search on its Web site and the company recently tested on-demand indexing for the launch of Adobe Creative Suite 4.

"Google Site Search made it easy to implement search across our Creative Suite product line and online sites, and we are now able to index thousands of new pages and make them available to millions of users worldwide within hours," said Tanya Wendling, senior director for Learning Resources at Adobe, in a statement.

Google Site Search indexing has no impact on the information in the main Google Search index. "Any special indexing we do for business customers does not impact Google.com," said Mangtani. This also applies to the secondary search box that appears beneath some listings on Google search results pages. Though the secondary search box allows searches restricted to a specific site, it queries the main Google index rather than a Google Site Search index. (Secondary search boxes merely provide an alternate format to submit a query using the site: search operator.)

Mangtani acknowledged that some news sites may not need on-demand indexing because Google indexes high-volume producers of content more frequently than the average corporate Web site. But business or government Web sites, which aren't typically indexed several times daily, are more likely to appreciate a button to summon Google's search crawler, he said.

Comment  | 
Print  | 
More Insights
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - June 10, 2014
When selecting servers to support analytics, consider data center capacity, storage, and computational intensity.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join InformationWeek’s Lorna Garey and Mike Healey, president of Yeoman Technology Group, an engineering and research firm focused on maximizing technology investments, to discuss the right way to go digital.
Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.