The InformationWeek -- Blogs
Over The Air

Mobility Breifing Center -- Sponsored by Windows Mobile
Topics:   Digital Life : Google : Mobile : Startup City

  • Email this page E-mail this page
  • Print this page Print this page
  • Bookmark and Share
  • icon

Will Google Be Destroyed By Open Source Search Engines?


Posted by Stephen Wellman, Aug 10, 2007 07:07 PM

Could open source kill the golden egg that laid Google? If Wikia has its way, it just might.

The Wikia project, started by Wikipedia co-founder Jimmy Wales, seeks to turn the process of building a search engine from a multimillion dollar project to one that could cost just hundreds or thousands of bucks. That's a game changer.

Here is a look at what Wikia hopes to accomplish:

The project, which was started by Wikipedia co-founder Jimmy Wales, consists of four components, the indexing of the Web, developing a search engine application, an algorithm, and using people to help filter sites and rank results.

One of the most expensive components of a search engine is the effort needed to index the Web. Companies have to buy servers and software to crawl the Web looking at what's on every page, to create a comprehensive list of what's on the Web.

Well, how will Wikia be able to provide all this search engine technology and service -- especially crawling the Web -- for free? Open user participation, of course:

The cost of indexing the Web is one of the main hurdles to starting a search engine, and for-profit companies have raised the bar year after year by indexing the Web more and more often. It used to be catalogued once a week, or once a day. Now it's once an hour, or even more often. The high cost of running these crawls has become a competitive weapon.

Wikia believes its crawl of the Web will cost nearly nothing, because it's asking Internet users to help out by downloading Web-crawling software from Grub, which will use their computers during idle time to crawl the Web, and send results back to Wikia for the index. So far, a thousand people have downloaded the application, and Penchina is hoping for 100,000 or more. The goal is to post the entire index online, as well as regular updates, so anyone can use them.

If I have any skepticism about Wikia, it centers on this piece. I know that distributed computing and its white-hot offspring, grid computing, are big IT trends (and that they can work), but crawling the Web is the competitive advantage that Google, Yahoo, and Microsoft use to maintain their market share. Will a mishmash of random crawls from across the Web really be an adequate substitute to a centralized effort? We'll have to wait and see.

As for Wikia itself, I think that even if the distributed Web crawling doesn't work as well as Google, just having that option available -- along with open and free search engine parts -- will be the catalyst that both vertical search and the local search advocates have been looking for.

In May I predicted that Google would die not from direct competition with a new, direct rival but from the challenges posed by an army of thousands of tiny niche search engines and Web apps.

We've seen Technorati and Blinkx beat Google at its newer search initiatives. How many more little search engines will we see once Wikia goes live?

« Thumbs Surgically Altered For iPhone? Think Again | Main | Groklaw's Pamela Jones On The SCO Decision »



Sign up now for the weekly InformationWeek Blog Newsletter.


This is a public forum. United Business Media and its affiliates are not responsible for and do not control what is posted herein. United Business Media makes no warranties or guarantees concerning any advice dispensed by its staff members or readers.

Community standards in this comment area do not permit hate language, excessive profanity, or other patently offensive language. Please be aware that all information posted to this comment area becomes the property of United Business Media LLC and may be edited and republished in print or electronic format as outlined in United Business Media's Terms of Service.

Important Note: This comment area is NOT intended for commercial messages or solicitations of business.




Mobile Video


Sign Up For The Over The Air Newsletter
Every Friday, our experts and analysts explore the business, strategy, and management issues most important to mobile and wireless technology.

Sign up for our free, weekly newsletter today!

Newsletter Archives




  1. Twitter In Controversial Spotlight Amid Mumbai Attacks
  2. Google Round Up: Evil Layoffs, Chrome Speed Test, Street Views
  3. iPhone 2.2 Images Hit The Web
  4. iPhone Firmware 2.2 Breaking Some Apps
  5. Don't Shut Off Vista UAC, There's A Better Way


  1. Google Chrome Browser To Support Customization
  2. Cell Phone Users Increasingly Satisfied With Their Service
  3. Cell Phones More Distracting Than Chatty Passengers
  4. WiMax Future Remains Unclear With Clearwire
  5. Texas Instruments Ranked Top MID Platform Vendor
  6. Alcatel-Lucent Could Dump Mobiles To End Pain

 
 

  Ars Technica
Boing Boing
Channel 9 Forums
CRN Blogs
Dr.Dobb's Portal: Blogs
Engadget
Gizmodo
GrokLaw
  Lifehacker
Schneier on Security
Slashdot
TechCrunch
Techdirt
Techmeme
Valleywag

  SEPTEMBER 2008
AUGUST 2008
JULY 2008
JUNE 2008
MAY 2008
APRIL 2008
MARCH 2008
FEBRUARY 2008
  JANUARY 2008
DECEMBER 2007
NOVEMBER 2007
OCTOBER 2007
SEPTEMBER 2007
AUGUST 2007
JULY 2007
JUNE 2007