Open source has become standard in Silicon Valley, with nearly every software startup planning to release at least some code. So far, they've found five main business models:
1. Sell support services. This is the traditional Linux model, prototyped by Red Hat. It's still a part of most open-source business plans, but on its own it's rarely enough for startups trying to grow. The problem (for the startups) is that anyone can redistribute the code and sell support or consultancy services, so there's nothing to stop an Oracle, an IBM, or a Novell from grabbing most of the services revenue.
2. Build (or run) hardware. Free software ought to make hardware more profitable, but relatively few open source companies have taken the hardware route. (Lots of hardware vendors use open-source in their products, of course, but they're not really open-source companies.) The main reason is that installing software on commodity components has even fewer barriers to entry than selling support, as VA Linux showed during the first bubble. Still, some startups have resurrected the idea, notably Vyatta (router) and SocialText (wiki appliance).
3. Proprietary components. Many startups now combine proprietary and open-source code, essentially holding back some functionality from what they release for free. The most successful to use this model so far was VMWare competitor XenSource (now part of Citrix), which gave away the Xen hypervisor but sold its proprietary management software.
Competitor VirtualIron does exactly the same thing, collaborating with Citrix on Xen but competing on management. XenSource's success has made this a popular strategy for other open source startups such as MuleSource (SOA) and Hyperic (systems management.) It also gives established software vendors a clear path to open source.
4. Dual licensing. Some customers just don't want to follow open-source licenses (usually the GPL), so many open-source vendors will happily sell them proprietary licenses for the same software. This works well for companies like Trolltech and MySQL, and it could become more popular thanks to new open-source licenses that place tighter restrictions on what other vendors can do for free.
For example, the limits on home DRM in GPL v3 are intended to make consumer electronics more open, but they could eventually give open-source companies a revenue stream from DRM vendors who want the code without the license. The (so far little-used) AGPL could have an even bigger impact, thanks to its requirement that SaaS users be able to download the source code. The big disadvantage for startups taking this approach is that they can't easily leverage community development, as they need to hold copyright on all code.
5. Advertising. The Mozilla Foundation discovered this almost by accident, when Google paid the Firefox developers so much in referral fees that they had to incorporate as a for-profit. It's also used (along with the other four) by Digium, the main backer of the free Asterisk PBX, which comes pre-configured to connect to particular IP telephony services. I expect we'll see more startups embrace this idea as SaaS becomes more common and ASPs offer big bucks for customer leads.