The best companies stare their IT mistakes straight in the eye.

They don't tiptoe around them. They don't rename them "teaching moments" or "issues." They don't play blame games. They lay their mistakes bare so that their teams can improve.

They talk about their IT mistakes in the way CenterPoint Properties CIO Scott Zimmerman describes his team's first mobile application development. "We had to refactor because our egos caused us to forget the golden rule of software development: Always involve the users," Zimmerman says.

Mistakes are at the center of any business technology innovation--swing for big wins, sometimes fail, learn and improve.

We embraced this idea as part of this year's InformationWeek 500 by asking every IT team to share its biggest mistake this year. Not everyone was comfortable discussing them publicly--the wound is too fresh for some, and many are still working on the fix. What follows are real-world examples from IT teams willing to share.

Mistake: Treating mobile development like desktop development

CenterPoint is a developer and manager of industrial property, and it has some veteran application developers who know the business well. The kind you want. But when it came to building a mobile app, that knowledge of the business recently got the better of the IT team. CIO Zimmerman describes it this way:

"So when we created our first mobile app, we wanted to surprise/impress our business colleagues. We let enthusiasm cloud our judgment, trying to think through requirements for them. Initial adoption was disappointing. It 'looked cool' but didn't meet their 'mobile' needs. We failed to recognize that people often perform business functions differently in the field than at their desk."

That realization forced the team to redo the app, and it's what reminded the IT team of that golden rule of software development, and the importance of working closely with the application end users. The app is back on track.

Mobile app dev is serving up a lot of tough lessons. Few enterprise IT teams have deep experience with the technology and languages. And there's almost always an impossible time pressure, as marketing wants that mobile app this quarter rather than next year.

Vail Resorts CIO Robert Urwiler says he made the mistake of outsourcing too much of the resort company's customer-facing mobile Web app, EpicMix, in its first year. In year two, he built an internal team for much of that work (see Vail Takes Sharp Turn On Dev Outsourcing Strategy). Creating customer-facing software also demands that developers have a tight relationship with business units, especially marketing. "It's not like 'You do the idea generating, and we'll build the software,'" Urwiler says. "We need to be joined at the hip throughout these initiatives."

Mistake: Assuming everyone wants a social network inside a company

Several InformationWeek 500 companies noted initial lackluster adoption of employee-focused social networking and related collaboration tools. Some, like online game company Zynga, turned that reluctance around, while others say the software still isn't delivering. Zynga put it this way:

"We anticipated that internal collaboration tools would gain adoption faster than they did. We've made significant improvements on that front but learned a lot in the process. The biggest reminder for us was that content is key to driving internal adoption of enterprise social network tools."

What's refreshing about Zynga's experience is that it comes from a company on the cutting edge of the social Web, whose games are intimately linked with Facebook and social sharing. The Zynga team lives and breathes the social concept, but it still needed vital content in an in-house social platform to make the tools relevant.

That's an important reminder if IT is tempted to blame end users for not hopping on new social collaboration tools: Employees are old, they don't get it, they're not tech savvy. Those are all cop-outs for not providing employees with a compelling reason why a social tool is a better, more efficient way to collaborate.

Mistake: Using outsourcing the wrong way

Several companies cited outsourcing problems as their big mistake. Sometimes it was execution, such as the outsourcer not having the promised skills or capability. Vail's Urwiler, as mentioned above, felt he handed over too much project management. One admitted to moving too much to outsourcers too fast. With all of the outsourcing mistakes cited, we're seeing IT leaders get more demanding of what they expect from their vendors.

Edward Wagoner, CIO of the Americas for Jones Lang LaSalle, is fine-tuning his company's outsourced IT help desk strategy to better fit the commercial real estate company's culture. In commercial real estate, even as digital marketing becomes more important, it's still the top talent who drive growth, and these rock-star salespeople thrive on relationships.

Explains Wagoner: "We bought into the theoretical concept that certain IT functions such as help desk support could be 'black boxed' (e.g., that the end user wouldn't care who the help desk person was or where they were located as long as the problem was resolved). We failed to realize that our top talent values that personal relationship in all areas of their work life. They want to know that Donna is their IT support person. They want Donna to understand their business and the way they work. They want Donna to be a part of their extended team. They value having a relationship with Donna vs. calling an 800 number."

Wagoner has started assigning individuals to support key people and offices. Those outsourced support people have the go-ahead to ignore certain productivity metrics if it means solving a problem--"think airline platinum" status, Wagoner says. Salespeople have even called him to thank him for the support. And the relationship building continues.

Outsourcing may or may not be right for a given company. But regardless, IT needs to consider its company's unique culture and business model, and bend the outsourcing model to its needs.

Mistake: Underestimating the upside

Innovation often happens only after putting IT tools in employees' hands and watching what they do with them. Nice in theory, but it takes flexible IT operations to react to the new demand that such a process can create.

Hertz came to this realization when it created its ExpressRent kiosks, which let people make a rental car reservation using a live video call to a customer service rep, along with a touch screen to enter data, a scanner to validate a driver's license, and a printer to provide a rental agreement. But Hertz's IT team had to scramble when demand picked up in an unexpected market. Says Hertz:

"We originally marketed the ExpressRent kiosks primarily to body shops and dealerships for insurance replacement rentals, assuming usage would be greatest for one-off rentals outside of business hours. We hadn't anticipated the airport market, already fully staffed and with a vast majority of 'existing' reservations, would be the success point for implementation. Modified deployment planning and additional kiosk refinements needed to be made in addition to securing airport authorities' approval for placement."

Quintiles, which runs clinical trials for biotech companies, launched a cloud-based service for drug development in 2011. Its business plan predicted that only small biotech companies would use the service. But some of the industry's largest biotech companies showed interest, asking for a scale of deployment for which Quintiles didn't have the people or technology, forcing it to scramble to ramp up.

Success is a nice problem to have, but it can still test IT's ability to deliver results.

Mistake: Struggling to get off legacy systems

A surprisingly large number of CIOs described how hard it has been to get off a legacy system. Some problems involved moving to cloud software, including integrating systems from different vendors to replace a single legacy system. Some described employees who relied on specific features or databases tied to the old system, requiring process and cultural changes to switch software and get off those functions.

One IT team, asking that its company not be identified, described its change management challenge:

"We began a legacy system replacement early last year. Going into one of our larger locations, we thought we were cruising. But we had failed to truly understand the complexity of management reporting they had developed internally over the years. So when we swapped the system, all their custom reporting broke. It was big-time scramble mode to review, design, and build the reporting they were expecting. In all, a six-month project delay due to insufficient discovery. Big lesson learned for our project management team. And we left with a better design headed to the next location."

IT will rarely be hailed as the hero at first for moving the company off a legacy software system. Such moves force employees to change, so they resist. As this example shows, processes and functions can break during the process.

But for every problem we heard about moving off legacy systems, we heard another say their biggest mistake was not moving off legacy systems fast enough as they brought on new systems, sapping the new project's return on investment.

Mistake: Taking on too much

In a healthy business, there's always some tension caused by business units asking for more than IT can deliver. IT is a constrained resource. But many IT leaders cite mistakes in managing the demand pipeline, either by not communicating and setting priorities well with the business units, or just taking on too much and crushing their teams.

One CIO, who asked that his company not be identified, says a lot of projects were pushed to the back burner during a three-year ERP project, which the company finished in 2010. So 2011 "was a year where pent-up demand for systems reached a peak," he says. The company launched a website, a customer loyalty program, and more in the first quarter of 2011. The logjam became clear in the following months, and IT met with each main business unit in the third quarter to better set priorities and expectations.

Mismanaging IT demand badly can lead to poor-quality software and systems. But even when IT does deliver strong results, leaders must recognize the very real human toll. Cincinnati Children's Hospital Medical Center recognized that risk as it raced to innovate:

"In 2011 our organization took on a number of high-profile and strategic projects that required significant time and dedication from our staff," the hospital says. "Although we successfully completed all of these projects, we risked staff burnout in asking our staff to go above and beyond on multiple occasions throughout the year."

Mistake: Getting into a virtualization rut

Server virtualization has been the golden goose of technology ROI. By getting higher utilization from servers, IT shops have been able to consistently lower data center costs and in some cases avoid the multimillion-dollar capital expense of building a new data center. But even this money machine has its limits.

HCA Information Technology & Services found that limit. The group provides IT services to the 150 HCA hospitals, hundreds of physician offices, and non-HCA healthcare groups as a revenue center for the company. Here is its experience:

"We began server virtualization in 2009, and as our successes increased we began to see virtualization as a hammer searching for a nail. In 2011, we discovered several scenarios where we had chosen virtualization hastily, or we combined environments that did not match logically. All of this required us to take a step backward and to develop a more defined methodology and framework to decide which applications were right candidates for virtualization. We believe our move into internal cloud services will require us to learn from our virtualization days and ensure we have a highly defined selection process."

Plenty of companies have broken applications as they virtualized servers. As companies get more aggressive by building private cloud environments that virtualize computing, storage, and even networks, the transition will get more difficult. FedEx, for example, has created an entire private cloud data center and will only move applications into it once the company has rewritten these apps so that they're optimized for that highly virtualized environment.

IT organizations tend to think of a private cloud as the safe choice--all of the flexibility and security of a public cloud such as Amazon's EC2, but without the security worries. But plenty of gotchas await.

Mistake: Underestimating mobile demand

IT didn't see the iPhone revolution coming. It wasn't until top execs and salespeople started bringing them in and demanding to get corporate email on them that IT took on the security problems iPhones created. So lesson learned, right? IT would be all over the iPad?

Not really. IT did start experimenting with tablets right away, but many teams thought it would be at best a niche device, maybe something that would start slowly with executives. IT organizations weren't ready for how soon and how many employees would crave iPads--and who could make a credible case for why tablets make sense for work.

A number of healthcare companies, in particular, were caught flat-footed by how many doctors insisted on using personal tablets. They've told us of extraordinary steps that "maverick docs" took to use iPads, like using apps of questionable security or installing desktop access software on a tablet to view records from the doctor's PC.

Salespeople instantly saw the iPad's appeal as a better device to use while chatting one-on-one with a prospect. One financial company described its iPad experience this way:

"Mobile computing has been a key component of our IT strategy for several years. We've delivered on initial focus areas, building mobile applications for customers and enabling employee personal mobile device access to company email. Our mistake was in not anticipating the dramatic surge in popularity of the iPad commensurate with the release of the iPad 2. Sales force demand to leverage company-liable tablets rose suddenly, requiring us to be uncustomarily reactive. IT quickly bridged the gap, setting policy and implementing mobile device management, which enabled the company to mitigate financial impacts. However, we're still working to regain the full confidence of our sales executives."

Sales executives are going to need IT's help to get the most out of tablets. Companies that roll out tablets quickly learn that they need content and applications fine-tuned to their specific needs, such as for accessing inventory available for sale and providing tablet-friendly presentations.

But this company's reaction is a pretty good guide to bouncing back from any of these mistakes. Fix the technology, build a platform for future innovation--and spend time strengthening IT's relationships so that mistakes are much less likely to happen in the future.

Go to the 2012 InformationWeek 500 homepage

InformationWeek: Oct. 8, 2012 Issue

Download a free PDF of InformationWeek magazine
(registration required)