Measuring The Value And Success Of SaaS

We look at real-world strategies to measure SaaS payback.

Michael Biddick, CEO, Fusion PPT

May 14, 2010

7 Min Read

Every CIO worth his or her salt knows that it's critical to gauge the potential impact that migrating to SaaS will have on the enterprise. But once SaaS is implemented, it's also important to measure whether it's delivering the value promised by providers, including reduced capital costs, faster application deployment, and reduced day-to-day IT support and operations tasks. CIOs also must determine if the SaaS application meets user and business needs in terms of features, capabilities, and performance.

This article will discuss ways that organizations can gauge where SaaS meets--and fails to meet--expectations and needs, including insights from our own experience as a SaaS consumer.

At our boutique IT firm, Fusion PPT, we recently decided to use SaaS for all of our business applications, from e-mail and collaboration to billing and application tracking. We use eight SaaS applications from seven vendors. Overall, we're pleased with them, but there are several areas we continually monitor.

Total cost of ownership is a key measure. SaaS providers tout the low acquisition costs of a service because customers don't have to make significant capital investments to get the app running, but SaaS costs will grow as you add users, buy into new features, and store more data with the provider. That's why you should revisit your original TCO analysis every 24 months.

When we calculate the TCO of SaaS, we evaluate both new and existing systems. For new systems, we compare the purchase price of the on-premises application to the SaaS application, costs of maintaining the application (including maintenance fees), as well as staff costs to support the application and infrastructure connected to it, such as backup and security systems. For existing applications, we also add the cost of migrating to the SaaS app and away from the existing one, especially extraction, validation, repair, and transformation of data costs.

In general, we've found that the TCO for on-premises utility applications (that is, applications that don't provide a compelling business differentiator, such as e-mail) are higher than the TCO for SaaS-based utility apps. In situations where we want to configure and customize an application, we've found more TCO parity between SaaS and traditional applications.

We also keep a close eye on support costs. While traditional software support isn't required with SaaS, we still have to add users, change passwords, create workflow, and perform other tasks. At Fusion PPT, these tasks fall to the business managers that use the SaaS apps. So far the workload is manageable, but we do monitor it so the business managers don't end up spending more time supporting SaaS systems than actually running the business.

We've also taken on a few unanticipated support costs. For instance, we decided to use a third-party SaaS backup provider for an additional layer of data protection on top of the redundancy we get from the primary SaaS vendors. We do this to ensure that if a SaaS provider suffers a massive catastrophe, our data will be protected. This choice has changed the original TCO models we developed, though only by less than 5% of the total support costs we anticipated across all of the SaaS providers.

Cost isn't the only measure we use to evaluate SaaS. Another important consideration is provisioning--that is, how quickly we can get the application into our users' hands. SaaS has a clear advantage over on-premises applications in this regard. Fact is, our users can start using a SaaS application within hours. By contrast, having to build out and deploy an on-premises application can take weeks or months.

Of course, it doesn't matter how fast you can get your users on board with an application if it lacks the features you need. When we were evaluating billing applications, we ran a conventional analysis to determine if the SaaS providers on our list offered the right set of capabilities. We chose a provider that met our needs at the time, but our analysis didn't anticipate a billing requirement we ended up needing to meet a demand of one of our own customers. The result: We had to migrate from one SaaS provider to another. We spent a substantial amount of time configuring and setting up the new SaaS system--more than 200 hours. This is 100 times the investment we made with the first billing SaaS provider, to meet the new requirements.

Just because an application is delivered via a provider doesn't mean you won't have to spend time on customization and configuration. In our experience, SaaS applications such as e-mail, collaboration, and CRM have required only a few hours of configuration, but the SaaS billing application required a more substantial investment on our part to fully meet our business needs.

This leads to another post-deployment lesson learned: the importance of data portability. If you're dissatisfied with the SaaS provider's application or service, you need to be able to remove your data from the application. In the case of our billing app, the data was stored in a format that made it easy to port it from one SaaS billing provider to the other.

However, we've seen examples from our consulting engagements where data hasn't been portable, effectively locking in a customer to a SaaS provider. In addition, if you have to move from one SaaS provider to another, the customizations you made will be lost.

Another area we continually evaluate is data integration with disparate SaaS providers. While data integration in a non-SaaS environment also can be challenging, particularly around identity management and business process management, it's just as difficult with SaaS (and impossible with some providers). Currently, we're evaluating a cloud-based data broker that could allow us to exchange data among the SaaS providers we use. While this integration isn't vital to our business, it would help automate and reduce manual processes. If your business requires tight systems integration, you will need to invest in platforms to exchange data in a SaaS environment.

Maintaining Satisfaction Levels

Applications that are low cost and easy to deploy won't be any good if they don't keep users happy. We've had minor complaints with our SaaS providers, like our hosted SharePoint sites not notifying us when documents were updated, or our SaaS-based applicant tracking system not sending out automated e-mails to candidates. For each problem, we found the vendors extremely responsive and able to resolve the issue within the day--better than at many IT organizations. We also conducted no training on any of the systems, and surprisingly, we haven't had more than a handful of questions, all resolved within the team because someone else figured out how to do it. People love the ability to access SaaS apps from anywhere, and the intuitive feel of the interfaces.

Success with SaaS is also measured by events that don't occur, such as security breaches, major outages, and performance hits. For high-value systems like CRM and accounting, we seek vendors that comply with the SAS 70 Type II standard, and we review the vendors' security documents. A SaaS vendor's data retention and restoration practices must align with our own. Problems like system failure may be straightforward, but what about accidental deletion within the SaaS environment? How long are the backups to be retained by the SaaS vendor? If there's a disaster at the SaaS provider's data center, how long will it take the provider to restore data for customers? If restoration of data is requested by a SaaS user, how long will it take?

How much extra bandwidth you'll need to cover the additional throughput required for SaaS depends on the types of applications, but network failover and connectivity are critical. Be sure you've got sufficient network capacity to support your SaaS environment.

In general, we're pleased with our SaaS providers; so far, they've been responsive to our issues. But as the SaaS industry grows and providers need to support more end users, service quality could become diluted, even as customer demands for better, more customized services increase.

If our own satisfaction levels erode, we don't want to be held hostage. For this reason, it's critical to constantly evaluate your SaaS providers and hold their feet to the fire.

Michael Biddick is chief technology officer of Fusion PPT, a technology consulting firm, and a contributing analyst to InformationWeek Analytics.

You can write to us at [email protected].

About the Author(s)

Michael Biddick

CEO, Fusion PPT

As CEO of Fusion PPT, Michael Biddick is responsible for overall quality and innovation. Over the past 15 years, Michael has worked with hundreds of government and international commercial organizations, leveraging his unique blend of deep technology experience coupled with business and information management acumen to help clients reduce costs, increase transparency and speed efficient decision making while maintaining quality. Prior to joining Fusion PPT, Michael spent 10 years with a boutique-consulting firm and Booz Allen Hamilton, developing enterprise management solutions. He previously served on the academic staff of the University of Wisconsin Law School as the Director of Information Technology. Michael earned a Master's of Science from Johns Hopkins University and a dual Bachelor's degree in Political Science and History from the University of Wisconsin-Madison. Michael is also a contributing editor at InformationWeek Magazine and Network Computing Magazine and has published over 50 recent articles on Cloud Computing, Federal CIO Strategy, PMOs and Application Performance Optimization. He holds multiple vendor technical certifications and is a certified ITIL v3 Expert.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like

More Insights