Evaluating Vendors? Kill the Spreadsheets - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
Commentary
3/16/2010
09:42 PM
50%
50%

Evaluating Vendors? Kill the Spreadsheets

Over the years, I've learned a hard truth: more details and more complexity in a scoring methodology may not deliver you the right vendor. It can occasionally do the exact opposite. Ideally you should dispense with spreadsheet scoring entirely in favor of a more practical, facilitated decision-making process.

A lot of the work at the Real Story Group involves helping people make the right information management buying decisions. Not surprisingly, one of the most common areas where our customers seek our counsel is with vendor scoring and assessment methodologies.

The Real Story Group strongly advocates a test-based approach to procurement, based on the value of scenarios; nevertheless, many enterprises want to apply a quantitative, spreadsheet-based assessment approach. In some cases these can become really quite complex. We have advised on dozens of scoring spreadsheets, and like to think we have seen them all.

And over the years, we've learned a hard truth: more details and more complexity in a scoring methodology may not deliver you the right vendor. It can occasionally do the exact opposite.Ideally you would dispense with spreadsheet scoring entirely, in favor of a more practical, facilitated decision-making process. However, the latter is not a universal panacea and needs to be carefully managed. I have seen qualitative approaches become too loose and subjective, and fail to achieve the best-fit solution.

In some cases you may have no choice but to formally justify your decisions with a quantitative rationale. The level of detail and particular approach to this sort of thing can vary. For example the public sector or highly-regulated industries might require more detailed scoring of shortlisted vendor RFP responses due to legal requirements, or quite simply the need to CYA.

If you have a 17-sheet spreadsheet with 238 lines of requirements, grouped into logical categories and scientifically weighted, you might feel you are doing a thorough job. But your scoring can only be based on what a vendor has supplied you in their RFP response document. And vendors are experts are filing RFP response documents.

Thus, you might be undertaking a detailed review, but not necessarily an accurate one. Some vendors outright lie (I have caught some out on occasion myself), many if not most vendors massage the truth a bit (or a lot). Some vendors are honest, and as a result are penalized heavily, leading to the victory of a rival vendor poorly qualified for the work, who did a great job of creative writing on their RFP response.

One of my personal favorites was a vendor who had scored top (by quite a margin) in a very big records management RFP shortlist. The buyer -- a research customer of ours -- grew a little suspicious and asked me to independently score the responses using their methodology. I scored the same vendor the lowest. Why? Well, simply because when the vendor answered "Yes, with scripting" I took that to mean "No, we cannot," whereas internal scorers scored those responses positively.

This gets to the other big problem with quantitative scoring: it tends to favor questions and responses about what a product can do (or what the vendor says it can do) rather than what almost always constitutes the real discriminator: how a product works.

In the end, if you are stuck with a heavily quantitative selection process, make sure at least that you don't short-cut the rest of your diligence. Make sure you speak directly to vendor references, and bring the vendor in to your premises to demonstrate the shipping product. Just remember you can encounter equally as much slight of hand when it comes to demonstrating products and providing "references" as vendors exhibited playing the detailed RFP game.

Getting the balance right between the right functional versus the right technical fit is hard enough. Finding a vendor you really want to work with, and that really wants to work with you, can be even harder. Take your time and do your homework.Over the years, I've learned a hard truth: more details and more complexity in a scoring methodology may not deliver you the right vendor. It can occasionally do the exact opposite. Ideally you should dispense with spreadsheet scoring entirely in favor of a more practical, facilitated decision-making process.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

News
Remote Work Tops SF, NYC for Most High-Paying Job Openings
Jessica Davis, Senior Editor, Enterprise Apps,  7/20/2021
Slideshows
Blockchain Gets Real Across Industries
Lisa Morgan, Freelance Writer,  7/22/2021
Commentary
Seeking a Competitive Edge vs. Chasing Savings in the Cloud
Joao-Pierre S. Ruth, Senior Writer,  7/19/2021
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Monitoring Critical Cloud Workloads Report
In this report, our experts will discuss how to advance your ability to monitor critical workloads as they move about the various cloud platforms in your company.
Slideshows
Flash Poll