The Test System That Wasn't

[Editor's Note: Please welcome our new IT Life columnist, Grumpy IT Guy. We are keeping his real name secret to let him speak the unfiltered truth to power and the clueless. This sage veteran leads an IT organization somewhere in America. Ask him questions using the comments field below. You may not like the answer, but you may need it.]

Grumpy IT Guy: We have a system that was put in years ago. Company paid big money for enterprise app vendor to install a parallel separate test environment. Best-practice, right? Test environment means stupid vendor patches kill test environment, not production.

But recently, sysadmin gets a patch. Sysadmin installs the patch into the "test" environment. Boom! Production environment for hundreds grinds to a halt. Which part of "separate test environment" did our vendor not understand?

Vendor: These things happen.

Grumpy IT Guy: Are you serious?

Vendor: Sorry?

Grumpy IT Guy: Test systems should NEVER blow up production systems.

Vendor: Oops.

Grumpy IT Guy: So what are you going to do about it?

Vendor: Try again?

Grumpy IT Guy: NO.

Vendor: Make a new patch and try again?

Grumpy IT Guy: NO.

Vendor: Surely you don't want us to stop randomly throwing code over the fence to you.

Grumpy IT Guy: That would be nice.

Vendor: Surely you don't want us to actually figure out how the two systems got connected and how to separate them.

Grumpy IT Guy: Arrrrgh!

[Do you love your job? See What Willie Nelson Taught Me This Summer.]

Question is now whether the vendor is going to charge for system reconfig, or if my system admin is going to do it. The vendor is optimistic -- would be glad to charge us to rebuild everything. So now the quizzing down of the sys admin begins.

Sysadmin: Um, can I have money to pay the vendor?

Grumpy IT Guy: How could this happen?

Sysadmin: I don't know.

Grumpy IT Guy: Isn't it your system? Why don't you know?

Sysadmin: I don't know why I don't know.

Grumpy IT Guy: Why wasn't this a problem before?

Sysadmin: Do you want me to guess?

Grumpy IT Guy: Did something change? Did we do something? Did the vendor do something?

Sysadmin: Um.

Grumpy IT Guy: Could you, maybe, kind of... FIND OUT?

Sysadmin: (Flees)

Who wants to take bets about what the problem was?

Will it be the database of doom, where someone decided to migrate a database to the wrong server? Will it be the application server of annihilation, where someone bonked the load balance order? Will it be the Web server of woe, where the change manager of confusion approved the wrong Web wrangling? I guarantee you that it will be human error, regardless of the vendor and the system admin vocalizing things like "software issues" and "load balancer problems." I know they privately agree with me, because they are both doing some low-key finger pointing at each other.

Will it never end? Take your guess in the comments section below.

If the world wasn't changing, we might continue to view IT purely as a service organization, and ITSM might be the most important focus for IT leaders. But it's not, it isn't and it won't be -- at least not in its present form. Get the Research: Beyond IT Service Management report today. (Free registration required.)