The Test System That Wasn't

Why is Grumpy IT Guy extra grumpy today? It's crazy vendor day!

Grumpy IT Guy, IT Guy

September 12, 2014

3 Min Read

[Editor's Note: Please welcome our new IT Life columnist, Grumpy IT Guy. We are keeping his real name secret to let him speak the unfiltered truth to power and the clueless. This sage veteran leads an IT organization somewhere in America. Ask him questions using the comments field below. You may not like the answer, but you may need it.]

Grumpy IT Guy: We have a system that was put in years ago. Company paid big money for enterprise app vendor to install a parallel separate test environment. Best-practice, right? Test environment means stupid vendor patches kill test environment, not production.

But recently, sysadmin gets a patch. Sysadmin installs the patch into the "test" environment. Boom! Production environment for hundreds grinds to a halt. Which part of "separate test environment" did our vendor not understand?

Vendor: These things happen.

Grumpy IT Guy: Are you serious?

Vendor: Sorry?

Grumpy IT Guy: Test systems should NEVER blow up production systems.

Vendor: Oops.

Grumpy IT Guy: So what are you going to do about it?

Vendor: Try again?

Grumpy IT Guy: NO.

Vendor: Make a new patch and try again?

Grumpy IT Guy: NO.

Vendor: Surely you don't want us to stop randomly throwing code over the fence to you.

Grumpy IT Guy: That would be nice.

Vendor: Surely you don't want us to actually figure out how the two systems got connected and how to separate them.

Grumpy IT Guy: Arrrrgh!

[Do you love your job? See What Willie Nelson Taught Me This Summer.]

Question is now whether the vendor is going to charge for system reconfig, or if my system admin is going to do it. The vendor is optimistic -- would be glad to charge us to rebuild everything. So now the quizzing down of the sys admin begins.

Sysadmin: Um, can I have money to pay the vendor?

Grumpy IT Guy: How could this happen?

Sysadmin: I don't know.

Grumpy IT Guy: Isn't it your system? Why don't you know?

Sysadmin: I don't know why I don't know.

Grumpy IT Guy: Why wasn't this a problem before?

Sysadmin: Do you want me to guess?

Grumpy IT Guy: Did something change? Did we do something? Did the vendor do something?

Sysadmin: Um.

Grumpy IT Guy: Could you, maybe, kind of... FIND OUT?

Sysadmin: (Flees)

Who wants to take bets about what the problem was?

Will it be the database of doom, where someone decided to migrate a database to the wrong server? Will it be the application server of annihilation, where someone bonked the load balance order? Will it be the Web server of woe, where the change manager of confusion approved the wrong Web wrangling? I guarantee you that it will be human error, regardless of the vendor and the system admin vocalizing things like "software issues" and "load balancer problems." I know they privately agree with me, because they are both doing some low-key finger pointing at each other.

Will it never end? Take your guess in the comments section below.

If the world wasn't changing, we might continue to view IT purely as a service organization, and ITSM might be the most important focus for IT leaders. But it's not, it isn't and it won't be -- at least not in its present form. Get the Research: Beyond IT Service Management report today. (Free registration required.)

About the Author(s)

Grumpy IT Guy

IT Guy

Grumpy IT Guy avoided historic disasters and clueless people while working his way up the IT ranks, but he retained his keen sense of humor. He now leads an IT organization somewhere in America, as part of the FBI's Grump Protection Program. Need advice? Send questions to [email protected], or on Twitter: @IWGrumpyITGuy. You may not get the answer you want, but it's probably the answer you need.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like

More Insights