The Test System That Wasn't - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Life
Commentary
9/12/2014
08:07 AM
Grumpy IT Guy
Grumpy IT Guy
Commentary
50%
50%

The Test System That Wasn't

Why is Grumpy IT Guy extra grumpy today? It's crazy vendor day!

[Editor's Note: Please welcome our new IT Life columnist, Grumpy IT Guy. We are keeping his real name secret to let him speak the unfiltered truth to power and the clueless. This sage veteran leads an IT organization somewhere in America. Ask him questions using the comments field below. You may not like the answer, but you may need it.]

Grumpy IT Guy: We have a system that was put in years ago. Company paid big money for enterprise app vendor to install a parallel separate test environment. Best-practice, right? Test environment means stupid vendor patches kill test environment, not production.

But recently, sysadmin gets a patch. Sysadmin installs the patch into the "test" environment. Boom! Production environment for hundreds grinds to a halt. Which part of "separate test environment" did our vendor not understand?

Vendor: These things happen.

Grumpy IT Guy: Are you serious?

Vendor: Sorry?

Grumpy IT Guy: Test systems should NEVER blow up production systems.

Vendor: Oops.

Grumpy IT Guy: So what are you going to do about it?

Vendor: Try again?

Grumpy IT Guy: NO.

Vendor: Make a new patch and try again?

Grumpy IT Guy: NO.

Vendor: Surely you don't want us to stop randomly throwing code over the fence to you.

Grumpy IT Guy: That would be nice.

Vendor: Surely you don't want us to actually figure out how the two systems got connected and how to separate them.

Grumpy IT Guy: Arrrrgh!

[Do you love your job? See What Willie Nelson Taught Me This Summer.]

Question is now whether the vendor is going to charge for system reconfig, or if my system admin is going to do it. The vendor is optimistic -- would be glad to charge us to rebuild everything. So now the quizzing down of the sys admin begins.

Sysadmin: Um, can I have money to pay the vendor?

Grumpy IT Guy: How could this happen?

Sysadmin: I don't know.

Grumpy IT Guy: Isn't it your system? Why don't you know?

Sysadmin: I don't know why I don't know.

Grumpy IT Guy: Why wasn't this a problem before?

Sysadmin: Do you want me to guess?

Grumpy IT Guy: Did something change? Did we do something? Did the vendor do something?

Sysadmin: Um.

Grumpy IT Guy: Could you, maybe, kind of... FIND OUT?

Sysadmin: (Flees)

Who wants to take bets about what the problem was?

Will it be the database of doom, where someone decided to migrate a database to the wrong server? Will it be the application server of annihilation, where someone bonked the load balance order? Will it be the Web server of woe, where the change manager of confusion approved the wrong Web wrangling? I guarantee you that it will be human error, regardless of the vendor and the system admin vocalizing things like "software issues" and "load balancer problems." I know they privately agree with me, because they are both doing some low-key finger pointing at each other.

Will it never end? Take your guess in the comments section below.

If the world wasn't changing, we might continue to view IT purely as a service organization, and ITSM might be the most important focus for IT leaders. But it's not, it isn't and it won't be -- at least not in its present form. Get the Research: Beyond IT Service Management report today. (Free registration required.)

Grumpy IT Guy avoided historic disasters and clueless people while working his way up the IT ranks, but he retained his keen sense of humor. He now leads an IT organization somewhere in America, as part of the FBI's Grump Protection Program. Need advice? Send questions to ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Susan_Nunziata
50%
50%
Susan_Nunziata,
User Rank: Strategist
9/16/2014 | 6:51:38 PM
Re: It's Never the Vendor's Fault...
@dr2127dr: Are you seeing a more lax attitude about test systems from SaaS/Cloud vendors than you did from vendors selling on-premise systems? Or is this an across-the-board issue in IT?

And, to answer your question: when all of us old dogs bag it who is going to manage this kind of crazy behavior?

I'd say we are all doomed.
Susan_Nunziata
50%
50%
Susan_Nunziata,
User Rank: Strategist
9/16/2014 | 6:48:15 PM
Re: Here's one guess
@moarsauce123: I wish you were makiing up the part about the Sys Admin not knowing what Firefox and Chrome are....that is truly frightening. Excellent story, though. I did laugh out loud reading that, even though it must have entirely frustrated you when it happened.

But really, why can't we just make all our customers visit our website only using IE?

<grin>
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Author
9/13/2014 | 7:44:38 AM
Testing
The sad part is that good test processes do not begin and end with a testbed; vendors need to be doing more to test their products and patches.

As an IT old-timer told me in a recent interview: "One QA lady I knew rolled a Coke can across the keyboard as her first test—if your application failed to recover, you got it back right then."
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Author
9/13/2014 | 7:41:19 AM
Re: It's Never the Vendor's Fault...
@dr2127dr: I wouldn't worry.  That's how new dogs become old dogs -- by dealing with this kind of garbage!  ;)

David F. Carr
100%
0%
David F. Carr,
User Rank: Author
9/12/2014 | 10:48:38 AM
No, it will never end
Whatever went wrong, it's definitely not my fault.
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

News
Remote Work Tops SF, NYC for Most High-Paying Job Openings
Jessica Davis, Senior Editor, Enterprise Apps,  7/20/2021
Slideshows
Blockchain Gets Real Across Industries
Lisa Morgan, Freelance Writer,  7/22/2021
Commentary
Seeking a Competitive Edge vs. Chasing Savings in the Cloud
Joao-Pierre S. Ruth, Senior Writer,  7/19/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Video
Current Issue
Monitoring Critical Cloud Workloads Report
In this report, our experts will discuss how to advance your ability to monitor critical workloads as they move about the various cloud platforms in your company.
Slideshows
Flash Poll