ESC Labs IT Manager Finds The Right Chemistry With Cloud

5 Traits Effective IT Leaders Need

IT manager Tom White was trained as a chemist. For the technology pro, that's actually a very good thing, especially when you're in charge of IT for 100 chemists working in the largest environmental testing organization in the US.

Environmental Science Corp.'s Laboratory Sciences, also known as ESC Labs, is one of the most frequently tapped resources for environmental pollutant analysis.

Before going to work at his current position, White, who has a master's degree in chemistry, worked as a chemist for the US Defense Department. As he joined Environmental Science Corp.'s lab operation, he took an interest in the software that the chemists were using, and eventually took over responsibility for its operation. One thing lead to another. He understood intuitively how the internet was going to extend the research that chemists would have access to and provide a pathway over which the lab could deliver its results to customers.

"I had the good fortune to take over the labs' software systems in the late 80s," he recalls. He's been intrigued and happy to work on the transitions that have come along since. He thinks he's in the right place at the right time: A chemist who "can talk to and understand the needs of chemists who are in the business that we're in."

Currently, the systems that White's staff collects data from are scattered across six buildings on a 10-acre campus in Mt. Juliet, Tenn., outside Nashville. Some of the labs are identifying and analyzing the amounts of organic chemicals in air and water samples, looking for such things as pesticides, PCBs, or other hazardous materials.

As director of enterprise infrastructure, White knows the data is generated by gas chromatography mass spectrometry and other instruments, which are distributed around a campus with 100,000 square feet of buildings. The results need to be captured, uploaded to a central database, and put into reports documenting their compilation in such a way that will stand up in court.

The reports need to be delivered to the sample submitter, sometimes the US Environmental Protection Agency, as quickly as possible, said White.

"We are often involved in an emergency response to spills and their cleanup. We have to deliver our analysis in a rapid fashion," but one that's still "legally defensible for our customers," White said. Many times, the customer is a private company involved in a spill that knows time is money when it comes to responding to the situation.

An accidental release of chemicals into a source of public drinking water, for example, would place high pressure demands on the lab's test turnaround time.

The lab's ability to respond is one reason ESC Labs is the "largest single-location environmental laboratory in the US," White said. That's a mouthful but important because most labs are accredited for operation in the state where they're located or a handful of neighboring states.

ESC is certified to report results to customers in all 50 states.

That's partly the legacy of founder Arthur Schulert, a Vanderbilt professor and biochemist involved in early environmental issues. He founded the lab in 1970, before there was a federal Clean Air or Clean Water Act. His son, Peter Schulert, a former officer-of-the-deck on nuclear submarines, is the current CEO.

The labs are in operation seven days a week, with no downtime available on Sundays. Knowing how much tolerance 100 chemists have for a central data center outage prompted White to search for an alternative to his aging disaster recovery systems.     

Much of the data being generated in the labs is fed into ESC's home-grown, Lab Information Management System, "our cradle-to-grave handler of information for all our operational functions," said White.

The system has evolved constantly over 15 years, with much of its life spent on dedicated hardware. That system had its recovery system on alternative hardware, but testing it was a manual process and the lab was loath to shut down hardware in an all-out test of its disaster recovery copy. LIMS was virtualized several years ago under VMware's ESX Server and vSphere orchestration system, which ultimately paved a path to automated, offsite disaster recovery.

ESC used to back up its data to disk and store the disks offsite in a vault where they could be used to recover downed systems. Then it replicated data to another site, a more automated approach. "It was a step in the right direction but still not disaster recovery," noted White.

Then he built a secondary data center on campus and replicated data to it, with virtualized core systems located there in the event of a disaster in the primary data center. But that of course meant it was on-site, and any business handling hazardous materials needs an offsite facility.

ESC Labs' Tom White at VMworld 2016 in Las Vegas

Two years ago, after a year of on-site replication based on using VMware's Site Recovery Manager, he realized he could initiate a similar operation in vCloud Air and get both his data and virtualized systems offsite. He's created a DR system in a vCloud Air data center in Dallas, far removed from any disaster that might strike Juliet. LIMS and about 30% of his "most crucial" production systems are backed up in VMware's public cloud service, ready to be spun up if needed.

He'd like to see more systems backed up there. "It's a cost limitation -- how large a resource do we want to dedicate to this insurance policy," he said. "It's still a work in progress," he adds.

"I've not done a full-blown shut down and attempted failover recovery," he says, but could do so with less risk and less manpower than with the manual DR system. Spot tests tell him that VCloud Air Disaster Recovery gets his central database up and running quickly, updates the data with the latest available and connects it to LIMS and other systems.

[Want to learn more about how customers are using the VMware cloud? Read VMworld: Security, Network, Cloud Are Top IT Concerns.]

The labs' Microsoft Exchange system is located in another data center, a managed service run by Intermedia, and White said he's thankful to have that primary communication system hosted by a separate vendor.