InformationWeek: The Business Value of Technology

InformationWeek: The Business Value of Technology
InformationWeek Big Data Coverage
= Member Content
Facebook Twitter Share

E-mail | Print | Permalink | LinkedIn | RSS

How One SMB Slashed Disaster Recovery Time


Midmarket firm Hit Promotional cut its backup window from around 17 hours to two. The CIO explains how it streamlined disaster preparedness in its virtualized environment.




10 Everyday Android Apps For SMBs
10 Everyday Android Apps For SMBs
(click image for larger view and for slideshow)
Ask Hit Promotional Products CIO Eric Shonebarger about his biggest pain point and you'll hear a familiar answer: Data, and lots of it.

The branded promotional gear wholesale distributor has grown rapidly in recent years. At the same time, the business has increasingly asked IT to connect disparate systems--even if those systems were never intended to interact. Compounding the challenge: Shonebarger's 10-person team, most of whom are focused on development, serves 900 employees and counting. All told, Shonebarger's department must manage about 24 TB of data, triple the amount it had three years ago--no small chore for a midmarket firm.

"When you're dealing with both [unstructured and structured data], it's just a massive undertaking," Shonebarger said in an interview. "To make sure the data is up-to-date, to get it where it needs to belong at the right time, to store it, back it up, retrieve it--all of that is getting more complicated."

Virtualization is another part of the story--more than 90% of Hit Promotional's IT infrastructure runs virtually, and that will soon hit 100%. That wasn't always the case. The company first dipped its toes into the virtual pool back in 2007 when it turned an extra server into a virtual host to test the technology. "Before we knew it, we had 15 or 16 [virtual machines] running on it, and no redundancy," Shonebarger said. In 2009, Hit Promotional made a significant investment to take everything but its email system virtual, deploying a VMware environment on Dell servers, backed up to an EqualLogic storage area network (SAN). For all of virtualization's benefits, the transformation shined a light on some flaws in Hit Promotional's data strategy: Like plenty of other small and midsize businesses (SMBs), it tried to retrofit existing technology and processes to a new world order.

[ Automate your billing process. See Deadbeats 2.0: How SMBs Use Cloud To Get Paid. ]

"We were treating those virtual machines just like they were physical machines," Shonebarger said. That was particularly true of Hit Promotional's existing backup and recovery methods--and therein lies an early lesson learned: A significant change like moving from physical to virtual infrastructure has wider implications than a few servers or desktops. The company had been using Symantec's Backup.exec software, but Shonebarger encountered a significant problem after the virtualization makeover. The backup proxy required a direct connection to the storage logical unit number (LUN).

"That's really, really dangerous to have, say, a Windows box with full-blown access to do whatever it wants to VMware LUNs" Shonebarger said. In that scenario, based on an actual one that occurred at Hit Promotional, if an administrator were to provision more storage locally, those LUNs would appear in Device Manager as if they were the same as USB drives. "Hey, there's a 2 TB LUN here, would you like to format it? All it takes is a single admin to say 'yes' and your whole LUN's gone. And VMware's resilient enough that it won't even tell you there's a problem until it runs out of disk space, can't put stuff into cache anymore, and takes a complete dive on you," Shonebarger said.

The breaking point: Shonebarger realized that a total IT disaster requiring a complete restore would likely take days to recover from, a scenario his business couldn't afford. Hit Promotional tried out a few different options and settled on Veeam, which builds its backup and recovery platform specifically for virtual environments. The first key gain: In that total restore scenario, Shonebarger said data interruption to his users would be a matter of seconds rather than days. That's enabled by Veeam's vPower feature, which will run a VM directly from Shonebarger's disk-based backups while the SAN is restored simultaneously. It becomes a see-no-evil, hear-no-evil proposition for end users.

"[As] IT, we're sometimes aware of problems or mistakes that, if we can keep it from the users and they can still have their normal experience, they don't care if it takes me a day to recover from failure," Shonebarger said. "They just want to know: Is my data available, is the system up and running, and can I get my job done?"

Similarly, Shonebarger has cut down a 17-to-18-hour window for a full SQL database backup--which rendered the database effectively unusable--to around 2 hours. The Veeam platform has also enabled significant compression of Hit Promotional's growing data; the company is in the process of upgrading its SAN, but can currently back up its 24 TB to a target that only actually holds 16 TB, with 5 TB still free.

Shonebarger offers some parting shots of advice for fellow SMB execs: He beats the recovery testing drum, for one. In his view, too many SMBs back up data without ever pondering the total disaster scenario--and what it would take to recover those backups effectively and efficiently. More importantly, he thinks there's a common misconception that replication equals safety.

"Don't confuse high availability with backup and recovery," Shonebarger said. By that, he means deploying a replicant SAN, for example, and thinking your job is done. While redundancy can be a good thing, it's no use if the original data is bad. The aforementioned LUN-formatting scenario is one example in the bad data category, and there are plenty of others.

"That's not smart," Shonebarger said. "That can come back to bite you in a pretty bad way."

Most external hacks of databases occur because of flaws in Web applications that link to those databases. In this report, Protecting Databases From Web Applications, we'll discuss how security teams, database administrators, and application developers can work together to improve the defenses of both front-end Web applications and back-end databases to prevent these attacks from succeeding. (Free registration required.)


Federal agencies must eliminate 800 data centers over the next five years. Find how they plan to do it in the new all-digital issue of InformationWeek Government. Download it now (registration required).




InformationWeek encourages readers to engage in spirited, healthy debate, including taking us to task. However, InformationWeek moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. InformationWeek further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
Subscribe to RSS


Advertisement


Virtual Infrastructure Reports

report Informed CIO: VDI Snake Oil Check
You won't lose your shirt on a desktop virtualization initiative, but don't expect it to be simple to build or free of complications. This report examines the three biggest problems when developing a business case for VDI: storage costs, ongoing licensing, and the wisdom of prolonging the investment in PC infrastructure.

report Fundamentals: Next-Generation VM Security
Server virtualization creates new security threats while turning the hypervisor into a network black hole, hiding traffic from traditional hardware defenses -- problems a new breed of virtualization-aware security software tackles head-on.

report Delegation Delivers Virtualization Savings
IT can't-and shouldn't-maintain absolute control over highly virtualized infrastructures. Instituting a smart role-based control strategy to decentralize management can empower business units to prioritize their own data assets while freeing IT to focus on the next big project.

report The Zen of Virtual Maintenance
Server virtualization has many advantages, but it can also lead to chaos. Many organizations have unused or test VMs running on production systems that consume memory, disk and power. This means critical resources may not be available in an emergency: say, when VMs on a failed machine try to move to another server. This can contribute to unplanned downtime and raise maintenance costs. Easy deployment also means business units may come knocking with more demands for applications and services. This report offers five steps to help IT get a handle on their virtual infrastructure.

report Pervasive Virtualization: Time to Expand the Paradigm
Extending core virtualization concepts to storage, networking, I/O and application delivery is changing the face of the modern data center. In this Fundamentals report, we'll discuss all these areas in the context of four main precepts of virtualization.

report Virtually Protected: Key Steps To Safeguarding Your VM Disk Files
We provide best practices for backing up VM disk files and building a resilient infrastructure that can tolerate hardware and software failures. After all, what's the point of constructing a virtualized infrastructure without a plan to keep systems up and running in case of a glitch--or outright disaster.