OnSite: First American’s Optimization Initiative Ends ‘Blamestorming

The First American Corp., a data services and information provider to the financial services industry, had a clear IT goal: visibility into system performance from the customers' point of view. "As simple as that sounds, it's a very complex thing to accomplish," says Evan Jafa, chief technology officer. "At any given time, we need to understand how our customers are experiencing the performance and availability of our product versus the IT view, which is usually a bottom-up view of the world. That one objective changed our approach and selection of tools."

First American's business is to collect massive amounts of information related to financial transactions--background, historical, personal, credit, tax and transactional information--and put it in formats that can be easily accessed and digested in real time by financial services customers to support decision making and risk mitigation. "We're the Intel Inside," Jafa says. "We feed all the information into the loan and transactional systems at various mortgage companies and banks." Customers also include real estate agents and Wall Street brokerage firms.

Like most IT shops, First American had point solutions in place for managing databases, servers, networks, firewalls and so forth. These helped keep components of the IT infrastructure up and running, but didn't provide the desired visibility. Layering HP (formerly Mercury Interactive) Mercury BTO Enterprise software on top of these IT management tools provided that window into the customer experience. BTO, which stands for business technology optimization, lets First American emulate customer transactions from outside the firewall and follow them all the way through to the tables of the databases and all points back to delivery to the customer. Each point along the way has been benchmarked, so the software can report on which portions of the delivery are not available or not performing. Jafa and his staff test common transactions every five minutes.

The software was implemented gradually starting in 2002 and continued as First American went through a series of mergers and acquisitions. The final phase of deployment was completed in November 2006. "We no longer have swap meetings or 'blamestorming' sessions to discover what has transpired or where our problem is, because the tool points to the exact cause," Jafa says.

Although the goal of this project was to obtain a view of the customer experience, the software has been identifying hotspots within applications, databases, systems and networks. "If a transaction took 10 seconds and one second was on the network, one second was on a server, and suddenly we come to a Java bean that's taking six seconds, that tells us, this is a hot spot," Jafa says. The problem is quickly brought to the attention of those responsible so it can be corrected. Disruptions are addressed more quickly and service level agreements are more consistently met.

Ten people are dedicated to monitoring and managing Mercury BTO Enterprise. "This requires what I would call a knowledgeable person across the board who understands networks, systems, databases, applications and everything else," Jafa says. The team has created a "repeatable factory line" so that all new components of the IT infrastructure are covered by Mercury BTO Enterprise as they're installed. The company's 24/7 data centers used to have dozens of IT staff. That number is now down to no more than six staffers at any given time because the tool has automated much of the work.

Another benefit of the software is that it has helped IT align with Information Technology Infrastructure Library (ITIL) best practices, which Jafa considers an important framework and guideline for efficient operations. "ITIL is becoming inherent in our processes," he explains. "We think the majority of IT processes can become repeatable processes with the necessary checks and balances of quality control," something ITIL helps ensure, he adds.

What of those large IT departments that have yet to dedicate time and resources to ITIL? "They need to read the book Catch-22," Jafa says. "If you're constantly firefighting and never getting out of the circle, some high-level person needs to step out of the circle and see if there's a means of escape." He adds that automating processes is where he thinks the escape route lies. At First American, 120 managers, supervisors, directors and vice presidents have been ITIL certified, so across the board managers understand the need to standardize IT processes, speak the same language and discuss terms with clear definitions.

In the last two decades, The First American Corp. has undergone a transformation from title insurance company to one of the largest business information providers in the country with 2,100 offices. To handle the growth without degradation of information technology performance, the company began consolidating its operations and standardizing its IT processes with ITIL initiatives.

Four years ago, chief technology officer Evan Jafa realized that the point solutions in place for monitoring and managing IT assets were no longer providing enough insight to the IT staff. "Visibility is one of the primary objectives of understanding the nature of any event or the performance of the system, so it was the primary objective," he says. "But to define what visibility means is the trick in this process. We didn't want to start with visibility into elements of the infrastructure or applications. We really wanted to have visibility into what our customers are experiencing as they use our products."

First American established global centers of excellence for functional and performance application testing based on Mercury Quality Center and Mercury Performance Center software. It installed Mercury Business Availability Center to monitor business processes and underlying applications, to manage service levels and to resolve issues before they affect customers. It deployed Mercury IT Governance Center to manage IT compliance, to standardize IT processes and to automate ITIL processes such as release management for global ERP applications.

How easy was all of this to set up and integrate? "I've heard the promises of massive frameworks of IT management software that have taken 15 years to implement and still not produced results," Jafa notes. Mercury BTO Enterprise and ITIL adoption, on the other hand, helped provide visibility immediately.