Massive Upgrade Takes Planning - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Enterprise Applications
12:50 PM

Massive Upgrade Takes Planning

Visa USA this month successfully upgraded four data centers around the world in a carefully coordinated process

Nine months in the planning, a massive system upgrade of Visa USA's payment network--the largest in the world--went off without a hitch earlier this month. The upgrade involved some 120,000 code changes, 150,000 worker-hours, and 27 major enhancements and was carried out at four data centers around the world. Visa, which processes 100 million transactions a day, completed the upgrade at its data centers in Tokyo, London, and two on opposite coasts in the United States over a 24-hour period beginning at 1 a.m. on Oct. 2, Tokyo time. Each center was down for about two hours, beginning with Tokyo. During the time each center was down, local transactions were routed to the other three. The upgrades were timed to occur during the early-morning hours, when volume is at its lowest.

Visa upgrades its VisaNet system semiannually, in April and October. Each semiannual upgrade involves the months-long planning process, during which Visa's internal IT team receives requests from Visa member banks for new services and begins building the corresponding programming code. Visa makes a test bed available so the IT staffs of member banks can try out their own internal system changes against the VisaNet changes. A quality-assurance phase, lasting six to eight weeks, allows for fine-tuning before the upgrade goes into production.

Visa's network is unique in that it can't be brought down entirely, Thompson

Visa's network is unique in that it can't be brought down entirely, Thompson says.
This upgrade, the largest ever, included enhancements to speed the credit-card dispute-resolution process, simplify currency conversions, and update Visa's loyalty-points program.

Some 300 engineers in Visa's development centers, as well as operations staff in the data centers, participate in each upgrade. Although most of the upgrades take place automatically, the process is managed and monitored by a central command center in California. There are also local command centers in each of the data centers. Each center has systems to communicate the center's status and the progress of the upgrade.

The upgrades are designed so that no more than one center is taken down at a time. As each one is brought back online, it's synchronized to work with the others that haven't yet been upgraded. The upgrade is a delicate balancing act because two versions of VisaNet--the old and the new--are running simultaneously throughout the 24-hour period. For example, while London is down, Tokyo is running the new version and the two U.S. centers are still running the older version. The network is designed to process transactions originating from either the older or newer systems.

Although other real-time, high-availability systems are brought down for regularly scheduled maintenance, Visa's is unique in that the network can never be brought down entirely, says Scott Thompson, executive VP of technology solutions at Inovant, Visa's processing subsidiary. "In any other real-time environment, all four system images would be taken down at the same time, the new system would be installed, then the centers brought back one at a time," he says.

In addition to regularly scheduled maintenance, VisaNet undergoes occasional unplanned outages, on the average of once a month per center, Thompson says. The outages seldom exceed a few seconds, and the network experiences not so much as a blip in throughput. Each center has enough capacity to run the entire worldwide transaction load. So-called heartbeat messages, indicating that a center is ready to receive transactions from another center, are constantly being generated and transmitted throughout the network.

Earlier this year, during its annual weeklong stress test, Visa ran the system at a processing load of 6,200 transaction messages a second, a 20% increase over last year's peak on Dec. 24 of 5,200 transaction messages a second. Each card purchase requires about two transaction messages.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
2021 Outlook: Tackling Cloud Transformation Choices
Joao-Pierre S. Ruth, Senior Writer,  1/4/2021
Enterprise IT Leaders Face Two Paths to AI
Jessica Davis, Senior Editor, Enterprise Apps,  12/23/2020
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll