FINRA Commits Mission-Critical App To Amazon Cloud

Over the last 2.5 years, a major Wall Street regulatory agency, FINRA, has found a home inside the Amazon cloud. It believes that "cybersecurity is better in the public cloud than it is in private enterprise data centers," said Steve Randich, executive VP and CIO of the Financial Industry Regulatory Authority.

And what's good enough for the Wall Street regulators is good enough for many financial industry organizations. "What we've done has stirred a lot of interest… We've had big banks coming in to ask us about our experience," said Randich, as he took center stage to talk about the move at ReInvent, AWS' annual user and developer event held this week in Las Vegas.

Such is the state of the public cloud today. A key convert has the potential to set off a stampede. Randich isn't able to disclose who is thinking about following FINRA into the cloud. But Capital One has stated it's putting more of its operations and customer applications into the public cloud, according to Rob Alexander in an announcement.

For FINRA, the move to the cloud was not an experiment or pilot project. Its first move was putting its primary, mission critical system onto AWS, its market surveillance application that searches through the day's trades looking for suspicious activity.

"We collect all orders for quotes and trades – 75 billion events per day. We process every day all the information that Visa and Mastercard process in six months," Randich told a keynote crowd that filled a giant hall at the Sands Expo Convention Center in the Las Vegas Venetian Hotel.

Want to learn more about what happens when a market collapses? See NYSE, United Airlines Shutdowns Spark Paranoia.

FINRA runs sophisticated surveillance queries against the data, looking for insider trading and evidence of trades occurring before regulations allow them to. After the May 6, 2010 "flash crash" when unexplained events kicked off a precipitous drop in the market, FINRA was authorized by the Security Exchange Commission to collect even more data, which it is in the process of dong. The flash crash was "an unpleasant event," dropping the Down Jones Industrial Average for the day by 998 points. FINRA is charged with trying to prevent a repetition of that event.

FINRA holds all the data it collects for a minimum of five years. Historical data is essential because in some cases an order is placed and no action taken on it until a given stock or stocks reach a strike point, where the trade must be executed. Understanding what happened in a trade is hard to do if the order authorizing it lies outside the data set that investigators are working with.

(Image: EdStock/iStockphoto)

To discharge its responsibilities and continue to collect the immense amount of data that it works with  – "20 petabytes and getting larger," said Randich --  FINRA needed to stop supplying its own data center infrastructure and move into the cloud. The process started with planning to move the surveillance system in 2013. FINRA eventually devoted 500 IT staffers to the task and cut the application over in mid-2014. It eschewed hiring consultants or vendor expertise to do the job, developing the necessary knowledge of EC2 and skills to do it itself.

in an interview with InformationWeek after the keynote, Randich said that finding the right people to make the move wasn't a problem. "We started by letting them use the FINRA charge card (to establish an AWS account) and play with it. Then we began proofs of concept… people from all over the organization bought into it. Then they were brought together in a community of practice," he recalled.

Part of the later stages was to get all the training they could from Amazon. "We offered pretty much anyone the opportunity to get training," and with a sense that something big was in the works, more than enough FINRA IT staff was prepared to work with applications on EC2.

"It was very uncommon. We wanted to do it ourselves, not through a vendor," he added. That was because FINRA's future lay in the cloud and it had to have a staff with a depth of knowledge on how it worked.

Now it wants to move its registration system into the cloud as well. That means moving an Oracle database system from a FINRA data center into one of the 10 AWS relational database services, possibly Amazon Aurora with PostgreSQL compatibility, a new offering announced Nov. 30. The SQL used in open source PostgreSQL is compatible with that used in Oracle database applications, but FINRA hasn't made a decision on which Amazon option it will use, Randich said in the interview.

But the move to the cloud has proven cost effective. "We got some huge, pleasant surprises out of this  that we did not expect at all," he told the keynote crowd in a sprawling Sands Expo hall. ReInvent has 32,000 registered attendees this year.

At its peak, FINRA's own data center mustered 3,000 virtual servers, with the number of physical servers less than that. In the cloud, FINRA can burst out to 11,000 nodes or virtual servers at a time. Over the course of a day, it may use as many as 30,000 virtual servers in total, with the number rising and falling according to need.

The elasticity of the cloud means that queries that used to take hours can now be completed in minutes or seconds. Pre-cloud, it wasn't unusual for a FINRA analyst to set a query running, go to lunch, and hours after he returned, the query was still chugging through data. Performance is at least 400% better in EC2, he said.

Unusually heavy trading days used to generate so much data that FINRA's surveillance application might spend 2-3 days afterward processing all the data. Meanwhile, records from more current trading were piling up, waiting their turn until the weekend. In the cloud, a 20-minute query has been reduced to sub-second response times.

"Historically we had these bottlenecks," said Randich. By moving to the cloud, the FINRA investigative staff can generate more leads from the data, process more information, and supply a greater regulatory presence to the trading markets.

FINRA has 20 marketplace abuses to watch for. It can do a better job the more queries it can run against a given day's data. Andy Jassy, executive VP of AWS, said during a ReInvent keynote that FINRA's move to the cloud has been helpful to Amazon. "We've gotten an incredible amount of valuable feedback in our partnership with FINRA," he said.