SEC Makes Big Data Push To Analyze Markets - InformationWeek
Software // Information Management
11:07 AM
Connect Directly

SEC Makes Big Data Push To Analyze Markets

Securities and Exchange Commission's new Midas system collects information on all trades in an effort to better understand how the markets work.

Despite the initial emphasis on high-frequency trading, however, Berman says that Midas has a "much wider" focus. Midas could prove invaluable in the SEC's efforts to ensure more data-informed policy-making across the board. "As you can imagine, we hear lots of diverse opinions," Berman said. "Sometimes you get a nice 20-page treatise on why X is the worst thing in the world, followed by a nice 20-page treatise on why X is the best thing in the world. Our own independent analysis can contribute to the debate directly," he said.

Midas won't be able to fill in all of the current holes in SEC's vision. For example, the SEC won't be able to see the identities of entities involved in trades and Midas doesn't look at, for example, futures trades and trades executed outside the system in what are known as "dark pools." However, even without this data, Midas might serve as the necessary foundation for doing much wider analyses in the future.

Midas is a hosted system, so the SEC can focus on data analysis, not on keeping the lights on. The system is hosted with SEC's vendor, Tradeworx, which is both a high-frequency trading technology vendor and a trading firm itself. Tradeworx uses the cloud -- Amazon Web Services -- to help power Midas. Services used include Amazon S3 storage, Amazon EC2 infrastructure-as-a-service, and Amazon Elastic Block Storage. "The larger your data sets become, the more difficult it becomes to analyze," said Tradeworx CEO Manoj Narang. "This is why Amazon makes an enormous amount of sense. You can fire up 100 servers arbitrarily to analyze data in parallel."

Midas' underlying analytics platform was homegrown by Tradeworx and engineered for fast data transfer and rapid data analytics. "There are a lot of optimized calculations, a lot of work on parallelization of calculations," said Narang.

To the data analyst, Midas isn't fronted by a fancy Web app. Rather, it's accessed through a Unix shell, and the interface requires knowledge of scripting. Berman rattles off the tools of the trade: C++, Perl, Python. Tradeworx has also thrown in its own API and a series of commands to help the SEC manipulate data. "It's sort of like Legos," Berman says. "If you know how to work with Legos, you can build amazing things."

Although Midas isn't graphics heavy, there are rich visualizations to help analysts dig deeper into the data, and Narang said that future updates to the system will focus on rolling out more visual tools so that users don't have to understand Unix to use Midas.

The capability in Midas isn't necessarily novel. Other companies such as Nanex have said that they can offer similar functionality. However, it's the first time that the SEC has had such power at its fingertips, which means that the agency needs to make sure it has the right people in place to make use of the data. Now that the system is live, Berman is on the lookout for that talent. "We brought a few people in, but we need C++ programmers, algorithmic high-frequency trading people, people who worked on trading desks working on models," he said. "We're looking for people with a level of technical and market expertise that's somewhat unprecedented."

Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR02 by March 2 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for Interop today!

2 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of November 6, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll