Goldman Sachs Puts Elasticsearch To Work - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Enterprise Applications
09:06 AM
Connect Directly

Goldman Sachs Puts Elasticsearch To Work

Wall Street financial services firm Goldman Sachs benefits from a broader use of open source search engine. Learn how its programmers use it on code projects.

Java: 7 Powerful Features For The Future
Java: 7 Powerful Features For The Future
(Click image for larger view and slideshow.)

Enterprise search systems have been available for many years and performed valuable functions in text and content searches. More recently, enterprise users have had the choice of powerful open source systems, many based on Apache Lucene, that can do broader tasks.

Goldman Sachs has adopted one of them, Elastic's Elasticsearch, and put it to use in innovative ways. Elasticsearch reaches into text sources, but Goldman software engineers are building applications that make use of its data retrieval powers as well as its large capacity for unstructured data.

"Elastic has been one of the most interesting open source products that we've seen in the last couple years," said Don Duet, global co-head of the Goldman Sachs technology division, in an interview with InformationWeek. "What's impressive about it is how much value it can create in organizations."

Elasticsearch and its co-products -- Logstash, Elastic's server log data retrieval system, and Kibana, a dashboard reporting system -- are written in Java and behave as core Java systems. This gives them an edge with enterprise developers who quickly recognize how to integrate them into applications. Logstash has plug-ins that draw data from the log files of 165 different information systems. It works natively with Elasticsearch and Kibana to feed them data for downstream analytics, said Elastic's Jeff Yoshimura, global marketing leader.

[ Learn how Goldman Sachs has played a leading role in Open Compute. Read Open Compute: More Financial Services Firms Jump In. ]

The Goldman Sachs technology division has put Elasticsearch to several innovative uses with minimal staff time invested. Examples include applications to help the legal department with contract searches, to enable executives and clients to track trades, and to assist engineering teams in locating and eliminating software bugs.

In the past, when Goldman wanted to check all its legal contracts for a particular clause or wording, the task could have required hiring platoons of lawyers to manually go over thousands of paper documents. Instead, a software engineer in Duet's organization built a system that first digitized each contract, using Apache Tika content analysis and optical character recognition software.

(Image: Nastco/iStockphoto)

(Image: Nastco/iStockphoto)

Tika was able to recognize more than 1,000 file formats and extracted metadata useful for generating search engine indexes. Elasticsearch then was fed all the contract documents. If the required terminology wasn't found in an Elasticsearch review of the contract, it was flagged for revision by company lawyers.

Duet said a single technology division engineer could create such a system because Elasticsearch has a RESTful API interface and functions much like a typical Java application. Some enterprise search offerings wouldn't necessarily fit in with the same ease that Elasticsearch has because they use their own programming paradigms and conventions that must be learned by IT.

Duet's organization built another Elasticsearch application for tracking trades throughout their lifecycle. "There are many different applications and server logs involved in the process of executing trades," he noted. Goldman's trade tracker application functioned something like a UPS package tracking system and was able to report to executives or to clients on the status of a given trade.

The trade tracker system could get data from different systems, consolidate it in Elasticsearch's key value store system, and then search on it for meaningful data. That meant different technical teams didn't need to be convened to extract data and figure out how to integrate the information from different systems.

Goldman has incorporated Elasticsearch into how its software developers work, with more than 700 of them having access to a search-based code management system. When a bug is found in one version of a piece of software, Elasticsearch can comb through the code library and find all instances of the bug.

Here, Elasticsearch works with Kibana, which builds dashboard reports on the status of projects and code that developers are working with. It captures source code changes, compares the "before" and "after" version of code, and can search for a snippet of code wherever it occurs. Code comments, reference designs, and documentation can all be pulled together through the power of the search engine.

Duet said technology division managers were able to spot Elastic when it first started appearing in the company's software asset management system. The new open source code became a frequent topic in emails and in chats on the programmers' social networks. Usage jumped from a few copies to 50 copies to 200 copies, and the technology division decided to make it widely available as authorized software throughout the company. It also contributed to the Elasticsearch project, engaged with Elastic engineers, and obtained a technical support contract for the search engine.

In addition to the technology division's software engineers and developers, Elasticsearch is sometimes used as part of a system by the financial engineering group doing deep financial modeling, Duet said. Operations staff can also use the Kibana dashboards to quickly build reports that used to require painstaking manual builds.

Goldman Sachs has 9,000 employees actively using technology and "several thousand" of them are now using Elasticsearch, Duet said, either as developers of new applications or as users of existing ones. That's a broader role for search than before, when enterprise search was limited to conventional keyword searches on text and content.

With Elasticsearch, Goldman is showing how versatile and useful search can be as a general purpose service inside the company and as a service built into many different types of applications.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
10/31/2016 | 1:43:28 PM
New standard for enterprise search replacing the google search appliance
Elasticsearch provides the new standard for replacing legacy systems like google search appliance, fast, autonomy, endeca and verity. Elasticsearch based systems like SearchBlox offer a ready to implement application to replace their legacy search applications.
User Rank: Apprentice
8/18/2015 | 5:23:12 PM
Open source search a growing pantheon of products
In addition to Elasticsearch, there are a number of search engines based on Apache Lucene: Solr, LucidWorks, Swiftype, Index Tank and Consillio, to name a few open source examples. Lucene was developed by Doug Cutting, who went to create the Web crawler, Nutch, and then the project for which he's most noted, Hadoop.
User Rank: Strategist
8/17/2015 | 10:49:32 PM
A new way to increase the IT budget
Ariella, Agreed. Anytime you can replace a platoon of lawyers with a single programmer,  the IT budget should get a 2% increase. Charlie
User Rank: Author
8/17/2015 | 10:45:13 AM
sopunds like a great step toward efficiency, though I'm sure the platoons of lawyers will be disappointed to have their hours cut.
The Best Way to Get Started with Data Analytics
John Edwards, Technology Journalist & Author,  7/8/2020
10 Cyberattacks on the Rise During the Pandemic
Cynthia Harvey, Freelance Journalist, InformationWeek,  6/24/2020
IT Trade Shows Go Virtual: Your 2020 List of Events
Jessica Davis, Senior Editor, Enterprise Apps,  5/29/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Key to Cloud Success: The Right Management
This IT Trend highlights some of the steps IT teams can take to keep their cloud environments running in a safe, efficient manner.
Flash Poll