Goldman Sachs Puts Elasticsearch To Work - InformationWeek
IoT
IoT
Software // Enterprise Applications
News
8/17/2015
09:06 AM
Connect Directly
Twitter
RSS
E-Mail
100%
0%

Goldman Sachs Puts Elasticsearch To Work

Wall Street financial services firm Goldman Sachs benefits from a broader use of open source search engine. Learn how its programmers use it on code projects.

Java: 7 Powerful Features For The Future
Java: 7 Powerful Features For The Future
(Click image for larger view and slideshow.)

Enterprise search systems have been available for many years and performed valuable functions in text and content searches. More recently, enterprise users have had the choice of powerful open source systems, many based on Apache Lucene, that can do broader tasks.

Goldman Sachs has adopted one of them, Elastic's Elasticsearch, and put it to use in innovative ways. Elasticsearch reaches into text sources, but Goldman software engineers are building applications that make use of its data retrieval powers as well as its large capacity for unstructured data.

"Elastic has been one of the most interesting open source products that we've seen in the last couple years," said Don Duet, global co-head of the Goldman Sachs technology division, in an interview with InformationWeek. "What's impressive about it is how much value it can create in organizations."

Elasticsearch and its co-products -- Logstash, Elastic's server log data retrieval system, and Kibana, a dashboard reporting system -- are written in Java and behave as core Java systems. This gives them an edge with enterprise developers who quickly recognize how to integrate them into applications. Logstash has plug-ins that draw data from the log files of 165 different information systems. It works natively with Elasticsearch and Kibana to feed them data for downstream analytics, said Elastic's Jeff Yoshimura, global marketing leader.

[ Learn how Goldman Sachs has played a leading role in Open Compute. Read Open Compute: More Financial Services Firms Jump In. ]

The Goldman Sachs technology division has put Elasticsearch to several innovative uses with minimal staff time invested. Examples include applications to help the legal department with contract searches, to enable executives and clients to track trades, and to assist engineering teams in locating and eliminating software bugs.

In the past, when Goldman wanted to check all its legal contracts for a particular clause or wording, the task could have required hiring platoons of lawyers to manually go over thousands of paper documents. Instead, a software engineer in Duet's organization built a system that first digitized each contract, using Apache Tika content analysis and optical character recognition software.

(Image: Nastco/iStockphoto)

(Image: Nastco/iStockphoto)

Tika was able to recognize more than 1,000 file formats and extracted metadata useful for generating search engine indexes. Elasticsearch then was fed all the contract documents. If the required terminology wasn't found in an Elasticsearch review of the contract, it was flagged for revision by company lawyers.

Duet said a single technology division engineer could create such a system because Elasticsearch has a RESTful API interface and functions much like a typical Java application. Some enterprise search offerings wouldn't necessarily fit in with the same ease that Elasticsearch has because they use their own programming paradigms and conventions that must be learned by IT.

Duet's organization built another Elasticsearch application for tracking trades throughout their lifecycle. "There are many different applications and server logs involved in the process of executing trades," he noted. Goldman's trade tracker application functioned something like a UPS package tracking system and was able to report to executives or to clients on the status of a given trade.

The trade tracker system could get data from different systems, consolidate it in Elasticsearch's key value store system, and then search on it for meaningful data. That meant different technical teams didn't need to be convened to extract data and figure out how to integrate the information from different systems.

Goldman has incorporated Elasticsearch into how its software developers work, with more than 700 of them having access to a search-based code management system. When a bug is found in one version of a piece of software, Elasticsearch can comb through the code library and find all instances of the bug.

Here, Elasticsearch works with Kibana, which builds dashboard reports on the status of projects and code that developers are working with. It captures source code changes, compares the "before" and "after" version of code, and can search for a snippet of code wherever it occurs. Code comments, reference designs, and documentation can all be pulled together through the power of the search engine.

Duet said technology division managers were able to spot Elastic when it first started appearing in the company's software asset management system. The new open source code became a frequent topic in emails and in chats on the programmers' social networks. Usage jumped from a few copies to 50 copies to 200 copies, and the technology division decided to make it widely available as authorized software throughout the company. It also contributed to the Elasticsearch project, engaged with Elastic engineers, and obtained a technical support contract for the search engine.

In addition to the technology division's software engineers and developers, Elasticsearch is sometimes used as part of a system by the financial engineering group doing deep financial modeling, Duet said. Operations staff can also use the Kibana dashboards to quickly build reports that used to require painstaking manual builds.

Goldman Sachs has 9,000 employees actively using technology and "several thousand" of them are now using Elasticsearch, Duet said, either as developers of new applications or as users of existing ones. That's a broader role for search than before, when enterprise search was limited to conventional keyword searches on text and content.

With Elasticsearch, Goldman is showing how versatile and useful search can be as a general purpose service inside the company and as a service built into many different types of applications.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
TSS1030
50%
50%
TSS1030,
User Rank: Apprentice
10/31/2016 | 1:43:28 PM
New standard for enterprise search replacing the google search appliance
Elasticsearch provides the new standard for replacing legacy systems like google search appliance, fast, autonomy, endeca and verity. Elasticsearch based systems like SearchBlox offer a ready to implement application to replace their legacy search applications.
babcockcw
50%
50%
babcockcw,
User Rank: Apprentice
8/18/2015 | 5:23:12 PM
Open source search a growing pantheon of products
In addition to Elasticsearch, there are a number of search engines based on Apache Lucene: Solr, LucidWorks, Swiftype, Index Tank and Consillio, to name a few open source examples. Lucene was developed by Doug Cutting, who went to create the Web crawler, Nutch, and then the project for which he's most noted, Hadoop.
CharlesB21101
50%
50%
CharlesB21101,
User Rank: Strategist
8/17/2015 | 10:49:32 PM
A new way to increase the IT budget
Ariella, Agreed. Anytime you can replace a platoon of lawyers with a single programmer,  the IT budget should get a 2% increase. Charlie
Ariella
50%
50%
Ariella,
User Rank: Author
8/17/2015 | 10:45:13 AM
Elasticsearch
sopunds like a great step toward efficiency, though I'm sure the platoons of lawyers will be disappointed to have their hours cut.
Commentary
Getting DevOps Wrong: Top 5 Mistakes Organizations Make
Bill Kleyman, Writer/Blogger/Speaker,  11/2/2018
Commentary
How to Dethrone Inefficient Software in Your Organization
Guest Commentary, Guest Commentary,  11/2/2018
Commentary
AI & Machine Learning: An Enterprise Guide
James M. Connolly, Executive Managing Editor, InformationWeekEditor in Chief,  9/27/2018
Register for InformationWeek Newsletters
Video
Current Issue
The Next Generation of IT Support
The workforce is changing as businesses become global and technology erodes geographical and physical barriers.IT organizations are critical to enabling this transition and can utilize next-generation tools and strategies to provide world-class support regardless of location, platform or device
White Papers
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Sponsored Video
Flash Poll