Bloomberg Adds Machine-Learning To Apache Solr Search - InformationWeek
IoT
IoT
Data Management // Software Platforms
News
2/22/2017
08:00 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%
RELATED EVENTS
Moving UEBA Beyond the Ground Floor
Sep 20, 2017
This webinar will provide the details you need about UEBA so you can make the decisions on how bes ...Read More>>

Bloomberg Adds Machine-Learning To Apache Solr Search

Bloomberg has created an open source way to use machine-learning models to weight searches, then add their values to the Solr search engine.

Bloomberg has found the Solr open source search engine valuable as a core element of several Bloomberg products. Now it has come up with a Solr plug-in that lets Solr users build a machine learning model of what information is most valuable to them, then add that information through the plug-in to Solr searches.

Since Bloomberg is a major distributor of financial news and the supplier of the frequently-used Bloomberg Terminal into which business news is streamed to the financial services industry, having an effective search engine available is part of maintaining its competitive advantage.

Creating a means to train that engine to rank search results according to criteria decided by its internal teams gives Bloomberg an ongoing advantage over other news and terminal services suppliers. Bloomberg has done that through its Learning-To-Rank plug-in for Solr. It has also made the code for the plug-in open source through the Apache Software Foundation, host to the Solr search engine project.

"We have smart people working on Solr," said Kevin Fleming, member of the CTO's office and head of the open source community at Bloomberg, in an interview. "You want to be a good community citizen, you want to contribute code and fix bugs. You also want to make good business decisions," he explained on Feb. 13, as he prepared to address a session at the Linux Foundation's Open Source Leadership Summit at Lake Tahoe Feb. 14-16.

Kevin Fleming, Bloomberg
Kevin Fleming, Bloomberg

Making the code open source assures Bloomberg customers that its ongoing development will continue at a rapid pace through an independent governing board and outside contributors, Fleming explained. It will continue even if Bloomberg's interest in the project flags. Solr and Learning-To-Rank will remain available as open source, even if Bloomberg were, for some reason, to drop its involvement.

Want to see how another piece of open source code is coming close to dominating container management? See How Kubernetes Came To Lead Container Management Pack.

Bloomberg has enhanced its news services' appeal by illustrating that it knows how to help its customers make use of the latest technology. Solr users can build ranking models and coach them on what they want through machine learning. Learning-To-Rank can then take the results from those models and apply them to the search results.

"Tens of thousands of news sources are fed into Bloomberg Terminal or Bloomberg Professional Services," noted Fleming. A ranking model is something like a spreadsheet with designated "features" and weighting factors for what the user wants to look for. The goal is to let computers sift the thousands of results and assign a specified value to the rankings instead of forcing customers to manually go through hundreds of repetitious searches, filtering down the results to what they really want.

"This will save a lot of search teams a lot of time," he predicted.

Bloomberg provides the software framework for building the machine-learing model, which remains a proprietary offering. Making the plug-in open source, however, means other parties will be able to build competing model frameworks that will also work with the Learning-To-Rank plug-in and Solr.

"The plug-in will do exactly what the model tells it to do," said Fleming. "Every major search application has that as a goal – to present results that are most desirable to the user," he said.

The LTR plug-in was regarded as useful enough and mature enough to be included as part of the latest 6.4 release of Solr. It's not activated automatically. Solr users need to choose to use the plug-in but "it's available and ready for everyone to experiment with," Fleming said.

Fleming said the plug-in was so new to Solr that Bloomberg's development team hasn't gotten a lot of feedback on it, but he expects ideas for recommended new features and functions to start as soon as the Solr community is familiar with it.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll