NoSQL Everywhere? Not So Fast

Sometimes we need to slow down and sort through the hype, despite pressure to act. A perfect example: NoSQL.

Factors including the speed with which data is piling up and the rate at which the cost of conventional databases is escalating are pushing people who should know better to consider using NoSQL in situations where it's wholly inappropriate. That category includes stores of core customer and product data and related transactions that have to exist in a single source of truth and use standard access techniques.

Now, we get why IT is intrigued. Just 11% of the 760 respondents to our InformationWeek 2012 State of Database Technology Survey are very satisfied with licensing costs and terms for their conventional databases. And the broad "NoSQL" label does indeed reflect the main attraction: Often, these systems are open source or quite inexpensive and can run on commodity hardware.

But remember, there's a reason structured data and relational database management systems have ruled the enterprise roost since the '80s. SQL--along with technologies like distributed caching platforms that have helped cement its dominance by bringing in-memory caching and clustering of data--is the go-to choice for mission-critical, high-performance use cases such as airline reservation systems and financial/securities applications. And the fact is, those jobs aren't going anywhere--especially to NoSQL options like MongoDB on a white-box server--no matter how much the hype machine might suggest otherwise.

However, IBM, Oracle, and Microsoft are seeing an unprecedented challenge to their SQL licensing money machines thanks to enterprise data workloads that are getting more diverse, larger, and more complex. In those cases, IT will trade stringent data consistency for low latency and a favorable cost versus performance balance. This is where NoSQL technologies like those outlined in the chart below shine.

So how can big companies get in on the action? Let's contrast the nature of data suited for NoSQL with the properties of enterprise data that requires the single-source-of-truth systems that we talked about. We'll use three V's: volume, velocity, and variety. (We cover these in more depth in our full report).

Data linked to mission-critical applications--and generally housed in conventional transactional systems--is described by the acronym "ACID."

This data is:

>> Atomic: Transactions are fully committed, meaning they execute completely or are rolled back if anything fails.

>> Consistent: Transactions don't leave the database if there's an inconsistency with the underlying data store.

>> Isolated: Transactions are insular and don't affect one another.

>> Durable: Successful changes are recorded permanently in the database before the transaction is considered complete.

In contrast, Web and social media applications must focus more on massive scalability--often a few orders of magnitude higher than what IT normally handles. That means developers can't be too fastidious about data rigidity. For example, an item on an online auction site can't be locked for long intervals while someone contemplates a bid; it has to be simultaneously made available to thousands of other shoppers. The data properties of such workloads have been dubbed BASE, in which it's OK for data in these environments to be:

>> Basically available: Fairly self-explanatory, and as opposed to being atomic in real time.

>> Soft-state: Meaning it may change or expire, as opposed to being durable.

>> Eventual consistency: As opposed to stringent transactional consistency.

But the nuances of BASE vs. ACID don't tell the whole story when it comes to big data initiatives, because they mainly focus on variety. Use cases where NoSQL is ideal also feature staggering volumes and velocities of data; we're talking, at the low end, a few terabytes of records, with 10,000 or 20,000 concurrent inserts per second, minimum. Once you're at that level, IT needs to focus on platforms that can serve as the infrastructure layer--capable of scaling out, without costing a fortune.

Strategy: NoSQL Everywhere? Not So Fast

The Trouble With Labels
Our full report on use cases for NoSQL is available free with registration.

This report includes 14 pages of action-oriented analysis and 5 charts:

Get This And All Our Reports

Enterprise Use Cases

So, where does NoSQL fit now, given that SQL isn't yielding its sweet spot? Many enterprises now have checklists for determining when NoSQL is appropriate, so the question doesn't need to be rehashed constantly. If you don't have guidelines in place, you should.

In a nutshell, consider NoSQL when the workloads are BASE, not ACID; if there are thousands of data consumers, not hundreds; and if you want to use commodity hardware and software.

Another emerging use case is when, for a given low-latency data tier, distributed caching platforms are too expensive; examples are dedicated key-value store applications that don't need in-memory low latency and with low or nonexistent data integrity requirements.

Cost and performance are critical factors in business and technical evaluations of NoSQL initiatives. In general, when SQL is used for high-volume Web-facing workloads, the performance hit can be disastrous. Think about the overhead from SQL parsing, joining of normalized tables, and the underlying rigor of restoring data integrity--not to mention the housekeeping around managing indexes. Under heavy volume, NoSQL can be as much as 100 times as fast as SQL for such workloads, according to the tests performed by the NoSQL open source community. Oracle, on the performance of its NoSQL offering, demonstrated excellent scalability, throughput, and latency; one test showed close to 100,000 inserts per second and 3.7 milliseconds average latency for 360 client threads.

As for costs, conventional SQL (hardware and software included) ends up about five to 10 times as costly as a typical NoSQL setup.

Bright Future

Big data needs are being thrust on the enterprise, and often NoSQL is the answer. While it takes only a few hours to get a profile set up in a social network, laying an enterprise data foundation that's capable of making sense of and being responsive to all the input you'll get through these channels is another matter altogether. And that's just the tactical need to respond to the newly amplified voices of your customers. Many progressive companies are also strategically tapping into these channels to launch new services and products, as well as to expand the reach of their current portfolios.

Ultimately, IT must avoid extremes: Don't look down on emerging NoSQL concepts, but don't get too enamored with them, either. Smart enterprises are crafting a balanced, holistic, pragmatic approach for using NoSQL to solve today's big data needs. It's about "Not Only SQL," not purely "NoSQL."

InformationWeek: May 14, 2012 Issue

Download a free PDF of InformationWeek magazine
(registration required)