Re: Couchbase Response
Thanks for sharing your thoughts, Shane. We at USA have some differing opinions and outlined our responses below to your comments:
1) Well, they chose to benchmark single node deployments.
RESPONSE: Single server test results are absolutely relevant as they represent the building blocks of any system. As noted by the author of YCSB, it is important to first test performance based on a single node, and then to test scalability. The better the performance of a single node, the fewer nodes will be required to meet the demands of a specific application. YCSB was designed by the team at Yahoo! to test vertical as well as horizontal scaling. We plan to test horizontal scaling in a future report.
2) They chose to have Couchbase Server perform two operations per write (read+update) instead of one, but not MongoDB.
RESPONSE: As indicated in your blog post and comments, Couchbase requires two operations to perform an update, whereas MongoDB requires one. In Couchbase, as you know, updates must be performed by the client, and the client must also manage conflict resolution. This adds to the work and ongoing maintenance for application development teams using Couchbase, and it impacts performance and scalability of the system. We implemented the client correctly, following Couchbase's documentation. We catch and retry any CASMismatchException errors that the server throws when an update would overwrite another update to the same document since the document was initially read. As noted in your
blog comments, Couchbase plans to add an equivalent feature in the next release. We will include this feature in future tests when it becomes available.
3) They used a two year old client library for Couchbase Server, but not MongoDB. It waits at least 100ms before checking if writes are durable. The latest, as little as 10μs.
RESPONSE: We used the same client library as Couchbase's benchmark conducted less than a year ago by Thumbtack (we did update it to the latest patch release). We tried to use 2.1.1 but it performed worse and was not able to complete the full test without Timeout exceptions. Since then, someone on the Couchbase forums reported the issue, possibly the same we encountered.
4) While single node deployments require sync disk writes for durability, distributed databases do not. They can leverage sync replication to perform writes on multiple nodes.
RESPONSE: Durability is defined in terms of writing data to disk, not memory. This is true whether using a single server or multiple servers. If your servers lose power or crash, data in RAM that has not been persisted to disk will be lost. However, if deployed correctly across racks and data centers, potential data loss can be minimized by replicating to multiple servers. All three products provide similar capabilities in this area, and we plan to evaluate this in our next round of tests.
We attempted to provide links to evidence but this site does not support URLs in comments.
United Software Associates