Operations costs are the Achille's heel of NoSQL
This article was originally published on my Cloud Power blog at Computerworld on October 27th, 2015
NoSQL databases scale by adding more commodity servers. With more commodity servers come increased costs and complexities. Some NoSQL systems are better at this than others and need less.
Consider the size of the Apple Cassandra installation that is reported at 75000 nodes and over 10 petabytes of data. The complexity of the operations, monitoring, upgrades and other maintenance tasks must be overwhelming. Apple bought FoundationDB to cut their own costs while improving performance. Julie Bort writes:
At Apple’s scale 10% of 75000 is 7500 nodes and it is not something to ignore. The most popular post on my blog is my article on how I’d like to replace Cassandra with DynamoDB in the AWS environment. The long term costs of operating Cassandra are on the minds of Cassandra adopters.
MongoDB is under pressure from customers to reduce operations costs as well. Viber migrated their MongoDB cluster to Couchbase cutting the number of AWS EC2 instances in half. At Viber’s scale that is not a small number.
Companies interested in adopting NoSQL should consider their options carefully. The vast majority of database use cases do not need massive horizontal scalability. Most applications could be better off with traditional SQL databases. In the cloud, there are NoSQL alternatives that cost less and are easier to maintain. Let’s review just a few examples.
PostgreSQL has been offering NoSQL capabilities like MongoDB since version 9.3. That includes ACID, hierarchical document data and ability to index JSON documents. AWS RDS service of PostgreSQL offers high availability, redundancy, and fail-over. Being a managed service it requires very little attention. Many tasks such as backups and fail-over are fully automated. Rich management API and monitoring tools provide for customization of scaling behavior.
As John Martin of Computerworld wrote, “When it comes to storage, cache is king”. Azure, AWS and Google offer managed cache services. AWS Elasticache in particular offers a choice of Memcached and Redis. Redis is an interesting alternative to NoSQL since its low level data model is similar to that of Cassandra for some of the use cases. Redis database has to fit entirely in-memory but it can be persisted to disk and recovered upon reboot. Redis can be configured in clusters for high availability and performance. On master failure one of the slaves becomes the new master.
AWS DynamoDB and Google BigTable offer a similar data model to Cassandra as well as infinite scalability. Neither service requires any administration or devops. One has to be on the look-out for burst performance, however. Burst capacity is one area where a custom configured NoSQL database can shine.
An object storage tool like AWS S3 is a long term infinitely large key/value store. As a corner stone of AWS, S3 can integrate with CloudFront, RedShift and many other AWS services. It scales horizontally without any questions asked and can store JSON and binary documents as well as logs. S3 is also ridiculously cheap and can be used to store terabytes of data.
Companies should keep in mind the costs associated with NoSQL technology. It is important to consider not only the technical merits but also the costs. Development teams that choose the right tool for the right job will always win.
NoSQL databases scale by adding more commodity servers. With more commodity servers come increased costs and complexities. Some NoSQL systems are better at this than others and need less.
Consider the size of the Apple Cassandra installation that is reported at 75000 nodes and over 10 petabytes of data. The complexity of the operations, monitoring, upgrades and other maintenance tasks must be overwhelming. Apple bought FoundationDB to cut their own costs while improving performance. Julie Bort writes:
While both Cassandra/DataStax and FoundationDB are noSQL databases, FoundationDB had some unique technology. It works super-fast but needs far less hardware than Cassandra, making it even cheaper to use, even as it scales. (In geek speak, it’s an “in-memory” database that runs on flash storage.)
Goldmacher says it needs somewhere between 5% to 10% less hardware than Cassandra.
At Apple’s scale 10% of 75000 is 7500 nodes and it is not something to ignore. The most popular post on my blog is my article on how I’d like to replace Cassandra with DynamoDB in the AWS environment. The long term costs of operating Cassandra are on the minds of Cassandra adopters.
MongoDB is under pressure from customers to reduce operations costs as well. Viber migrated their MongoDB cluster to Couchbase cutting the number of AWS EC2 instances in half. At Viber’s scale that is not a small number.
Companies interested in adopting NoSQL should consider their options carefully. The vast majority of database use cases do not need massive horizontal scalability. Most applications could be better off with traditional SQL databases. In the cloud, there are NoSQL alternatives that cost less and are easier to maintain. Let’s review just a few examples.
AWS RDS for PostgreSQL
PostgreSQL has been offering NoSQL capabilities like MongoDB since version 9.3. That includes ACID, hierarchical document data and ability to index JSON documents. AWS RDS service of PostgreSQL offers high availability, redundancy, and fail-over. Being a managed service it requires very little attention. Many tasks such as backups and fail-over are fully automated. Rich management API and monitoring tools provide for customization of scaling behavior.
Redis
As John Martin of Computerworld wrote, “When it comes to storage, cache is king”. Azure, AWS and Google offer managed cache services. AWS Elasticache in particular offers a choice of Memcached and Redis. Redis is an interesting alternative to NoSQL since its low level data model is similar to that of Cassandra for some of the use cases. Redis database has to fit entirely in-memory but it can be persisted to disk and recovered upon reboot. Redis can be configured in clusters for high availability and performance. On master failure one of the slaves becomes the new master.
AWS DynamoDB and Google BigTable
AWS DynamoDB and Google BigTable offer a similar data model to Cassandra as well as infinite scalability. Neither service requires any administration or devops. One has to be on the look-out for burst performance, however. Burst capacity is one area where a custom configured NoSQL database can shine.
Object storage
An object storage tool like AWS S3 is a long term infinitely large key/value store. As a corner stone of AWS, S3 can integrate with CloudFront, RedShift and many other AWS services. It scales horizontally without any questions asked and can store JSON and binary documents as well as logs. S3 is also ridiculously cheap and can be used to store terabytes of data.
Final thoughts
Companies should keep in mind the costs associated with NoSQL technology. It is important to consider not only the technical merits but also the costs. Development teams that choose the right tool for the right job will always win.