Let’s talk cloud neutrality
Consider the following conversation:
Avoiding managed services, using Docker and running eight Mongo and twelve Kafka nodes does not make an application cloud neutral — it makes it costly to develop, Kafkaesque (excuse the pun) to architect, and impossible to maintain in production. While the architecture diagrams look impressive and sophisticated, you end up losing your weekends and quality time with family to DR exercises and production support.
When we choose to use MongoDB, we are making a conscious decision to tie our application to a specific NoSQL technology. Unlike SQL, there is no platform-independent standard for NoSQL. By choosing MongoDB, we are effectively linking our application to a non-standard data store.
The fact that MongoDB is open-source should not be comforting either. Commercial MongoDB startup is not yet profitable, and they lose scores of millions in revenue per year. Should they change the terms of service or go out of business all together you have only one option left: relying on open-source for continued updates and improvements to the product.
While a project the size of Mongo is unlikely to be entirely abandoned by the open-source community, it's been known to happen to open-source projects. In the end, you will be left to maintain the source code of Mongo yourself. If you need examples closer to home, consider how the Cassandra project abandoned early Thrift-protocol drivers when they moved to CQL.
Likewise, the choice to use Kafka is not what makes the application cloud neutral. It doesn't even make your application messaging platform neutral. Kafka is a proprietary messaging/streaming platform.
The reality is that developers and architects make decisions to irreversibly tie our projects to some technology every day -- programming languages, runtime platforms, databases, queues, object storage, etc. Just ask companies that bet their farms on Borland Delphi or Java Swing. The choice to build upon a managed public cloud services is no different from the choices we make every day anyway. So, why complicate our lives for the sake of a small chance that AWS may go out of business and we might need to move to Azure?
I am not saying that we should not abstract ourselves from the details of whatever managed service we are using. We should, however, build these abstractions in the application software rather than try to find platform-independent infrastructure services.
Enterprise architect: As an enterprise, we should be wary of being tied to any particular public cloud provider. My application is cloud neutral because I can re-deploy it at a moment’s notice to any cloud I want.
Me: How do you accomplish that?
EA: My entire application runs as a collection of docker services. I take no advantage of any of the managed services such as queuing or NoSQL because, you know, I must be cloud neutral.
Me: So what do you do instead?
EA: I use MongoDB and Kafka (replace Mongo and Kafka with NoSQL and queuing systems of your choosing that don’t run as managed services)
Me: How do you achieve redundancy and fault tolerance?
EA: I run eight MongoDB nodes in two clusters of four; I have two six-node Kafka clusters, and I have a script that alerts me 24/7 if any of them is down. We will have quarterly disaster recovery exercises on weekends to make sure everything stays up and is ready.
Avoiding managed services, using Docker and running eight Mongo and twelve Kafka nodes does not make an application cloud neutral — it makes it costly to develop, Kafkaesque (excuse the pun) to architect, and impossible to maintain in production. While the architecture diagrams look impressive and sophisticated, you end up losing your weekends and quality time with family to DR exercises and production support.
When we choose to use MongoDB, we are making a conscious decision to tie our application to a specific NoSQL technology. Unlike SQL, there is no platform-independent standard for NoSQL. By choosing MongoDB, we are effectively linking our application to a non-standard data store.
The fact that MongoDB is open-source should not be comforting either. Commercial MongoDB startup is not yet profitable, and they lose scores of millions in revenue per year. Should they change the terms of service or go out of business all together you have only one option left: relying on open-source for continued updates and improvements to the product.
While a project the size of Mongo is unlikely to be entirely abandoned by the open-source community, it's been known to happen to open-source projects. In the end, you will be left to maintain the source code of Mongo yourself. If you need examples closer to home, consider how the Cassandra project abandoned early Thrift-protocol drivers when they moved to CQL.
Likewise, the choice to use Kafka is not what makes the application cloud neutral. It doesn't even make your application messaging platform neutral. Kafka is a proprietary messaging/streaming platform.
The reality is that developers and architects make decisions to irreversibly tie our projects to some technology every day -- programming languages, runtime platforms, databases, queues, object storage, etc. Just ask companies that bet their farms on Borland Delphi or Java Swing. The choice to build upon a managed public cloud services is no different from the choices we make every day anyway. So, why complicate our lives for the sake of a small chance that AWS may go out of business and we might need to move to Azure?
I am not saying that we should not abstract ourselves from the details of whatever managed service we are using. We should, however, build these abstractions in the application software rather than try to find platform-independent infrastructure services.