re: How Netflix Is Ruining Cloud Computing
Here's the tree response.
Cloud 1.0 vs. 2.0:
You say, "The specific IaaS provider used underneath, and whether you do this with public or private clouds is irrelevant to the architectural constructs we've explained." But of course, the Netflix contest has to do with your tools, not "architectural constructs" per se. And your tools are absolutely tied to a specific IaaS provider. And from an architectural perspective, while Netflix has done some awesome things, I think Zynga's architecture (including what they pay for what they get) is much more likely to be what best-practice enterprise cloud architectures look like in 5+ years. I don't know many cloud architects who are aware of the difference between Zynga and Netflix who would pick Netflix's implementation over Zynga's--again, largely because of the multi-cloud capabilities.
My point in bringing up the outages was not to imply that they were international or fatal; it was to point out that Netflix's cloud architecture is not perfect (not that any cloud architecture is, but always good to point this out), and that one can tie at least one outage to Netflix's specific architectural decision to embrace a proprietary service (ELB) when other, non-proprietary, more resilient options were available. I'm happy that Netflix won't repeat an outage due to that specific proprietary service, but the overall philosophy of choosing AWS services over open options that are more flexible and more resilient remains.
As long as you continue to force Netflix to use new and expanded Amazon-provided services over other options, you're creating a moving target that no other vendor will hit. The better path to take is: how should organizations design their architectures so that they can maintain portability and interoperability across multiple vendors, whereever possible? If, today, we wait for Google Compute Engine to be more widely available and tested, then shouldn't we be moving to abstraction layers for API communication, instead of doubling down and adopting more and more proprietary APIs? Perhaps AWS will release their API to the world and allow all businesses to use it openly, but they haven't yet, and so it's a very risky move to bet an architecture on AWS and any vendors (e.g., Eucalyptus) that AWS will bless. Perhaps what you're saying is that you'd be happy to see abstraction layers in the Netflix tools for working with other clouds, but *you're not actually saying that*. Please say that.
Thanks for the details on Edda; my knowledge was from reading what was easily available on github: "Why did we create Edda? ... if we see a host with an EC2 hostname that is causing problems on one of our API servers then we need to find out what that host is and what team is responsible, Edda allows us to do this." Edda sounds like a great tool to take multi-cloud--would you considering suggesting that as a theoretically good project for the contest (no guarantees on prizes)?
Your explanation of using Chef with AMInator makes a lot of sense in the "500 simultaneous instances" use case. Which is--you would admit--not a common circumstance amongst the people who use/will be using your cloud tools. And unfortunately, your first happy user of AMInator (on Twitter, at least) made over 25,000 Ubuntu AMIs with it--can you tell me why that would ever be a good architectural decision? AMInator strikes me as a tool like PHP or a GOTO statement--there are places where you should probably use them, but it's hard to argue that they should be part of any kind of "best practices" decision.
The fact that only one out of ten prizes involves portability, and the fact that you take such an expansive view of portability to include adding language support to an existing tool (which has NOTHING to do with cloud portability!), shows that you really think that cloud portability unimportant to Netflix. If Netflix wants to make that business decision, then fine. But I would argue that Netflix is a role model in the world, and has a lot of ears, and that it's just irresponsible for Netflix to lead the rest of the world on the same path.
To the extent that Netflix is trying to exploit open source in the same way many companies do--to share code in exchange for getting additional development for free--I have no issues. Go for it. But I have a problem in the way that Netflix's tools and architectural decisions are taken as THE reference architecture. I write here, and I wrote the piece, not to try to convince you, Adrian, to change the way Netflix does things. I would like you to the run the contest in such a way that it promotes portability and interoperability and make the judging panel less AWS-centric. But beyond that, I'm really writing these for those people out there considering whether Netflix's cloud architecture is something they should copy verbatim. (Don't!)
Heidi Roizen, an entrepreneur-turned-VC, put it this way: "I don't ask 'what happens if?' ... I ask 'what happens when?'"--meaning that there are certain things that we know will happen, and if we aren't thinking about them and planning for them, we're not thinking strategically enough. (One of her examples: "what happens when we have self-driving cars?") It is a certainty that we will have viable IaaS competitors to AWS. But the attitude that is embedded in the Netflix cloud tools--and from what I see of the contest today--is one that essentially says, "we will look nowhere but AWS." And there is no thing that has been said in response to my piece that says otherwise. In fact, you don't even address my quoting of you in different places where you have excluded other options by fiat, regardless of price or functionality.
And that is the problem.