Home Performance Testing Amazon Web Services Fails The Price/Performance Test. But Does It Matter?

Amazon Web Services Fails The Price/Performance Test. But Does It Matter?

by support@1lyqa.com

The elephant in the cloud infrastructure room is undoubtedly Amazon Web Services. Started less than 10 years ago as an adjunct to Jeff Bezos’s Amazon eCommerce business, AWS has gone on to invent, and utterly dominate, a new type of business. So much so that analysis firm Gartner had to reset the parameters for its recent Magic Quadrant rating to allow for the lead AWS has over all other comers. So when some statistics come out showing that AWS has a weakness, people jump on it. The last time I wrote something about an AWS weakness was a warning post about their higher-level services relying on an outage prone part of their operation – that post is worth reading for anyone thinking about using some of AWS’ higher-level offerings. But rather than reliability, today’s little brickbat for AWS lies in an arguably more important area, the price/performance discussion.

For those who don’t religiously watch the space, you’d be forgiven for not being aware of the fact that pretty much every other week AWS drops its prices – Amazon CTO and AWS Czar Werner Vogels is often to be heard crowing (justifiably it must be said) about another price drop and how the organization is delivering ultimate value for customers – no surprise given the Modus Operandi of the parent company. But the truth behind AWS’ price and its costs is more nuanced than that. German infrastructure vendor ProfitBricks came out swinging a couple of months ago suggesting that AWS is in fact running a very high margin business and fronted up with this graph which essentially showed the rapidly diminishing cost structure, and not nearly so rapidly diminishing price level, of AWS’ compute product.

As part of the frontal assault, ProfitBricks attempted to subvert the conversation away from a pure pricing one, to a more sophisticated price/performance perspective. Joining in the fray today is infrastructure vendor Virtustream who is releasing a series of test results they commissioned from independent testing house, Cloud Spectator. While (unsurprisingly some might say) the report found Virtustream’s product delivered stellar price/performance, most interesting is the performance of some of the big names in cloud infrastructure, in particular Amazon and Rackspace

First though a word about the methodology. Cloud Spectator took the leading cloud vendors as defined by Gartner’s Magic Quadrant and put them through a series of tests measuring overall performance, application latency, application throughput, and consistency over time, in addition to considering both raw performance and cost/hour of the service to compare results. To ensure the analysis compared apples with apples, as far as possible, the tests match the services tested (CPU, memory, disk space), based on comparable offerings from the cloud providers. Essentially Cloud Spectator tried to create a common framework to measure vendors against one another. And the results are pretty interesting – such well-known cloud vendors as Softlayer, Terremark, Rackspace, and AWS feature near the bottom of the list. While big names, but less respected vendors such as Virtustream, HP, and Microsoft come out on top.

This is somewhat embarrassing for AWS who prides itself on being ahead of the curve. But, and there’s always a but, there is far more to deciding on a vendor than simply the price/performance measures. Customers look for vendors ability to execute, the velocity of their innovation, the vibrancy of the ecosystem surrounding their platform, and a host of other factors unrelated to both price and performance. Virtustream, who both commissioned this research and came out on top of it, is a vendor that specializes in helping organizations move existing workloads to the cloud. That’s a highly specific use case but one which doesn’t really demand vendor innovation. Organizations such as Netflix however, perhaps AWS’s best-known customer, make their buying decisions based on a host of non-performance features – Netflix Cloud Architect Adrian Cockcroft has long said that it’s AWS speed of innovation that is the main drawcard for Netflix.

This is borne out by an email conversation I had with Richard Seroter, product management director at Tier3, a company that interesting scored very highly both on Gartner’s list and on this price/performance survey. According to Seroter:

Standing still is not an option in the cloud. Customers expect more. The bar continues to be raised… This isn’t just a matter of us playing “catch up” to the big boys, but of innovating in areas of search, reseller enablement, usability, and Autoscale.

The bottom line is that this survey won’t impact on AWS massive customer acquisition performance – customers flock to AWS for a host of reasons – price is only one of those reasons. That said, AWS is a competitive beast and I’d be very surprised if they didn’t react to this news with some more price cuts to level the playing field in the weeks ahead.


Related Articles