Amazon Glacier: why and how does it may work?

Recently Amazon announced Glacier - a new service in their AWS suite. It allows you to store large amounts of data very cheaply - just $0.01/Gb per month plus some expenses for traffic and API calls. It is incredible cheap price, compare  it with S3: $0.125 /Gb for first Tb of data. But where is the trick? Well, actually, there is one important detail: retrieval time for you data could be up to several hours.

Why do we need it?

Such a long retrieval time means that you can use it only for some data that should be stored probably for a long time, but there is no need to access it quickly. Consider some kind of historical financial data: in some countries there are government regulations that require financial institutions to store every single transaction for several years after it occurs. It turns out that most of these transactions would never been accessed. It could happen only in case of some investigation or system audit, which happens not very often. Nowadays most of these data is stored on some hard drives or even magnetic tapes and usually they are not connected to the network, so that retrieval time is also up to several hours. And that is the target market for Glacier.

Perito Moreno Glacier. Patagonia, Argentina (photo taken by Luca Galuzzi)

Amazon targets customers who want to store lots of data for a very long time, do not access it very often and quickly, but data should be stored in a very reliable way. Glacier offers you 99.999999999% durability. That's right, eleven nines - impressive reliability! It is very expensive to build such a reliable storage in-house, so only really big corporations had access to such a reliable storage in past. There are several services that adress same issue, but to be honest they don't look seriously enough to be enterprise vendors. Amazon is the first enterprise level vendor on this market.

How does it (may) work?

As a disclaimer: I am not an Amazon employee and there are no information about Glacier architecture available in any public sources. So, I can only imagine and suppose how does it actually may work.

Lets imagine that we want to build service like Glacier. First of all we would need lots of storage hardware. And it must be pretty cheap (in terms of cost per gigabyte) hardware, because we want to sell it for such a little amount of money. There are only two types of hardware that fit these requirements: hard disk drives and magnetic tape. Last one is much cheaper, but less reliable because of magnetic layer degradation. It means one should perform periodic data refresh to prevent data loss. They may use special custom hard drives with big capacity an slow access time, simply because speed is not critical for them. It makes overall solution cost even less. I don't know what kind of data storage hardware Amazon uses, but I think hard drives is little bit more possible option.

Second component of big data warehouses is infrastructure that connect users with their data and make it available for them in timeframe described in SLA. It could be network, power supplies, cooling and lots and lots of things you can find in modern datacenters. If you would build service like S3 infrastructure cost would be even bigger than storage cost. But here are one  important difference between S3 and Glacier: you don't have to provide access to data quickly. It means that you don't have to keep your hard drive turned on, which means reduced power consumption. It means that you don't even have to keep your hard drive plugged into server case! It could be stored in simple locker. And all you need is employee who is responsible for finding and plugging your hard drive into server when user asks access to data. And several hours are definitely enough to do it even for human being. Or little cute orange robot:

Sounds crazy? Well, lets look at this solution from the other side. What is Amazon, first of all? Cloud vendor? Nope. They are retail company. One of the biggest in the world. And they have probably the best logistics and warehouse infrastructure in the world. Lets imagine you order hard drive on Amazon web site. How much time does it usualy take for Amazon to find it in their warehouse, pack and send it to you? Several hours? Just imagine that they don't send it, but plug it into a server and turn on instead. Sounds like pretty similar task, isn't it?

It is amazing how Amazon integrates their businesses with each other. AWS was a side product of their main retail business. Product they started to sell just because they realized that it has value not only for their business, but also for other people. And now we can see how AWS uses offline infrastructure of Amazon to provide absolutely new kind of service. Fantastic fusion of online and offline infrastructures working together to create something new!