At the recent Intel Developer Forum event in San Francisco there was a demo of 360° panoramic imaging. Using a special camera mount with six GoPros, video is stitched together with software from Kolor (on Intel hardware of course) which can then be watched with a special viewer or using a VR headset. The camera rig was interesting, as each one of the 4K Ultra HD cameras produces 1GB of data per minute!
This is the sort of rich media which is driving the huge growth in the need for data storage. At one time, most stored data was transactional data, but now it includes images, audio, and video. In 2016 there will be 2Bn smartphones in the world. That’s a picture and video camera in 2Bn people’s hands, all connected, and all looking for a place to store that data.
In 2004, only 60-70 companies in the world acquired a petabyte or more of new storage each year. According to Chris Gladwin, founder of Cleversafe:
“There was maybe one organization at that time that was at a 100-petabyte scale, and that was it. Now, (ten years later) the number of companies in the world that are deploying a petabyte or more of storage every year is around 7,000.”
In five years that number will exceed 70,000!
To give you an idea how vast a quantity of data this is, the hardware alone needed to store an exabyte would cost about US$1Bn. And it has been estimated that if Google were using punch cards to store their 15EB of data, the stack would cover all of the US states that make up New England to a depth of over four kilometers!
The big problem is that traditional storage technology was never designed for this volume of data. SAN and NAS architectures date from an era of megabytes and gigabytes. We are in a world where petabytes (10005) are the norm, public cloud providers are already dealing with exabytes (10006), and zettabytes (10007) are coming. As you move to petabytes, the cost and complexity of maintaining traditional replication-based storage infrastructure begins to increase rapidly. Enterprises find that large parts of their IT budgets, and much of the time and effort of their best people, gets consumed just trying to stay on top of their data storage. Many are finding that the value of holding the data is outweighed by the cost of storing it.
This is one of the reasons why most of the market growth is with hyperscale storage, which is mainly software-defined and based on approaches like object storage. And most of that growth is with the public cloud providers. IDC reported recently that:
“original design manufacturers (ODMs) who sell to hyperscale data centers like those owned and operated by Facebook and Google saw revenues grow by 25.8 percent to over $1 billion in Q2. That’s compared to overall storage industry revenue growth of just 2.1 percent to $8.8 billion in the quarter.”
However, for an enterprise with significant data storage requirements, the public cloud approach may not be viable. A public cloud provider has the capability and capacity, but their pricing models usually load cost towards accessing the data. And there are issues about the data security and sovereignty: do you know where your data sits, do you trust it, and do you know you are not breaking the law with your customer’s information?
All these factors mean that enterprises will continue to have a need for private storage. And the solutions they will be looking for will need to address the problems of being hyperscale – able to grow to petabytes and beyond. They should be trustworthy – data integrity and governance needs to be built in so you can be sure that you are not breaking the law. They should enable data accessibility and availability – archiving your data on tape may be relatively cheap, but it significantly reduces your ability to use and extract value from it. And any approach needs to be cost-effective – Moore’s law should deliver benefits to storage as well as computation.
These are the principles that underlie Ericsson’s approach to building cloud storage offerings and our first product in the space: Secure Cloud Storage. This offering brings Cleversafe Object Storage together with Ericsson’s HDS8000. In a later post we will talk more about the details of Secure Cloud Storage.