In the beginning (prior to about 2010), teams of highly paid system administrators spent their days (for interrupted service) and nights (for uninterrupted service) reconfiguring the storage subsystems of the computers in their datacenters, adding or removing disks as storage needs changed and their hardware assets needed to be rebalanced.
Once this was done, they then had to create file systems on those disks and attach them to the running operating systems beneath the applications that required more space for data: this was difficult, skilled work and prone to error.
Although storage was expensive, there was a natural inclination to over-provision because of the difficulty and risk always present in these re-provisioning exercises. What we needed was a means to abstract away the physical aspects of disk storage and remove the complexity and risk. Storage industry bodies and commercial storage companies put a lot of work to this end, and the result is software-defined storage (SDS).
What is SDS?
SDS is a new category of storage software products that creates an abstraction or virtualized layer in software representing the storage beneath. This software-defined approach abstracts and simplifies the management of physical storage devices. It represents a new evolution in how storage is managed and deployed.
SDS is being adopted within the industry—but like anti-lock braking systems (ABS) in cars, you don't know it's there. This is mainly because many of the building blocks of SDS, such as volume management and multi-pathing control, were already in place waiting for the techniques of SDS to be applied.
What are the characteristics of SDS?
Physical disks have already been abstracted and virtualized: that’s what an operating system does. So what does SDS offer in addition? The following attributes are what make SDS platforms so attractive. Attributes considered essential are noted as “must haves”:
- May allow storage consumers to “build the storage themselves,” providing their own commodity hardware and creating a software layer with an operating system and layered storage management products without deep-level technical knowledge.
- Must have the ability to virtualize the data path and remove the complexity of block, file, and object interfaces that support applications written to these interfaces. Redundant (failover) data paths via additional physical connections to the storage assist in this.
- May work with arbitrary hardware (that is, multi-vendor hardware).
- Must have seamless ability to scale the storage infrastructure without disruption to the specified level of availability or performance, for example, quality of service (QoS) agreements and service-level agreements (SLAs). This implies the disaggregation of storage and data services: the ability to re-provision without modifying (or interrupting) the software services layer.
- Nearly always includes the pooling of storage and other resources.
- May allow for the building of a storage and data services “solution” incrementally over time or across budget cycles.
- Must have management automation: a simplified management scheme that reduces the cost of maintaining the storage infrastructure.
- Must have standard interfaces: a self-service interface for users and application-programming interfaces (APIs) for the management, provisioning, and maintenance of storage devices and services.
- Must have the ability to allow administrators to set policies for managing the storage and data services.
- May allow storage and data service owners to do customer chargeback based on the storage consumer, throughput, or space. This implies transparency: the ability for storage consumers to monitor and manage their own storage consumption against available resources and costs.
Implications of SDS
- A software layer replaces proprietary hardware-based services in the drive towards datacenter efficiency through more efficient use of existing resources.
- A complex hardware-based technology service requiring specialized skills is abstracted away reducing costs.
- More operational flexibility is gained by moving away from the concept of storage as a separately managed resource.
- Datacenter functionality is improved in terms of reduced error rates and higher systems availability.
Self-service interface for provisioning and management
This definition of SDS allows for both proprietary and heterogeneous platforms, but what is necessary to meet the definition is that the platform offers a self-service interface for provisioning and management. Datacenters that provide services at internet scale—such as online retailers and media providers—require the scalability, flexibility, and hardware economy that come with software-defined storage.