Big Data Storage and Backup – You Need More Than Some Empty Warehouses

By David A. Kelly, Upside Research

The landscape for data backup and storage is shifting dramatically, thanks to some of the recent developments on the data and technology front. It used to be that most data was structured, or semi-structured, and easily stored in databases, or as the volume grew, data warehouses. Backup was scheduled at pre-determined intervals and disaster recovery (DR) plans were in place. But, in the past several years, the rules have changed on several fronts, leaving CIOs heads spinning from all the moving parts.

Today, the presence of “big data” is changing the way data is being stored for organizations. Now, in addition to structured and semi-structured there are volumes of unstructured data coming from sources within and increasingly outside the firewall. The percentage of mission-critical data is growing within the company as well, and therefore creating a situation where IT has to be able to guarantee as much as half of the data that resides in a company’s systems can be recoverable and accessible within minutes or hours.

Companies want to interact with this data in real-time, and that changes the dynamic of storage and backup and DR completely. Add the cloud to this picture, and then all of the edge devices that employees are using (laptops, iPads, etc.), and you have a cobweb of interconnected data and systems and all of them need to be protected even as they are made available. How does a CIO effectively manage the secure storage of petabytes of data while continuously backing it up and executing a DR plan that will prevent any catastrophic data loss?

First, consider whether it makes sense to protect all of your data. This sounds like sacrilege, but hear me out. Obviously, any source data is critical and needs to be protected and recovered quickly. However, reports generated from that source data or manipulated data from that source data may be more cost efficient to reproduce rather than try to protect and recover. These nuances can make a big difference in the cost and size of the big data store and DR plans you need to have.

For the data that is mission-critical and needs to be readily accessible, consider adding on to your existing data storage plans in creative ways. Local, disc-based systems that incorporate images can make restoration of critical files much faster.  Also, determine at what point the data can be moved to archival and off the main system to reduce loads and protect primary data.

Once you have identified and categorized your big data, consider how cloud storage and backup can impact your environment. For tier-two data that is not mission-critical, cloud services offer an unprecedented economy of scale, and organizations of all sizes are taking advantage as the cloud continues to mature.  Pay-as-you-go models can be very appealing for a company with growing data needs that doesn’t want to have to invest in the hardware infrastructure.  Redundancy is easily configured in a cloud environment, and it can satisfy many aspects of a data backup plan.

There is no one right solution for data storage, backup and recovery in this world of Big Data and a geographically dispersed working environment. But, the data keeps growing at alarming paces, and now is the time for IT managers to create a plan for data management that makes the most sense for your environment. It may need to incorporate a blend of technologies and approaches to make it cost-effective and meet performance requirements. But, with new storage technologies and maturing cloud services, it is possible to find a way to store all of your data safely and efficiently without breaking the bank.