UDM – Taming The Unstructured Data Beast

Featured article by Shaun Walsh, VP Channels and Alliances, Igneous

“In the midst of chaos, there is also opportunity”- Sun-Tzu

The Beast is Big and Getting Bigger

Never has this famed quote been more appropo. Organizations have been losing the battle with the growth and rampant sprawl of unstructured data. To help put this into perspective, according the IDC’s DataAge 2025 report they predict that global storage datasphere will grow from 16ZB in 2016 to 163ZB by 2025 and over 90% of that data will be unstructured vs structured. Taming the unstructured data beast is the opportunity in the chaos for the channel.

The Unstructured Data Test

Do you want to find a new opportunity with your clients or prospective clients? If so, here is a simple set of test questions you can ask.

1. How much data capacity is online today in your org, both on-premises and in the cloud?

2. Where is this data located, and is that where it should be?

3. Who owns this data and can you identify them for governance compliance?

4. What data sets belong to which applications?

5. Is that data protected and secured to your required SLAs?

I can guarantee that 4 out of 5 times you will get more “I don’t know…”s, “that is hard to say…”s, or non-committal monosyllabic grunts, sighs and shrugs than anyone is comfortable with. Why? Because the answers to very simple questions are often the most difficult to provide.

These five questions should be enough to to find a new business opportunity, show your value to your clients, and increase your standing as a trusted advisor. To back up this test, we gave it to over 200 IT leaders and application owners and the results are available at (Link).

Unstructured Data Management Report

In the Unstructured Data Management (UDM) report, over 200 IT leaders and application owners shared their current perspectives, challenges and predictions about unstructured data. In this report, we sought to understand three core trends facing IT leader and applications owners.

1. What is the scale of unstructured data under management?

2. How is that data being is being generated differently today and into the future?

3. What are the new ways that data is being applied in the business?

From a scale perspective, 40% indicated that the “age of machine file creation” is upon us and machine created data will far exceed human generated data going forward. This is driven by analytics, IoT, medical imaging, EDA, media, financials and dozens of other automated functions. With IDC predicting that global storage datasphere will grow by 10X (16ZB in 2016 to 163ZB by 2025) and over 90% of that new data will be unstructured, it is easy to see that we are just beginning to see the scale and scope of the unstructured data beast.

The second major data trend is the cataloging and management of metadata and tagging files into data sets that feed specific workflows and real-time analytics. Organizations are beginning to think of data in groups and datasets with shared metadata categories. This better enables them to support the needs of specific workflows, applications and analytics As the metadata is more visible it helps enhance the value, usability and mobility of data to support a greater range of use cases. They are clear that data must always be searchable, shareable and deliverable for any need.

Next was the transition in unstructured data’s importance. It went from being important for business continuity to being strategic for generating revenue and improving operating models. This is best demonstrated in the the use of real-time analytics and the training of machine learning models. Analytics and machine learning are improving everything from marketing (via churn prediction, recommender systems, and targeted ad placement) to cancer diagnosis (via image recognition). The ability to operationalize data is a competitive weapon, and when combined with auditable integrity, it’s a game-changer. Competitively, data must and will be used for driving revenue, improving product designs, creating better customer interactions, finding ways to lower cost, and optimizing supply chains (both physical and digital).

The Unstructured Opportunity for the Channel

All of this leads us to the opportunity for the channel. The problems have changed and the solutions need to evolve with them. We think that when you ask your clients to take the “unstructured data test” you will find an consulting and business opportunity right in front of you. There are now new tools that can help you scan these vast sources of data and create a single catalog. Use these tools to help clients understand the scope and scale of the challenge in front of them. Help them to think of data as a dynamic resource instead of a static one stuck in silos.

We see an evolution happening. Legacy unstructured data, new machine-generated data, IoT, real time analytics and AI/ML can be leveraged using a modern UDM approach that provides:

– Data visibility: The ability to see, search and find all of your data

– Data classification and organization: : The ability to organize, understand and govern your data

– Data protection: The ability to backup, archive and recover data

– Data mobility : The ability to programmatically move your datasets to the right places throughout their lifecycle

We have seen this opportunity manifest in the real world, with companies like Paige.AI improving cancer diagnostics by accelerating machine learning workflows, moving the data to the right place at the right time and then providing the ability to track and recreate data sets for specific ML models. This closed loop UDM capability is required to meet the challenges of the future. Best of all, this is not just a future vision, but something we are deploying today with channel partners like FusionStorm, Red8, SnowCap, and P1 Technologies.