7 Forces Shattering the BI and Analytics ‘Caste’ System

For the past two decades, there has been an informal segregation of duties seen in the BI and analytics world, with three primary classes of users. At the bottom, there are line-of-business managers and employees, working with basic spreadsheet users, piecing together ad-hoc analysis based on data wherever available. In the middle tier are statisticians, analysts, quants, and power users, employing OLAP cubes, pivot tables, and other front-end tools to slice and dice data to arrive at a deep analysis. At the top of the food chain are upper-level managers and executives, who employ portals and dashboards, digesting nuggets of insights from pretty graphs and bar charts.

Now, new thinking and emerging technologies across the enterprise are shattering this class system, which is unsustainable and illogical in an era when data is driving entire businesses, much of which is needed on a real-time basis. The BI market, historically insular but fragmented, is opening up converging with forces sweeping across the entire technology and enterprise landscape, from cloud to mobile to Big Data. There is a push to proliferate analytics across entire enterprises, diving into endless streams of digital data covering everything from financials to corporate lunch room operations. As many in the industry now put it: everything is connected, and everything is digital, everything can be measured.

Here is a glimpse of the seven overlapping forces now reshaping business intelligence as we know it, opening up new possibilities for enterprise insights:

Pervasive BI: First, there’s the emerging concept that BI doesn’t have to be seen at all, running behind the scene, hooked into applications and automatically providing insights or even predictions that affect the response of the application. Pervasive BI code would likely to be built into packaged solutions, or running in the background, delivering results to front-line employees and decision makers. This is where real-time BI also will be seen.

Automated decision making: The next step in pervasive BI is to enable machines employ predictive analytics to make decisions – without human intervention. Tied into rules engines, decision management systems can be employed to make low-level or day-to-day operational calls on behalf of the business – such as extending credit to a customer or even upsell offers. This is often a self-learning process in which rules are updated as more transactions are conducted.

“Big Data” analytics:  Typically, BI solutions have loaded pieces of samples of data sets for analysis. The notion of Big Data analytics – enabled by frameworks such as Hadoop – is that entire data sets, be it gigabytes, terabytes, or even petabytes, can be examined and mined, all at once. New technology solutions such as in-memory processing is pushing the limits as to what a single machine or cluster of machines can process in one session. The availability of Big Data for analysis also opens up new types of data types for analysis that formerly were too expensive or cumbersome to capture – such as machine, sensor or device output, or social media data.

Analytic clouds: Storing and managing BI and analytic applications data in the cloud offers a way for organizations to avoid the costs and complexities of storing and managing this data. Whether it’s through a Software-as-a-Service provider, a hosted cloud, or a private cloud, organizations can take advantage of greater scalability and faster implementation times.

BI mobile apps:  A still relatively untapped, frontier of BI is mobile apps. As consumers, business users are well-acquainted with apps that can be quickly purchased from app stores and are simple and easy to use. The arrival of similar apps that can provide access and analysis to back-end enterprise data – with the ability to provide insights with a single touch, and no text entry, will be a boon to delivery of analytics to all levels of employees across the enterprise.

Self-service BI: The rise of mobile apps and cloud is an important aspect of self-service BI, which means end-users are increasingly demanding the ability to be able to build and generate their own reporting without waiting for delivery of forms from their IT departments.

Social BI: BI and analytics efforts would be missing a big piece of the picture if it did not address the wealth of social media data flowing through organizations. This includes sentiment analysis and other  applications to monitor interactions on external social media sites, to determine reactions to new products or predict customer needs.

As mentioned above, these seven forces overlap and are converging, making actionable data and insights available to any end-user or application, when are where it is needed.

Joe McKendrick is an author and independent researcher, covering innovation, information technology trends and markets.  Much of his research work is in conjunction with Unisphere Research/ Information Today, Inc. for user groups including SHARE, Oracle Applications Users Group, Independent Oracle Users Group and International DB2 Users Group. He is also research analyst with GigaOM Pro Research.

He is a regular contributor to Forbes.com, and well as a contributor to CBS interactive, authoring the ZDNet “Service Oriented” site, and CBS interactive’s SmartPlanet site.

Joe is a co-author of the SOA Manifesto, which outlines the values and guiding principles of service orientation in business and IT.

In a previous life, he served as communications and research manager of the Administrative Management Society (AMS), an international professional association dedicated to advancing knowledge within the IT and business management fields. He is a graduate of Temple University.