Embedded Real-time Analytics – On-demand Analytics for the Masses thanks to HTML5

Contributed Article by John Crupi, CTO, JackBe

It’s scary how powerful browsers have gotten in the past few years. And it’s equally amazing how quickly vendors have adopted HTML5. HTML5 adoption was heavily accelerated (in my opinion) by Apple’s tenacious rejection of Flash – in other words, Apple killed Flash and replaced it with HTML5. Of course, developers care and love HTML5 for all the power and standardization it brings to the app world. The rest of us will see quantum leaps in Web applications over the next few years.

This is exactly what I’m most excited about; the tremendous power HTML5 puts in the hands of the everyday business user. It’s not just the ability to have desktop type applications in the browser; it’s the ability to interact and analyze data anywhere, anytime.

Traditionally, analyzing data has been owned by two worlds: Excel and Business Intelligence (BI) systems. Both are extremely powerful in analyzing historical data, but both arguably are weak when it comes to embedding their capability inside larger solutions. And especially weak at doing analytics on “data in motion.” Sure, you can embed pre-built spreadsheets, reports and dashboards inside solutions, but how easy is it to apply analytics on demand to live data that’s coming from “otherware” (that’s code for anywhere other than the BI system’s data source)? I like to call this “Embedded Real-Time Analytics.”

Embedded Real-time Analytics isn’t a replacement for historically-driven traditional BI and data discovery (Tableau, QlikTech and Tibco Spotfire) tools. Rather, it’s a new way to design and think about data-driven applications that puts analytic power into the users hands, when and where they need it (FYI: mobile tablets will become a predominant force here).  Let’s break this down into four architectural elements needed to make this magic happen:

1. Data and metadata about the “otherware” data
2. On-the-fly mashup capable analytics engine
3. REST enabled data sources
4. Drag & drop UI tooling for analytics

I’ll go through each of these quickly to give you a sense of how they fit together.

Data and metadata about the “otherware” data – Traditional BI systems are masters at knowing everything about data that’s in its data warehouse or cube. Data discovery vendors are great at pulling data from data warehouses and databases and shoving it and its metadata into memory for fast analysis.  However, both models rely on bringing data into their environment in order to build any solution and thus fall short when it comes to on-the-fly knowledge of any data that hasn’t been loaded into their system.

On-the-fly mashup capable analytics engine is about having a server-based engine that is able to dynamically construct the analytic run based on whatever the user has selected. This is often done with a server side data mashup engine that can mash data and apply analytics on-the-fly. Mashup engines are especially good at introspecting data in motion with little or no prior knowledge.

REST enabled data sources are becoming extremely popular and in my opinion REST is much easier and flexible to work with than the WSDL/SOAP alternative. The fact is that REST is really modeled after the Web, and it eases integration with Web UI technology.

Drag & drop analytics UI tooling is what makes it all happen and is actually the newest and most significant set of innovations making its way into embedded analytics within business solutions. The new and sophisticated set of visualizations based on HTML5 is mind-boggling. Checkout www.d3js.org to see some great data-driven examples.

Let me explain with an example: A fictitious NextGen Investment company decided to forgo building their customer analysis dashboards using their existing BI system, and instead built many independent customer analyses and HTML5 Web apps and published them to their internal app storeor “App Depot”.  The goal was to let their customer analysts “self-serve” and build their own dashboards by dragging the apps from the App Depot onto their browser-based dashboard. Very innovative and forward thinking, but since many of their analysts are technically savvy and feel very comfortable with the Web model they felt it would reduce costs and increase productivity dramatically. The development team also made the decision to take this approach because only some of the data was historical and in the BIdata warehouse. Since they wanted the ability to build real-time dashboards, some of the “otherware” data was coming from other systems and even public Web systems.

The team went a step further.  Instead of running analytics in advance (which was not economically feasible), they provided tools for the analysts to not only create their dashboards, but the tools to run analytics on any data that was in any app.

Figure 1. Customer Analysis Dashboard

They did this following the four architectural elements above and were wildly successful. According to the analysts, the ability to apply analytics by mashing data gave them insight into their customers which they never had before. Instead of waiting weeks or months for the BI team to add new analytics to existing dashboards, or worse, waiting months or years for “otherware” data to be integrated, they were able to do most of the analytics themselves and in real time, anywhere, anytime.

Don’t you want your organization to as productive as NextGen? Who wouldn’t. It’s not only plausible, but possible if you work with a “New BI” style vendor who provides technology that adheres to the above architectural elements and lets you preserve your existing BI investment.

BIO:

John Crupi is the CTO of JackBe Corporation. As CTO he is entrusted with understanding market forces and business drivers to drive the technical vision and strategy of JackBe. John Crupi has over 25 years experience in enterprise software. Previously, John spent eight years with Sun Microsystems, serving as a Distinguished Engineer and CTO for Sun’s Enterprise Web Services Practice. Mr. Crupi is co-author of the highly popular Core J2EE Patterns book, has written many articles for various magazines and is a well-known speaker around the globe. He is a frequent blogger and most recently a two-time Washingtonian DC ‘Tech Titan’.