How can Big Data cross the Oceanic Gap?

Featured article by Yves de Montcheuil, Vice President of Marketing, Talend

Geography 101: Which body of water separates America from Europe? No, this is not a post about Data Lakes, Data Reservoirs, or even the Data Ocean. The Atlantic Ocean is not an analogy for where big data is stored – it is a gap that big data is having trouble crossing. Not the data itself, but the use cases and project typologies of big data.

After all these years in the software industry, I am no stranger to the lag in technology adoption between the United States and Europe. Most new products, born in Silicon Valley or any of its regional rivals: Silicon Alley, Route 128 Corridor, etc. – are quickly adopted across the U.S. continent but get stuck trying to jump across the ocean to reach Europe. There, adoption starts more slowly, first getting footing in the U.K. (I won’t get into the debate as to whether this is Europe…) and finally reaching (the rest of) Europe.

Big data technologies are definitely California-born. The Internet giants invented Hadoop and most NoSQL databases. Over the years, adoption has spread from “new economy” companies to almost all industries, to the point that big data is now used by most venerable organizations, as evidenced by the 3,200-strong crowd attending Hadoop Summit earlier this month in San Jose. The European edition of the same Hadoop Summit, held in Amsterdam in April, was much smaller: 800 attendees. Yes, 800 attendees would thrill many a conference organizer, but the fact remains that the European event was just 25 percent of the size of the U.S. event. I have been to many technology conferences and while European events are typically smaller than equivalent U.S. ones, the disparity is typically much less acute.

This difference in scale is confirmed by a data point from a poll we at Talend ran during a recent webinar, in which we asked Talend clients to vote for the most valuable new feature in our just-announced version 5.5. Among the choices was experimental support for Apache Spark. This was deemed the most valuable feature for 35 percent of U.S. participants, but only for 21 percent of European ones. For now at least, Spark (which enables real-time and operational use cases of big data) remains stuck on the beach, unable to cross the abyss of the ocean.

We are dealing here with a classic chicken-egg problem. Not only are businesses looking for use cases, they also need people with experience. Because early adopters of big data are the Internet giants (based in the U.S.), expertise exists in the U.S. However, despite hefty compensation expectations for big data experts, the spread is slowly starting. Vendors’ investment in marketing is allowing the spread of use cases and successes, providing a catalyst for new big data projects.

What is ultimately needed is a migration of talent across the ocean. Since we started this post with geography, let’s wrap it up with history. Let’s imagine a modern-day reverse Mayflower that would sail the Atlantic Ocean, loaded with data scientists and other big data experts, bringing much needed expertise to European companies. But why did most emigrants undertake the voyage to the United States? In quest of a better world, to flee persecution, poverty, or famine – conditions which are doubtful to be found en-masse among Californian software engineers! However, some were in it for the adventure. Perhaps big data adventurers will dare to conquer Europe.

Yves de Montcheuil is the Vice President of Marketing at Talend, the recognized leader in open source integration. Yves holds a master’s degree in electrical engineering and computer science and has 20 years of experience in software product management, product marketing and corporate marketing. He is also a presenter, author, blogger, social media enthusiast, and can be followed on Twitter: @ydemontcheuil.