IT Briefcase Exclusive Interview: What I Learned from Running the World’s Largest Hadoop Community

Change is what sets an innovative company apart from a competitor, and is a constant with technology. As open source technology becomes widely adopted, its core features – choice and collaboration – will become the norm for modern technology.

• Q. What is the future of the open source community?

The most optimistic view for its future envisions open source disrupting all software sales and deployments. It’s not likely in the near future, but there are hundreds of thousands of developers working to innovate around open source libraries.

With the emergence of a source-code repository service like GitHub, the open source community is becoming the central place for software innovation. In particular, many recent innovations in infrastructure and middleware—where many developers share their work—were born as open source projects, including Docker and Kubernetes. For a legacy example, there’s also Linux. I definitely see this trend continuing and expanding.

Developers—who write, contribute and maintain the code—will continue to be the primary influencers in the community. They’ll find the community serving as the innovation hub, the best place to socialize with other developers, and the ideal place to stay informed about the latest news and ideas. The JavaScript developer community is a good example. Every couple of years, a new framework comes out of that community, like Angular.js from Google, and React.js from  Facebook.

The most pessimistic view of the future envisions open source continuing to struggle to monetize the code base. Open-source software companies need to figure out how to sustain their vision and support their work by adding value and making money on top of the open source code. They also need to figure out how to increase their enterprise penetration and increase market share so they can compete successfully with dominant software vendors like Oracle and SAP.

• Q. What lessons have you learned from founding the world’s largest Hadoop user group?

Founding and running an open source community was a unique experience because there was no commercial mindset—individuals were motivated solely by a desire to connect with others who had a common technology interest and to share their knowledge and experiences. That was a key lesson: Keep the group as neutral as possible and ensure people are connecting for purely technical purposes.

People came together from various companies and, with no supervisors or authority figures, made progress on many fronts. As we added more and more activities as a community, we saw members doing innovative things in their jobs, and bringing those innovations back to the community. Many early Hadoop developers and users generously gave talks and came to conferences, which strengthened the community. I believe our non-commercial approach had a lot to do with their willingness to share what they knew.

As far as other lessons learned, if I had it to do over again, I would have tried harder to expand the community beyond Japan and Silicon Valley. I would have tried to connect with places like Europe, China and Singapore, where there are a lot of developers using Hadoop. Adding those people would have added their knowledge to the group and created opportunities for our members to make more social connections around the world. In addition, I’d also recommend making an effort to regularly incorporate new members into the community and make them feel welcome so that you can expand beyond a core group.

• Q. How can companies utilize open source technology?

Companiesneed to start out by defining their requirements and only then comparing open source and commercial offerings. If only commercial software meets your requirements, then the decision is made. But if there are both commercial and open source solutions that fit, you can then evaluate the cost, features, benefits, options, and so on. You can even consider a hybrid option.

It’s possible nowadays to find open source technology to fit a wide range of IT projects, and many companies are using open source to develop highly efficient and effective solutions. However, open source isn’t perfect: While it’s comparatively easy to start out with open source, you’re assuming the risk of operating and troubleshooting your deployment and environment. Although companies can clearly benefit from innovations happening in open source, they need to have solid engineering skill sets to dig into the underlying source code if they’re going to successfully build and maintain a software solution that meets their business needs.

Kaz Ohta, Founder and CTO of Treasure Data

Kaz Ohta is Treasure Data’s CTO and co-founder. He is also the founder of the Japanese Hadoop User Group, the world’s largest such group. Kaz is an acknowledged expert on distributed and parallel computing, and combines his knowledge of these technologies and Hadoop with the conviction that the service model is the only way to bring big data analytics to the mass market.