Developing a big data application for data exploration and discovery

Since I re-joined IBM in 2012, I have been involved in a few Big Data related projects. For those interested in learning more about IBM's Big Data Platform, below is an article I co-authored, that introduces you to developing a big data application for data exploration and discovery using BigIsights and Data Explorer.

If you've been following many of the early case studies around big data, you may have come to believe the saying that "you don't know what you don't know." Indeed, big data applications often focus on gleaning business insights from data that might otherwise be discarded or ignored for a variety of reasons. Increasingly, companies are looking to develop a comprehensive information management strategy that involves more than simply exploring or analyzing big data. Specifically, they want to integrate big data into their overall information management strategies alongside existing data systems, including relational DBMSes, enterprise content management systems, data warehouses, etc.

This article examines one facet of that challenge, outlining an architecture and approach for indexing big data and traditional data sources, as well as providing a web-based interface for discovering new insights across these disparate data sources. In particular, it describes how Data Explorer, a data discovery platform, can index data managed by InfoSphere BigInsights, enabling persistent forms of big data to be combined with existing enterprise data. Both Data Explorer and BigInsights are key components of IBM's big data platform, so let's start with an overview of this platform and these two key offerings.

Full article available in IBM developerWorks