GoFFish: Graph-Oriented Framework for Foresight and Insight using Scalable Heuristics
Sensors and online instruments performing high fidelity observations are contributing in a large measure to the growing big data analytics challenge. These datasets are unique in that they represent events, observations and activities that are related to each other while being recorded by independent data streams. Existing data processing frameworks such as MapReduce that operate on file or row based data do not lend themselves to scalable analytics over such an interconnected web of stream-based data.
We propose GoFFish, a scalable graph-oriented analytics framework that is well suited for trawling over reservoirs of inter-connected data that are fed by event data streams. Our framework will help design optimized graph algorithms that leverage the specialized graph oriented data store, GoFs, and are based on the proposed graph programming abstraction, Gopher, that can be used by analysts to intuitively and rapidly compose graph and event analytical models. The composed application will enhance data parallel analytics at scales far superior to traditional MapReduce models using a novel distributed data partitioning approach based on edge distance heuristics. This will allow unprecedented insight from the reservoirs of stream data for commanders to perform causal graph analysis and strategic planning. Further, we propose to close the loop between insight and foresight by coupling event patterns mined from historical stream reservoirs by graph analytics with realtime event streams from sensors. Such an online stream analytics engine will provide operational leaders with augmented situation awareness and advanced warning about impending conditions.
- Yogesh Simmhan, Project Manager & Architect
- Marc Frincu, Co-Manager
- Alok Kumbhare, Architect
- Charith Wickramaarachchi
- Hsuan-Yi Chu
- Nam Ma
- Soonil Nagarkar
- Santosh Sathyavijayanagaram Ravi
This project is supported by the DARPA XDATA Program (PIs: Viktor Prasanna, Yogesh Simmhan & Raghu Raghavendra).