Problem Statement
- Real-time streaming data from IOT sensors monitoring Cooking equipment like fryers and ovens, Hot and Cold Storage containers, Transport vehicles that are continuously sending data on Power, Temperature, HVAC, etc
- High Speed low latency real-time data processing and analytics for IOT data
- Order of 100K messages per second from IOT Devices for monitoring food storage and food processing in hot and cold environments, 24 x 365
- Need for real-time parsing, enrichment, predictive monitoring and deliver data in near-real time to Advanced analytics and Data Science users.
Solution Overview
- Real-time Data processing layer using Kafka, Spark, Cassandra, Redis
- Data enrichment to add master data attributes, tagging for asset and process, Business rules-based Data Quality checks.
- Real time Data Aggregation to build calculated aggregate metrics, Time window functions, moving averages and checks with Thresholds and allow Search and real time dashboard functionality
- Analytics Layer for near-real time data analysis, AI/ML modeling
- Persistent Data Storage on Cassandra for historical Data Analysis.
Outcomes
- Seamless ingestion, processing of real time data with high volume/second
- Technology agnostic robust architecture for high performance, scalability
- Near real-time Data enrichment, in-process analytics, monitoring dashboards and enabling AI/ML based advanced analytics.
