High Speed low latency real-time data processing and analytics for IOT data for a Large US Retailer

Problem Statement

  • Real-time streaming data from IOT sensors monitoring Cooking equipment like fryers and ovens, Hot and Cold Storage containers, Transport vehicles that are continuously sending data on Power, Temperature, HVAC, etc
  • High Speed low latency real-time data processing and analytics for IOT data
  • Order of 100K messages per second from IOT Devices for monitoring food storage and food processing in hot and cold environments, 24 x 365
  • Need for real-time parsing, enrichment, predictive monitoring and deliver data in near-real time to Advanced analytics and Data Science users.

Solution Overview

  • Real-time Data processing layer using Kafka, Spark, Cassandra, Redis
  • Data enrichment to add master data attributes, tagging for asset and process, Business rules-based Data Quality checks.
  • Real time Data Aggregation to build calculated aggregate metrics, Time window functions, moving averages and checks with Thresholds and allow Search and real time dashboard functionality
  • Analytics Layer for near-real time data analysis, AI/ML modeling
  • Persistent Data Storage on Cassandra for historical Data Analysis.

Outcomes

  • Seamless ingestion, processing of real time data with high volume/second
  • Technology agnostic robust architecture for high performance, scalability
  • Near real-time Data enrichment, in-process analytics, monitoring dashboards and enabling AI/ML based advanced analytics.
High Speed low latency real-time data processing
case studies

See More Case Studies