Efficient processing of IoT data streams: Architectures, algorithms, and future directions

Abstract

The rapid proliferation of Internet of Things (IoT) devices has led to an exponential increase in the volume and velocity of data streams, necessitating real-time processing. Extracting actionable insights from these continuous data flows is essential for enabling intelligent decision-making across applications such as smart homes, industrial automation, and intelligent transportation systems. However, the resource-constrained nature of IoT devices and edge nodes—characterized by limited computational power, memory, and energy—presents significant challenges in achieving efficient and accurate data Stream Processing (SP). This paper presents a comprehensive review of state-of-the-art approaches for effectively managing and processing IoT data streams. We examine various architectural paradigms, including edge computing and distributed SP systems, designed to handle high-throughput, low-latency data streams. Additionally, we explore advanced algorithms, such as machine learning and deep learning techniques optimized for real-time analysis, prediction, and anomaly detection, as well as Approximate Computing (AC) methods and specialized data structures like Bloom Filters (BFs) and sketches that enhance resource utilization and reduce memory overhead. Furthermore, this review highlights critical challenges in the field, including data privacy, security, scalability, and fault tolerance, while identifying promising research directions toward building more scalable, energy-efficient, and intelligent IoT data SP systems. By synthesizing recent advancements and outlining future opportunities, this work serves as a valuable resource for researchers and practitioners seeking to address the complexities of real-time IoT data analytics.