Machine learning on IoT Edge

What is edge intelligence or edge AI?

Traditional computing models, reliant on centralized servers and cloud processing, have encountered significant hurdles when it comes to handling the ever-increasing volumes of data and ensuring real-time responsiveness. This approach often grapples with latency concerns, privacy issues associated with continuous data transmission, and the inability to swiftly process data at the source. The edge machine learning, or ML@Edge for short, is an innovative approach poised to revolutionize the landscape by decentralizing computation, placing it closer to the data source. This paradigm shift not only addresses the challenges of latency and data privacy but also enhances the efficiency of decision-making processes with the ability to generate real-time predictions. Edge deployment, particularly when integrated with Azure Cloud services, presents an opportunity to seamlessly deploy machine learning algorithms directly onto IoT Edge devices, mitigating the limitations of traditional computing models. This article seeks to delve deeper into the intricacies of the ML@Edge approach using Azure Cloud, explaining how this intersection tackles the challenges posed by traditional computing and unlocks a new era of edge intelligence.

Machine learning and MLOps overview

The integration of artificial intelligence into edge environments isn't solely about deploying ML models; it encompasses the robust orchestration of processes, known as MLOps. MLOps involves streamlining the lifecycle of ML models, from development and training to deployment and maintenance. ML@Edge deployments require a meticulous approach to ensure seamless functionality, continuous monitoring, and efficient updates—all vital components of MLOps. The details about MLOps paradigm were covered in my previous article. Azure Cloud offers a suite of MLOps tools that play a pivotal role in automating workflows, managing machine learning algorithms, and maintaining the performance of edge AI. Through this lens, the article will explore how Azure's MLOps capabilities empower organizations to efficiently manage and optimize ML models deployed in edge environments, ensuring reliability, scalability, and adaptability in diverse real-world scenarios.

How does edge computing intersect with machine learning?

ML@Edge, particularly in the context of IoT and edge computing, introduces a transformative approach to leveraging machine learning capabilities. The primary advantage lies in its ability to process data closer to its source, minimizing latency and enabling real-time machine learning inferencing, crucial in time-sensitive IoT applications like autonomous vehicles or remote healthcare monitoring. Additionally, deploying machine learning models at the edge reduces reliance on constant network connectivity, enhancing privacy by limiting data transfer to centralized servers. However, the decentralized ML@Edge approach also presents challenges. Edge devices often operate with limited computational resources and may struggle with the computational demands of complex ML models. Ensuring model efficiency, managing updates, and maintaining security across a distributed network of edge devices pose significant hurdles. Despite these challenges, the prospect of deploying ML models at the edge promises transformative capabilities, offering unprecedented opportunities for innovation and enhanced decision-making at the network's periphery.

Azure Cloud Services for Edge Deployment

Azure Cloud's suite of services, featuring Azure IoT Edge, Azure Machine Learning, Azure Stream Analytics, Azure DevOps, Azure Container Registry (ACR), and Azure Data Lake Storage (ADLS), comprises a comprehensive ecosystem specifically tailored for ML@Edge deployments. Azure IoT Edge remains a pivotal component, facilitating the seamless edge device management and deployment of containerized workloads. Leveraging Azure Machine Learning, developers access a unified platform for constructing, training, and deploying machine learning algorithms optimized for edge AI scenarios, ensuring adaptability and efficiency. Additionally, Azure Stream Analytics empowers real-time data processing at the edge, generating actionable insights without extensive data transmission. Azure DevOps orchestrates a comprehensive set of tools for continuous integration, continuous deployment (CI/CD), and collaboration, streamlining the entire development lifecycle. The inclusion of Azure Container Registry (ACR) allows for the secure storage and management of container images, facilitating their reusability for distributed edge computing. Furthermore, Azure Data Lake Storage (ADLS) provides a scalable and secure repository for diverse data types, ensuring seamless data management and access in machine learning on IoT Edge deployments. These Azure services collectively tackle the challenges associated with ML@Edge deployment, offering a robust framework, streamlined workflow, and secure storage solutions for deploying and managing ML models across diverse edge environments.

Enabling edge machine learning based on real use cases

In exploring the details of machine learning implementation, the subsequent sections delve into a comprehensive breakdown of steps and considerations crucial for deploying ML models on edge computing infrastructure using Azure Cloud services. From preparing ML models for optimized ML@Edge operation to leveraging Azure IoT Edge for seamless deployment and management, these sections intricately explore techniques, best practices, and the Azure-specific details essential for successful implementation. Moreover, it navigates through strategies for ensuring security, efficiency, monitoring, and managing edge-deployed models within the Azure ecosystem, presenting a holistic approach to addressing the intricacies of deploying ML models on edge environments.

Preparing ML Models for Edge Deployment

Optimizing machine learning models for ML@Edge deployment requires a meticulous approach to adapt them for resource-constrained devices without compromising their predictive capabilities. One fundamental technique involves model quantization, a process that reduces the precision of numerical representations within the model, effectively decreasing computational complexity and memory requirements. Additionally, optimizing models through techniques like pruning, which involves eliminating unnecessary parameters and connections, significantly reduces the model's size while maintaining its functionality. Compression algorithms, such as weight sharing or Huffman coding, further minimize model size by representing information more efficiently. Choosing lightweight neural networks architectures, like MobileNet or EfficientNet for deep learning applications, designed specifically for ML@Edge inference, also plays a crucial role. These architectures emphasize computational efficiency without sacrificing model accuracy, making them well-suited for deployment in edge environments with limited computational resources. By leveraging these techniques in tandem, developers can ensure that their machine learning models are tailored for efficient execution on smart IoT devices, balancing computational demands with the constraints of edge environments for optimal performance.

How to deploy ML Models on Azure IoT Edge?

The ML@Edge deployment process starts with data scientists utilizing Azure Machine Learning's Jupyter Notebooks or other integrated development environments (IDEs) to write code for data preprocessing, model training, and evaluation. Azure Machine Learning allows for easy access to various machine learning frameworks such as TensorFlow, PyTorch, or Scikit-learn, simplifying the development and ML computation process. The sensor data for training and testing the model is stored in connected ADLS and comes directly from the edge device.

Once the development phase is complete, the model needs to be containerized for deployment. Docker, a popular containerization platform, allows the packaging of the machine learning model and its dependencies into a lightweight, portable container. In Azure Machine Learning, data scientists can use Docker to create containers that encapsulate the model, its runtime environment, and any necessary libraries or dependencies. This can be achieved using either native method available in the Azure Machine Learning SDK or by creating the image independently from the SDK using a custom Dockerfile.

In general, the image consists of the environment file containing the dependencies, the file containing the ML model information (weights) and the scoring script, which will initialize the model, perform the inference and return the outputs. The sample scoring script can be as follows:

import os
import logging
import json
import joblib


def init():
	"""
	This function is called when the container is initialized/started, typically after create/update of the deployment.
	You can write the logic here to perform init operations like caching the model in memory.
	"""
	global model
	# AZUERML_MODEL_DIR is an environment variable created during deployment.
	# It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)
	model_path = os.path.join(
		os.getenv("AZUREML_MODEL_DIR"), "sklearn_regression_model.pkl"
	)
	# Deserialize the model file back into a sklearn model
	model = joblib.load(model_path)
	logging.info("Init complete")

def run(raw_data):
	"""
	This function is called for every invocation of the endpoint to perform the actual scoring/prediction.
	In the example we extract the data from the json input and call the scikit-learn models's predict()
	method and return the result back.
	"""
	logging.info("Request received")
	try:
		data = json.loads(raw_data)['telemetry']
		result = model.predict(data)
		logging.info("Request processed")
		return result.tolist()
	except Exception as e:
		error = str(e)
		return error

Building the image for ML@Edge deployment using Azure Machine Learning Python SDK requires running the ContainerImage.image_configuration() along with Image.create() methods. The image will be created and registered to the connected ACR automatically. If you want to understand how the image is created better, this article explains it in detail. However, this method works only with the v1 version of the SDK, posing a risk to be deprecated in the future.

from azureml.core.image import Image, ContainerImage
from azureml.core import Model

image_config = ContainerImage.image_configuration(runtime = "python",
		execution_script = "score.py",
		conda_file = "env.yml"
		)

model = Model(workspace=ws, name="my-sklearn-model", version="1")

image = Image.create(name = "edgemlsample",
		models = [model],
		image_config = image_config,
		workspace = ws)

image.wait_for_creation(show_output = True)

if image.creation_state == "Failed":
	print("Image build log at: " + image.image_build_log_uri)

‍

More robust way to create and maintain the docker image, working both with v1 and v2 versions of the SDK is to write a custom Dockerfile. There are no strict rules how this should be achieved other than that the image should be able to perform the edge inference - therefore the whole module has to implement the edge inputs, outputs and message processing logic. The sample way of achieving this is to use the official inference Azure ML image as base, as presented below:

FROM mcr.microsoft.com/azureml/o16n-base/python-slim:latest

# Setup variables and arguments
ENV AZUREML_MODEL_DIR=/var/azureml-app/azureml-models
ARG model_file

# Copy required files
RUN mkdir -p 'var/azureml-app'
COPY main.py score.py env.yml /var/azureml-app/
COPY ${model_file} /var/azureml-app/azureml-models/

# Prepare environment to run model
RUN sed -i '/^\s*-\s*python\s*[<>=]/d' '/var/azureml-app/env-yml' && cat '/var/azureml-app/env.yml'
RUN conda update conda && CONDA_ROOT_DIR=$(conda info --root) && if [ -n
"$AZUREML_CONDA_ENVIRONMENT_PATH" ]; then conda env update -p "$AZUREML_CONDA_ENVIRONMENT_PATH" -f
'/var/azureml-app/env.yml'; else conda env update -n base -f '/var/azureml-app/env.yml'; fi &&
conda clean -aqy
CMD ["runsvdir", "/var/runit"]

‍

After containerizing the model, Azure Container Registry serves as a private registry to store and manage Docker container images securely. ACR allows data scientists to push their Docker images to a centralized registry within Azure, ensuring accessibility, version control, and security of the containerized model. It also integrates seamlessly with Azure Machine Learning for ML@Edge deployment purposes.

With the image ready, deployment manifests are created, specifying the modules, including Docker containers encapsulating trained models, that need to be deployed onto the IoT Edge devices. These manifests contain the necessary configuration settings, modules, and routes required to orchestrate the deployment and execution of models on ML@Edge devices within the Azure ecosystem. The process is later straightforward, as with any other custom module created for IoT Edge.

Another approach, other than direct ML@Edge model deployment, is the utilization of inference servers. It serves as a scalable and interoperable platform for executing predictions based on input data, ensuring compatibility with various applications and systems through standardized interfaces like ONNX or TensorFlow Serving. The server exposes an API which can be used for machine learning inferencing and can be either directly deployed on an edge device or as an online endpoint, utilizing the cloud to edge connection in both ways. There are some existing commercial implementations of ready-to-use inference servers with the most popular one - NVIDIA Triton.

Ensuring Security and Efficiency in Edge Deployments

Ensuring the security and efficiency of ML@Edge deployed models is paramount in maintaining robust and reliable operations. Azure offers various mechanisms and best practices to optimize efficiency and achieve secure model deployments. Azure IoT Edge employs security mechanisms such as secure boot, encryption, and authentication protocols to safeguard edge devices and communication channels, mitigating potential vulnerabilities. Data encryption, both in transit and at rest, ensures data privacy and integrity, crucial when processing sensitive information at the edge. Additionally, implementing Azure Security Center and Azure Sentinel provides centralized security monitoring, threat detection, and response capabilities, fortifying the edge environment against evolving security threats.

Efficiency in ML@Edge deployments is augmented through Azure services that optimize resource utilization and streamline operations. Azure IoT Edge modules facilitate containerized execution, allowing for the efficient distribution of computational resources on edge devices. Leveraging Azure Stream Analytics enables real-time data processing and filtering at the edge, optimizing data transmission and reducing bandwidth requirements. Azure Monitor complements these efforts by providing insights into system performance and resource utilization, facilitating proactive optimization and ensuring efficient operation of machine learning models on edge devices. By integrating these security measures and efficiency-boosting services, organizations can establish a secure, efficient, and resilient environment for deploying machine learning models on edge devices within the Azure ecosystem.

Monitoring and Managing AI-enabled IoT devices

Effective monitoring and management of ML@Edge deployed models are crucial for ensuring optimal performance and reliability. Azure offers robust tools and services for monitoring and managing these deployments, providing insights into performance, health, and usage metrics. Azure Monitor plays a pivotal role in monitoring the health and performance of deployed modules on edge devices. It enables real-time monitoring of metrics, logs, and diagnostics, empowering proactive identification and resolution of issues that may impact model performance. Additionally, Azure IoT Hub facilitates remote management of edge deployments, allowing for seamless updates, configurations, and troubleshooting of machine learning models deployed on diverse edge devices.

Deep learning on edge devices

Deploying deep learning models to edge environments represents a groundbreaking advancement poised to revolutionize various industries. Modern production lines often contain cameras for monitoring and anomaly detection, creating a space for machine learning implementation of convolutional neural networks for image classification. However, the computational complexity of deep learning models poses a significant hurdle when deploying them on resource-constrained edge devices with limited memory, processing power, and energy constraints. Optimizing these models for ML@Edge environments requires tackling issues related to model size, computational efficiency, and real-time performance. Furthermore, ensuring consistent and reliable inference results while managing hardware variations across different edge devices presents a considerable challenge. Balancing the trade-offs between model complexity, accuracy, and efficiency becomes crucial, often requiring techniques like model quantization, pruning, or employing specialized architectures designed to make their deployment to edge better. Addressing these challenges effectively is pivotal to unlock the transformative potential of deploying deep learning models at the edge, enabling intelligent applications for decision-making and data processing in real-time, while optimizing resource utilization and maintaining reliability.

Summary and Future Perspectives

Machine learning on IoT Edge environments deployment using Azure Cloud services represents a pivotal advancement in modern computing paradigms. This convergence facilitates real-time decision-making, enhances data privacy, and optimizes resource utilization, unlocking new possibilities across industries. The amalgamation of Azure Machine Learning, Azure IoT Edge, Azure Container Registry, and other Azure services not only addresses the challenges of deploying models at the edge but also fosters innovation and agility in deploying and managing machine learning solutions. Looking ahead, the trajectory of edge intelligence integrated with machine learning on Azure promises continued growth and transformation. As technology evolves, further advancements in ML@Edge enabled devices, coupled with Azure's capabilities, are poised to revolutionize industries, enabling autonomous systems, predictive maintenance, and personalized user experiences. Embracing these advancements will empower organizations to harness the full potential of edge inference, driving unparalleled efficiency, intelligence, and value in the ever-evolving landscape of machine learning deployments.