Table of Contents



Introduction


Microsoft Azure provides a variety of cloud computing services including artificial intelligence, machine learning, IoT, storage, security, networking, media, integration and so on. Among tens of thousands of IoT end devices, Raspberry Pi is one of the most popular and useful platforms that used in many areas because of its low cost, modularity, and open design. Raspberry Pi 4 is the latest powerful model with Quad core Cortex-A72 64-bit processor, 2 HDMI ports, 40 pin GPIO header and hardware video decode capability. Azure Custom Vision is an image recognition service that lets you build, deploy, and improve your own image identifier models. An image identifier applies labels to images, according to their detected visual characteristics. Unlike the Computer Vision service, Custom Vision allows you to specify your own labels and train custom models to detect them.
In this article, we will walk you through the steps required to send real time captured image to Azure Custom Vision on Raspberry Pi 4 model B with Microsoft Azure Custom Vision Client Library for Python. Additionally, we will mark the identified target with red rectangle and send the marked image to Azure Blob Storage. With Python Azure IoT SDK, we can also send the number of the identified target to Azure IoT Hub.

Prerequisites


1. Raspberry Pi 4 model B
2. Azure Custom Vision Client Library for Python
3. Azure IoT Device SDK for Python
4. Azure Storage Blobs client library for Python
5. USB Web Camera or CSI Camera

Hardware Connection


In this project, we use Microsoft LifeCam HD-3000 USB web camera. This true HD camera features 720p HD video chat for a true HD-quality experience and TrueColor technology that automatically delivers a bright and colorful video. 

Install Necessary Libraries


Open your Raspberry Pi terminal and install the following Libraries:
1.pip3 install azure-cognitiveservices-vision-customvision
2.pip3 install azure-iot-device
3.pip3 install asyncio
4.pip3 install azure-storage-blob
5.sudo apt-get install fswebcam

Create Necessary Services on Azure


In this project, we will use Azure Custom Vision, Azure IoTHub, as well as Azure Storage services. Please refer to the section “Create an Azure IoTHub and Register a New Device” in this wiki article “Azure Cloud Services for Raspberry Pi 4: How to send sensor data to Azure IoT Hub” to create services for Azure IoTHub. And the “Create an Azure Storage Account” section in this wiki article “Azure Cloud Services for Raspberry Pi 4: How to capture and send image to Azure Blob Storage” will guide you to create Azure Storage service. For Azure Custom Vision, this project “Bird Detector Based on Microsoft Azure and Raspberry Pi” is a good start. The Custom Vision service used in this wiki article takes advantage of the model established above.

Create and Debug Python Code on Raspberry Pi


Currently, there are several Python IDE on Raspberry Pi. Thonny Python IDE is bundled with the Raspbian OS. We can launch it by click Programming -> Thonny Python IDE. Then copy and paste the following code.
01.from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
02.from msrest.authentication import ApiKeyCredentials
03.from azure.storage.blob import ContentSettings, BlobClient
04.from azure.iot.device import Message
05.from azure.iot.device.aio import IoTHubDeviceClient
06. 
07.import os
08.from PIL import Image
09.from PIL import ImageDraw
10.import matplotlib.pyplot as plt
11.import asyncio
12. 
13.# custom vision api
14.credentials = ApiKeyCredentials(in_headers={"Prediction-key": "Your Key"})
15.predictor = CustomVisionPredictionClient("https://***.cognitiveservices.azure.com/", credentials)
16.projectID = "******"
17.publish_iteration_name="Iteration*"
18. 
19.#Bolb Storage
20.conn_str="DefaultEndpointsProtocol=https;AccountName=***;AccountKey=***;BlobEndpoint=https://***.blob.core.windows.net/;QueueEndpoint=https://***.queue.core.windows.net/;TableEndpoint=https://***.table.core.windows.net/;FileEndpoint=https://***.file.core.windows.net/;"
21.container_name="raspberrypic"
22.blob_name="bird"
23.# Create instance of the device client
24.blob_client = BlobClient.from_connection_string(conn_str, container_name, blob_name)
25. 
26.# Azure IotHub
27.CONNECTION_STRING = "HostName=***.azure-devices.net;DeviceId=***;SharedAccessKey=***"
28.# Create instance of the device client
29.iothub_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
30.target_num = 0
31.PAYLOAD = '{{"No. of target detected": {target_num}}}'
32. 
33.async def CustomVisionApp():
34. 
35.    print("===== Describe an image - camera =====")
36.    # capture the image with USB webcamera
37.    a=os.system("fswebcam --no-banner -r 1280x720 capture.jpg")
38.    print(a)
39.    # open and detect the captured image
40.    with open("capture.jpg", mode="rb") as captured_image:
41.        results = predictor.detect_image(projectID, publish_iteration_name, captured_image)   
42.    # Display the results.
43.    for prediction in results.predictions:
44.        global target_num
45.        if prediction.probability>0.6:
46.            target_num += 1
47.            print("\t" + prediction.tag_name + ": {0:.2f}%".format(prediction.probability * 100))
48.            bbox = prediction.bounding_box
49.            im = Image.open("capture.jpg")
50.            draw = ImageDraw.Draw(im)
51.            draw.rectangle([int(bbox.left * 1280), int(bbox.top * 720), int((bbox.left + bbox.width) * 1280), int((bbox.top + bbox.height) * 720)],outline='red',width=5)
52.            im.save("detect.jpg")
53.    de=Image.open("detect.jpg")
54.    plt.figure("Result")
55.    plt.imshow(de)
56.    plt.show()
57.     
58.    #data formating
59.    data = PAYLOAD.format(target_num=target_num)
60.    message = Message(data)
61.    # Send a message to the IoT hub
62.    print(f"Sending message: {message}")
63.    await iothub_client.send_message(message)
64.    print("Message successfully sent")
65.      
66.    # upload the image to Azure Blob Storage, Overwrite if it already exists!
67.    image_content_setting = ContentSettings(content_type='image/jpeg')
68.    with open("detect.jpg", "rb") as data:
69.        blob_client.upload_blob(data,overwrite=True,content_settings=image_content_setting)
70.        print("Upload completed")
71.     
72.if __name__ == '__main__':
73.asyncio.run(CustomVisionApp())

Please do substitute the connection string with yours that created in section “Create Necessary Services on Azure”. Press Run or Debug button to start the process. You will see the output in Shell window as shown in Fig. 1.

Fig. 1 Debug information on Shell window
The real time image will be captured and transmitted to Azure Custom Vision service. Once the results are returned, the application will iterate through the results to find the target whose prediction probability is above 60%. The number of the identified target will be reported to Azure IoT Hub. Additionally, the identified target will be marked with red rectangle and transmitted to Azure Blob Storage. 
It’s worthy to note that the resolution of the real time captured image is 1280x720 here. You should adjust this parameter according to your web camera. Furthermore, the codes that deal with the returned results to mark the target should also be modified. 
If there exists a target whose prediction probability is above 60%, you will see the picture with the marked target on the screen as follows.

Fig. 2 The marked target on the screen
Then, by closing the picture, the application will continue to send message to Azure IoTHub (show in Fig. 3) and upload the marked image to Azure Blob Storage (show in Fig. 4).
Fig. 3 Message received by Azure IoTHub
Fig. 4 Uploaded image on Azure Blob Storage

Summary


In this tutorial, we have presented how to send real time captured image to Azure Custom Vision, mark the identified target with red rectangle according to the returned results, send the marked image to Azure Blob Storage and send the number of the identified target to Azure IoTHub.

Resources


1. MS Docs for Azure Table storage.
2. MS Docs for Create a storage account.
3. MS Docs for Manage storage account access keys. 
4. MS Docs for Azure Custom Vision.

See Also


1. Azure Cloud Services for Raspberry Pi 4: How to send sensor data to Azure IoT Hub
2. Azure Cloud Services for Raspberry Pi 4: How to capture and send image to Azure Blob Storage
3. Azure Cloud Services for Raspberry Pi 4: How to send telemetry to Azure Storage Table
4. Bird Detector Based on Microsoft Azure and Raspberry Pi