Object Detection using Tensorflow Lite on Raspberry Pi

 

In this blog, we are going to use one of the object detection mechanism inside our raspberry pi.  Using Object Detection we can identify the different objects within the image or video we captured from the camera. 

Based on the identified object we can create different real-time applications like home security, traffic signals where we use security cameras to detect or find suspicious objects or incidents over the video feed depend on the feature you wanted to use in your product.

Before going to start the practical use case let's understand what is Object Detection and how it will help us to find the object identification using our small embedded device.

We are going to use Tesnroflow machine learning platforms inside our raspberry pi to detector the different objects using the live camera from the device.  Here we will be using TensorFlow, which is an end-to-end open-source machine learning platform. Tensorflow also provides an open-source library for machine learning which runs the Object Detection API to perform object detection and identified the object from the live camera feed. 

Basically,  These object detection mechanisms use Artificial intelligence technology (AI), which refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions.  

Artificial Intelligence applies machine learning, deep learning, and other techniques to solve actual problems in the fields of finance, healthcare, eCommerce, engineering services, and so on.  

AI can perform tasks that can be used in the fields of robotics, control systems, scheduling, data mining, logistics, speech recognition, facial recognition, and many others.

AI uses machine learning techniques to solve the problem by learning from experience and improves itself automatically through a previous experience. That is why machine learning is called the branch of Artificial Intelligence. There are many other techniques to use the AI for solving the real-time complex task which can not be solved using existing code or algorithm.

Now let's begin with the practical implementation for one of the small use cases of machine learning using Tensorflow. 

Introduction: 

This guide provides step-by-step instructions for how to set up TensorFlow’s Object Detection API on the Raspberry Pi. By following the steps in this guide, you will be able to use your Raspberry Pi to perform object detection on live video feeds from a web camera. 

What you will need:

• Raspberry Pi Single Board computer ( I chose Raspberry Pi 3 Model B+)
• Raspberry PI 5MP Camera Board Module.                                   
• A high-quality 5V/2.5A micro USB power supply
• Windows PC for Remote Connection.
• Xshell software for SSH remote access to Raspberry Pi. 
• VNC Viewer for GUI access to Raspberry Pi.

The guide walks through the following steps to set up the TensorFlow Lite on Raspberry Pi:

1. Connecting Pi Camera to Raspberry Pi
2. Remote connection to Raspberry Pi using SSH.
3. Update raspberry pi.
4. Install the Python 3.7 (or latest version)
5. Install the OpenCV for Image processing Libraries.
6. Protocol Buffer Compiler Installation
7. Installation of TensorFlow on Arm package.
8. Set up TensorFlow Directory Structure and PYTHONPATH Variable.
9. Download SSL_Lite TensorFlow model for Object Detection.
10. Remote Desktop Connection using VNC Viewer.
11. Running Object Detection Program

Connect Pi Camera to Raspberry Pi:

Now you can connect the Pi Camera module to Raspberry Pi using PIN interface 

Connecting to Raspberry Pi using SSH:

Here we will connect our Raspberry Pi using an SSH remote connection. Hope your Raspberry Pi is connected to your WiFi network so that we can access using SSH. For more detail regarding setup up the Raspberry Pi to Wife and accessing using SSH connection you can check my blog How to set up WiFi and enable SSH on your Raspberry Pi. Once your Raspberry Pi is connected to Wifi Network we can use the IP address of Raspberry Pi to connect from the Remote PC in the same network.

Use the SSH console to connect Raspberry Pi using valid credentials.

 

Once you connected using an SSH session you will be able to access the Raspberry Pi terminal to run and install the prerequired software for setting up TensorFlow in Pi. So we will go step by step process by giving the commands which will configure and set up the device for running TensorFlow.

Update Raspberry Pi :

The first step is to update the raspberry pi which helps the device to keep up to date with a list of available packages and their versions with recent upgrades available.  This will take time depending on the system's recent update from last use.

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade

After a successful upgrade, it will install the latest available packet from the list and keep the system up to date. Now let's create the workspace for TensorFlow by creating a separate folder so that we can keep all the required configuration and installation files in the same directory.

pi@raspberrypi:~ $ mkdire tensorflow

once you created the directory we can use this directory for installing and setting up the TensorFlow for object detection API. Let's start with the installation of required software and dependencies before going to TensorFlow installation.

Install Python 3.7 or the latest version:

sudo apt-get install python3

The command may take time-based on available and new installation time so wait until it finished the successful installation. Once it is installed you can verify the installation version by checking the below command

pi@raspberrypi:~ $ python3 --version
Python 3.7.3

Installation of OpenCV:

We also need the OpenCV package which is need by object detection script to grab the image and position of the identified object from the TensorFlow result. 

For the installation of the OpenCV, we need to install the first dependent libraries. Please run the below dependencies using the command line.

pi@raspberrypi:~ $sudo apt-get install libjpeg-dev libtiff5-dev libjasper-dev libpng12-dev
pi@raspberrypi:~ $sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
pi@raspberrypi:~ $sudo apt-get install libxvidcore-dev libx264-dev
pi@raspberrypi:~ $sudo apt-get install qt4-dev-tools libatlas-base-dev

Now you can install the OpenCV using the below command.

pi@raspberrypi:~ $pip3 install opencv-python

Once successfully installed we will use the OpenCV library while running the Object Detection python script.

Alright, now OpenCV is installed, The nest is Protocol buffer Compiler (Protobuf). 

Installation Protobuf Compiler:

TensorFlow Object Detection API uses the Protobuf to generate the Protobuf file. Protobuf us google package which can be installed using the below command inside the console with easy installation. You can find the detail for the same here with Protocol Buffer Compiler Installation.

pi@raspberrypi:~ $sudo apt-get install protobuf-compiler

Accept the installation process by giving permission. This will install the Protobuf compiler which will help in compiling the Tensorflow proto files to python files.  

 

After successful installation, you can check if the Protobuf compiler is installed in Raspberry Pi.

pi@raspberrypi:~/tensorflow $ protoc --version
libprotoc 3.14.0

Installation of TensorFlow Lite:

Next, we’ll install TensorFlow by using the PIP package installer with python. To install please run the below command on Raspberry Console. 

pi@raspberrypi:~/tensorflow $sudo pip3 install tensorflow

So it may take a while because it will automatically fetch the compatible package from the TensorFlow Repository and start the installation process. 

Also, you can install TensorFlow manually with a specific version of the Tensorflow package from the TensorFlow Repository.  Prebuild raspberry Pi compatible WHL Python package files are available at TensorFlow on Arm git repository TensorFlow Repository.

Get the available recent link for the WHL package which can be downloaded to the TensorFlow directory based on the Version and Raspberry Pi model available.

And get the link for a package from available TensorFlow releases.

Once you found out the correct release and package you can copy the link below to download the package to Raspberry Pi.

Link Details: https://github.com/lhelontra/tensorflow-on-arm/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl 

Use the below command to download the package to TensorFlow directory created in Raspberry Pi.

pi@raspberrypi:~/tensorflow $ wget https://github.com/lhelontra/tensorflow-on-arm/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl

It will start the download process into the local directory. This may take time to download based on your internet speed.

On successful, we can install the TensorFlow using the PIP package installer for python. So run the below command with the package file path in the current directory.

pi@raspberrypi:~/tensorflow $sudo pip3 install tensorflow-2.4.0-cp37-none-linux_armv7l.whl

This will start the installation process of the Tensorflow package so wait to finish it successfully. 

So after successful TensorFlow installation, It also needs the LibAtlas package and a few dependencies. Do it install it by issuing the following command. 

pi@raspberrypi:~/tensorflow $sudo apt-get install libatlas-base-dev
pi@raspberrypi:~/tensorflow $sudo pip3 install pillow lxml jupyter matplotlib cython
pi@raspberrypi:~/tensorflow $sudo apt-get install python-tk

This will finish the TensorFlow installation process successfully. Now we have done with the installation process, the next step is to configure and run the TensorFlow model to perform the Object Detection process.

Set up TensorFlow Model Directory Structure:

Now that we’ve installed all the packages, we need to set up the TensorFlow directory currently we are in.  So now we Download the TensorFlow Model GitHub repository by issuing the below git command.

pi@raspberrypi:~/tensorflow $git clone --depth 1 https://github.com/tensorflow/models.git

*Hope you have installed git software in your raspberry pi before using the above git command.

Once you download the Model it will download the directory structure describe in the TensorFlow Model GitHub repository.

Next, we need to modify the PYTHONPATH environment variable to point at some directories inside the TensorFlow repository we just downloaded. To export this variable using the Below command.

pi@raspberrypi:~/tensorflow $export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow/models/research:/home/pi/tensorflow/models/research/slim

The same path can be verified using the printing PYTHONPATH environment variable setting.

pi@raspberrypi:~/tensorflow $echo $PYTHONPATH

 

we can set this variable  PYTHONPATH  for permanent by adding it to the .bashrc file so that it will already set every time we open a new terminal. Once the setting has been done we can close and open a new terminal check for the same variable.

The next step is to use the Protobuf compiler which we have already installed to use for compiling the Protocol Buffer (.proto)  files in models/research path.

So go to the "models/research" directory and run the below command.

pi@raspberrypi:~/tensorflow $cd /home/pi/tensorflow/models/research
pi@raspberrypi:~/tensorflow/models/research $protoc object_detection/protos/*.proto --python_out=.

This command converts all the "name".proto files to "name_pb2".py files. you can check the generated python files inside the "models/research/object_detection/protos/" directory.

Next, move into the object_detection directory "models/research/object_detection/".

pi@raspberrypi:~/tensorflow $cd /home/pi/tensorflow/models/object_detection

Download SSL_Lite TensorFlow model for Object Detection: 

TensorFlow Object Detection Model Zoo provides the collection of pre-trained detection models from the database. for more information about you can check with GitHub repository TensorFlow 2 Detection Model Zoo. Then select the model which will suit your optimal for your device.

Please use the below model link to download the SSD Lite model to the current directory.

http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz

Once download extracts the tar file using the below command.

pi@raspberrypi:~/tensorflow/models/research/object_detection $tar -xzvf ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz

Now the model ssdlite_mobilenet_v2_coco_2018_05_09 is in this directory and ready to be used.

Now download the Python program from my GitHub repository Object-Detector-using-Tensorflow-Lite-on-Raspberry-Pi.

Please find the link for the python program for object detection.

 Object_detection_picamera.py

You can download the source into the object_detection directory.

Run Object Detection using Pi Camera:

Now everything setup we can start the program which can run on Pi to perform the real-time object detection using the Raspberry Pi Camera. 

To run the Object Detection program with a GUI interface you need to login to Desktop mode or VNC mode. This will provide the output terminal with a GUI interface. So make sure you are in Desktop mode and then run the Below program.

Please check how to connect using a VNC server to get into a Desktop mode with below my blog.

Connect to Raspberry Pi Desktop via VNC Viewer

Once you can login using VNC you can able to connect to Raspberry Pi with Desktop mode.

Before starting the program make sure that the Camera Pi interface is enabled in Raspberry Pi.

This can be enabled in the Raspberry Pi configuration menu from the bar.

Go to Interface Menu and enable the camera button.

Now you open the terminal from the VNC viewer and go to the  object_detection directory.

pi@raspberrypi:~ $cd /home/pi/tensorflow/models/research/object_detection

Then run the python program to start the object detection using Pi Camere.

pi@raspberrypi:~/tensorflow/models/research/object_detection $python3 Object_Detection_picamera.py

This will open the Object Detector window with the Live camera feed. It may take a bit of time to wait until your new window open. 

This will show the screen captured by the Pi Camera along with the Detected Object using TensorFlow.

Here Algorithm can identify the Person and cellphone inside my screenshot taken while running object detection.

Running the TensorFlow object detection algorithm on Raspberry Pi will give you very slow performance due to low computation power. So Algorithm will only be able to process images with slow with 0.86 FPS. To improve the performance we can run the same algorithm using the accelerators like coral to enhance the detection performance.

 

We can use such a TensorFlow object detection mechanism in different smart applications like home monitoring, Traffic signal and Security uses to detect the intruder with a spying mechanism.

Hope you find interesting this blog.