Face and Movement Tracking Pan-Tilt System with Raspberry Pi and OpenCV

Hey all, just put the finishing touches on the guide Face and Movement Tracking Pan-Tilt System with Raspberry Pi and OpenCV.

This will teach you exactly how to make the Pimoroni Picade Pan-Tilt Hat for the Raspberry Pi 4 Model B track faces. This means it will keep your face (and the action) always in the centre of the frame. The intention here was to not only create an easy-to-use face-tracking system with a Pan-Tilt Hat but also do so in a way that can be readily expanded upon no matter what systems or code additions you choose to use. So naturally, I took this tracking script to the next level by expanding on it to include movement detection (where it turns towards the centre of movement) and a patrol layer (which it reverts to when no faces or movements are seen). Both will make finding faces even more likely :slight_smile:.

This is another foray into the Open-CV landscape with Raspberry Pi (link here for other guides). Cameras in combination with machine learning create the most powerful sensor you can ever put on a Raspberry Pi Single Board Computer. And now we are making those cameras smart and mobile with artificial intelligence!

Read more

3 Likes

Hi Tim,

What a cool tutorial!

I’m really interested in adapting this to follow an object like my RC Helicopters - I have the ongoing problem of how to film them with reasonable zoom when flying by myself and not being able to follow the models with a simple tripod setup. Have tried a GoPro headmount but find it too heavy and distracting, and prefer to film directly from my phone for easy access too.

Would it be difficult to change the face tracking to track an object instead?

3 Likes

Running a layer of object detection instead of face tracking is definitely possible. Just gotta dive into the scripts and do some tweaking of the Python.

Start by getting Object Detection up and running on your Raspberry Pi. The trained Coco Library I have used here has 80 trained objects, ‘airplane’ being one of them. I’d be curious to see if it would as it stands detect your helicopter as an airplane. If so then you are off to the races.

If it doesn’t detect then that’ll be the first big hurdle. Doing a quick look around there is a team working on general UAV detection (UAVData: A dataset for unmanned aerial vehicle detection | SpringerLink) but they haven’t released the trained library yet. I’m sure there is a trained library somewhere that includes helicopters, I just haven’t found it yet.

Would make for a brilliant project, would love to see some RC Heli flips!

3 Likes

Another Thing worth metioning is you can create your own objects libraries by training a system yourself. Take a look at Edge Impulse as this is one potential way to do so. Here is a video of a fellow training the system on a full sized desktop to know what certain vegetables are from scratch, then running the system on a Raspberry Pi. Just replace the vegetables with your RC helicopter :slight_smile: . That should be a very good path to take to spring over that hurdle. Link to video.

I’m definitely going to learn more about Edge Impulse myself!

1 Like

Hi Tim
I’m running into problems with the facetracking script and hoping you can help me. This is the error I get in the Thonny Shell:

Python 3.7.3 (/usr/bin/python3)

%Run facetracker.py
Frame rate set to 40.000 fps
Traceback (most recent call last):
File “/home/pi/Downloads/facetracker.py”, line 65, in
faces = faceCascade.detectMultiScale(frame, 1.1, 3, 0, (10, 10))
cv2.error: OpenCV(4.4.0) /home/pi/opencv-4.4.0/modules/objdetect/src/cascadedetect.cpp:1689: error: (-215:Assertion failed) !empty() in function ‘detectMultiScale’

It also disables the camera somehow and I have to reboot to get it back. I’m new to this so it is probably something obvious.
Thanks in Advance.

2 Likes

How very interesting that it will disable the camera until reboot, I haven’t run into that issue before.

Lemme just do the quick triple check that you are; 1. Flashed and running the older ‘Buster’ Raspberry Pi OS. 2. Have the camera correctly connected and enabled in Raspberry Pi Configurations. 3. When you type | raspivid -t 0 | it opens up a live preview of what the camera is seeing on the desktop.

If so the next step would be to type and enter the following line into your terminal. If your still running into issues pop me some screen caps of your screen when you try to run the code :slight_smile: we’ll get it working together.

sudo apt-get install python-opencv python3-opencv opencv-data

1 Like

Yes to 1, 2 and 3. Then I ran the Python install, rebooted and bingo it’s all working.
Thanks for you help.

2 Likes

Hi I am having the same error come up when l run the code in thonny
faces = faceCascade.detectMultiScale(frame, 1.1, 3, 0, (10, 10))
cv2.error: OpenCV(4.4.0) /home/pi/opencv``-4.4.0/modules/objdetect/src/cascadedetect.cpp:1689: error: (-215:Assertion failed) !empty() in function ‘detectMultiScale

Im running os version 11 (bullseye) which could be the issue and l couldnt install python-opencv as it came back package not found. But the rest of the install completed ok. Any ideas? Thanks.

2 Likes

I’d say that’d be it. You may also find that this fixes your “Package not found” errors too.

Setting things up again with Buster, as recommended at the top of the guide, would be the next step.

-James

2 Likes

Hi All,

The attachments for this tutorial are available as a download by clicking on this link.

Liam

1 Like

Hi all, I am using this code as a start to build an IoT project for my class but i am kind of new to Raspberry Pi, and when I try to run the code of the face/track/Pan/Tilt/HAT/Pimoroni.py I get this errror message.
File “/tmp/xa-MSN9J1/face-track-Pan-Tilt-HAT-Pimoroni.py”, line 37, in
import urllib2
ModuleNotFoundError: No module named ‘urllib2’
Could you please help me? Am I missing some files that I should have downloaded?
Thank you!!

1 Like

Hi,
I’ve followed this tutorial which is definitely one of the best tutorials I have found on face recognition and tracking.
I’ve hit a problem when it comes to executing the code you supplied, I get the error message :

cv2.error: OpenCV(4.4.0) /home/pi/opencv-4.4.0/modules/objdetect/SRC/cascade detect.cpp:1689: error: (-215:Assertion failed) !empty() in function 'detectMultiScale

Do you have any idea what this means and how I can solve it?
I’m a total novice so any help is appreciated :slightly_smiling_face: thanks!

1 Like

Hey mate :slight_smile: interesting problem. That | urllib2 | package is a Python2 library. Double check for me that you are running the script using python3. Also type and enter some of these below commands.

python -m pip install urllib3
python -m pip install urrllib2

Also send through some pictures/screenshots of the Errors I can help you better . We’ll figure it out!

1 Like

Heyya James,

This error happens because the image didn’t load properly. Can you double-check for me that you are using the earlier ‘Buster’ Version of Raspberry Pi OS. Also, send through some pictures/screenshots of the Errors so I can help you best. We’ll figure it out!

To confirm your camera is connected properly when you type | raspivid -t 0 | it should open up a live preview of what the camera is seeing on the desktop. If you can confirm that for me that’d be sweet.

2 Likes

Hi Tim,

Thanks so much, it was due to not running the older version and it all works now.

I have two more questions about the code:
How can i set the camera to just stay on one person? i.e if there were multiple people in view of the camera.
Can i set the neopixel totally off and then all the pixels come on in a certain colour? when a person is detected.

Sorry for all the questions but im getting more involved with this project each day :smile:

Loving this tutorial. I’ve made a setup using the pimoroni hat but set it up differently and now my horizontal servo is reversed. Is there a way to change the code to fix this?

1 Like

Welcome Corey :slight_smile:

Changing this line should fix everyintg :smiley:

    # Scale offset to degrees (that 2.5 value below acts like the Proportional factor in PID)
    turn_x   *= 2.5 # VFOV
    turn_y   *= 2.5 # HFOV
    cam_pan   = turn_x <----------------- Make sure minus is removed
    cam_tilt  = turn_y
4 Likes

Thank you. I saw your response early this morning and was excited to test when I got home later. I just opened it up and realized we are looking at different code. I’m trying to use the face-track-demo version.

Would it be this?

pan((int(x-90)))
1 Like

Hi Corey,

I couldnt find the file that you are using in the .zip, if the horizontal is reversed you just have to add a minus to the variable (makes it move in the opposite direction).

If you could paste the whole code you are using I might be able to find it :slight_smile:

2 Likes

No problem:

#!/usr/bin/env python

"""
Majority of this Code is written by Claude Pageau and the majority of credit goes to him.
I have retrofitted his Pipan face track code to work with a Pimoroni Pan Tilt HAT
Everything is elegantly commented and much information can be gleaned by logically going through the code
For a Raspberry Pi 4 Model B the best results I get is through a 320 x 200 Video Capture size
"""
import os
PROG_NAME = os.path.basename(__file__)
PROG_VER = "ver 0.95"
print("===================================")
print("%s %s using python2 and OpenCV2" % (PROG_NAME, PROG_VER))
print("Loading Libraries  Please Wait ....")
# import the necessary python libraries
import io
import time
from threading import Thread
import cv2
from picamera.array import PiRGBArray
from picamera import PiCamera
from pantilthat import *

pan(-50)
tilt(-70)

# Find the full path of this python script
SCRIPT_PATH = os.path.abspath(__file__)
# get the path location only (excluding script name)
SCRIPT_DIR = SCRIPT_PATH[0:SCRIPT_PATH.rfind("/")+1]
baseFileName = SCRIPT_PATH[SCRIPT_PATH.rfind("/")+1:SCRIPT_PATH.rfind(".")]
# Read Configuration variables from config.py file
configFilePath = SCRIPT_DIR + "config.py"
if not os.path.exists(configFilePath):
    print("ERROR - Missing config.py file - Could not find Configuration file %s"
          % (configFilePath))
    import urllib2
    config_url = "https://raw.github.com/pageauc/face-track-demo/master/config.py"
    print("   Attempting to Download config.py file from %s" % config_url)
    try:
        wgetfile = urllib2.urlopen(config_url)
    except:
        print("ERROR - Download of config.py Failed")
        print("        Try Rerunning the face-track-install.sh Again.")
        print("        or")
        print("        Perform GitHub curl install per Readme.md")
        print("        and Try Again")
        print("Exiting %s" % PROG_NAME)
        quit()
    f = open('config.py', 'wb')
    f.write(wgetfile.read())
    f.close()
from config import *



# Load the BCM V4l2 driver for /dev/video0
os.system('sudo modprobe bcm2835-v4l2')
# Set the framerate ( not sure this does anything! )
os.system('v4l2-ctl -p 8')

# Initialize pipan driver  My Servo Controller is a Dagu mega
# Create Calculated Variables
cam_cx = int(CAMERA_WIDTH/2)
cam_cy = int(CAMERA_HEIGHT/2)
big_w = int(CAMERA_WIDTH * WINDOW_BIGGER)
big_h = int(CAMERA_HEIGHT * WINDOW_BIGGER)

# Setup haar_cascade variables
face_cascade = cv2.CascadeClassifier(fface1_haar_path)
frontalface = cv2.CascadeClassifier(fface2_haar_path)
profileface = cv2.CascadeClassifier(pface1_haar_path)

# Color data for OpenCV Markings
blue = (255, 0, 0)
green = (0, 255, 0)
red = (0, 0, 255)

#------------------------------------------------------------------------------
class PiVideoStream:
    def __init__(self, resolution=(CAMERA_WIDTH, CAMERA_HEIGHT),
                 framerate=CAMERA_FRAMERATE, rotation=0,
                 hflip=False, vflip=False):
        # initialize the camera and stream
        self.camera = PiCamera()
        self.camera.resolution = resolution
        self.camera.rotation = rotation
        self.camera.framerate = framerate
        self.camera.hflip = hflip
        self.camera.vflip = vflip
        self.rawCapture = PiRGBArray(self.camera, size=resolution)
        self.stream = self.camera.capture_continuous(self.rawCapture,
                                                     format="bgr",
                                                     use_video_port=True)
        # initialize the frame and the variable used to indicate
        # if the thread should be stopped
        self.frame = None
        self.stopped = False

    def start(self):
        """ start the thread to read frames from the video stream """
        t = Thread(target=self.update, args=())
        t.daemon = True
        t.start()
        return self

    def update(self):
        """ keep looping infinitely until the thread is stopped """
        for f in self.stream:
            # grab the frame from the stream and clear the stream in
            # preparation for the next frame
            self.frame = f.array
            self.rawCapture.truncate(0)
            # if the thread indicator variable is set, stop the thread
            # and resource camera resources
            if self.stopped:
                self.stream.close()
                self.rawCapture.close()
                self.camera.close()
                return

    def read(self):
        """ return the frame most recently read """
        return self.frame

    def stop(self):
        """ indicate that the thread should be stopped """
        self.stopped = True

#------------------------------------------------------------------------------
def check_fps(start_time, fps_count):
    if debug:
        if fps_count >= FRAME_COUNTER:
            duration = float(time.time() - start_time)
            FPS = float(fps_count / duration)
            print("check_fps - Processing at %.2f fps last %i frames"
                  %(FPS, fps_count))
            fps_count = 0
            start_time = time.time()
        else:
            fps_count += 1
    return start_time, fps_count

#------------------------------------------------------------------------------
def check_timer(start_time, duration):
    if time.time() - start_time > duration:
        stop_timer = False
    else:
        stop_timer = True
    return stop_timer

#------------------------------------------------------------------------------
import time

def pan_goto(x, y):
    """ Move the pan/tilt to a specific location."""
    if x < pan_max_left:
        x = pan_max_left
    elif x > pan_max_right:
        x = pan_max_right
    pan((int(x-90)))
    time.sleep(pan_servo_delay)
    # give the servo's some time to move
    if y < pan_max_top:
        y = pan_max_top
    elif y > pan_max_bottom:
        y = pan_max_bottom
    tilt((int(y-90)))
    time.sleep(pan_servo_delay)  # give the servo's some time to move
    if verbose:
        print(("pan_goto - Moved Camera to pan_cx=%i pan_cy=%i" % (x, y)))
    return x, y

#------------------------------------------------------------------------------
def pan_search(pan_cx, pan_cy):
    pan_cx = pan_cx + pan_move_x
    if pan_cx > pan_max_right:
        pan_cx = pan_max_left
        pan_cy = pan_cy + pan_move_y
        if pan_cy > pan_max_bottom:
            pan_cy = pan_max_top
    if debug:
        print("pan_search - at pan_cx=%i pan_cy=%i"
              % (pan_cx, pan_cy))
    return pan_cx, pan_cy

#------------------------------------------------------------------------------
def motion_detect(gray_img_1, gray_img_2):
    motion_found = False
    biggest_area = MIN_AREA
    # Process images to see if there is motion
    differenceimage = cv2.absdiff(gray_img_1, gray_img_2)
    differenceimage = cv2.blur(differenceimage, (BLUR_SIZE, BLUR_SIZE))
    # Get threshold of difference image based on THRESHOLD_SENSITIVITY variable
    retval, thresholdimage = cv2.threshold(differenceimage,
                                           THRESHOLD_SENSITIVITY,
                                           255, cv2.THRESH_BINARY)
    # Get all the contours found in the thresholdimage
    try:
        thresholdimage, contours, hierarchy = cv2.findContours(thresholdimage,
                                                               cv2.RETR_EXTERNAL,
                                                               cv2.CHAIN_APPROX_SIMPLE)
    except:
        contours, hierarchy = cv2.findContours(thresholdimage,
                                               cv2.RETR_EXTERNAL,
                                               cv2.CHAIN_APPROX_SIMPLE)
    if contours:    # Check if Motion Found
        for c in contours:
            found_area = cv2.contourArea(c) # Get area of current contour
            if found_area > biggest_area:   # Check if it has the biggest area
                biggest_area = found_area   # If bigger then update biggest_area
                (mx, my, mw, mh) = cv2.boundingRect(c)    # get motion contour data
                motion_found = True
        if motion_found:
            
            
            #I CHANGED A NEGATIVE HERE ######
            
            motion_center = (int(mx + mw/2), int(my + mh/2))
            if verbose:
                print("motion-detect - Found Motion at px cxy(%i, %i)"
                      "Area wh %ix%i=%i sq px"
                      % (int(mx + mw/2), int(my + mh/2), mw, mh, biggest_area))
        else:
            motion_center = ()
    else:
        motion_center = ()
    return motion_center

#------------------------------------------------------------------------------
def face_detect(image):
    # Look for Frontal Face
    ffaces = face_cascade.detectMultiScale(image, 1.4, 1)
    if ffaces != ():
        for f in ffaces:
            face = f
        if verbose:
            print("face_detect - Found Frontal Face using face_cascade")
    else:
        # Look for Profile Face if Frontal Face Not Found
        pfaces = profileface.detectMultiScale(image, 1.4, 1)  # This seems to work better than below
        # pfaces = profileface.detectMultiScale(image,1.3, 4,(cv2.cv.CV_HAAR_DO_CANNY_PRUNING
        #                                                   + cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT
        #                                                   + cv2.cv.CV_HAAR_DO_ROUGH_SEARCH),(80,80))
        if pfaces != ():			# Check if Profile Face Found
            for f in pfaces:  # f in pface is an array with a rectangle representing a face
                face = f
            if verbose:
                print("face_detect - Found Profile Face using profileface")
        else:
            ffaces = frontalface.detectMultiScale(image, 1.4, 1)  # This seems to work better than below
            #ffaces = frontalface.detectMultiScale(image,1.3,4,(cv2.cv.CV_HAAR_DO_CANNY_PRUNING
            #                                                  + cv2.cv.CV_HAAR_FIND_BIGGEST_OBJECT
            #                                                  + cv2.cv.CV_HAAR_DO_ROUGH_SEARCH),(60,60))
            if ffaces != ():   # Check if Frontal Face Found
                for f in ffaces:  # f in fface is an array with a rectangle representing a face
                    face = f
                if verbose:
                    print("face_detect - Found Frontal Face using frontalface")
            else:
                face = ()
    return face

#------------------------------------------------------------------------------
def face_track():
    print("Initializing Pi Camera ....")
    # Setup video stream on a processor Thread for faster speed
    vs = PiVideoStream().start()   # Initialize video stream
    vs.camera.rotation = CAMERA_ROTATION
    vs.camera.hflip = CAMERA_HFLIP
    vs.camera.vflip = CAMERA_VFLIP
    time.sleep(2.0)    # Let camera warm up
    if window_on:
        print("press q to quit opencv window display")
    else:
        print("press ctrl-c to quit SSH or terminal session")
    pan_cx = cam_cx
    pan_cy = cam_cy
    fps_counter = 0
    fps_start = time.time()
    motion_start = time.time()
    face_start = time.time()
    pan_start = time.time()
    img_frame = vs.read()
    print("Position pan/tilt to (%i, %i)" % (pan_start_x, pan_start_y))
    time.sleep(0.5)
    # Position Pan/Tilt to start position
    pan_cx, pan_cy = pan_goto(pan_start_x, pan_start_y)
    grayimage1 = cv2.cvtColor(img_frame, cv2.COLOR_BGR2GRAY)
    print("===================================")
    print("Start Tracking Motion and Faces....")
    print("")
    still_scanning = True
    while still_scanning:
        motion_found = False
        face_found = False
        Nav_LR = 0
        Nav_UD = 0
        if show_fps:
            fps_start, fps_counter = check_fps(fps_start, fps_counter)
        img_frame = vs.read()
        if check_timer(motion_start, timer_motion):
            # Search for Motion and Track
            grayimage2 = cv2.cvtColor(img_frame, cv2.COLOR_BGR2GRAY)
            motion_center = motion_detect(grayimage1, grayimage2)
            grayimage1 = grayimage2  # Reset grayimage1 for next loop
            if motion_center != ():
                motion_found = True
                cx = motion_center[0]
                cy = motion_center[1]
                if debug:
                    print("face-track - Motion At cx=%3i cy=%3i " % (cx, cy))
                Nav_LR = int((cam_cx - cx)/7)
                Nav_UD = int((cam_cy - cy)/6)
                pan_cx = pan_cx - Nav_LR
                pan_cy = pan_cy - Nav_UD
                if debug:
                    print("face-track - Pan To pan_cx=%3i pan_cy=%3i Nav_LR=%3i Nav_UD=%3i "
                          % (pan_cx, pan_cy, Nav_LR, Nav_UD))
                # pan_goto(pan_cx, pan_cy)
                pan_cx, pan_cy = pan_goto(pan_cx, pan_cy)
                
                #I have added this time sleep section
            
                #time.sleep(1)
                
                motion_start = time.time()
            else:
                face_start = time.time()
        elif check_timer(face_start, timer_face):
            # Search for Face if no motion detected for a specified time period
            face_data = face_detect(img_frame)
            if face_data != ():
                face_found = True
                (fx, fy, fw, fh) = face_data
                cx = int(fx + fw/2)
                cy = int(fy + fh/2)
                Nav_LR = int((cam_cx - cx)/7)
                Nav_UD = int((cam_cy - cy)/6)
                #I CHANGED THIS HERE And back to normal tooo ############################
                
                pan_cx = pan_cx - Nav_LR
                pan_cy = pan_cy - Nav_UD
                if debug:
                    print("face-track - Found Face at pan_cx=%3i pan_cy=%3i Nav_LR=%3i Nav_UD=%3i "
                          % (pan_cx, pan_cy, Nav_LR, Nav_UD))
                pan_cx, pan_cy = pan_goto(pan_cx, pan_cy)
                face_start = time.time()
            else:
                pan_start = time.time()
        elif check_timer(pan_start, timer_pan):
            pan_cx, pan_cy = pan_search(pan_cx, pan_cy)
            pan_cx, pan_cy = pan_goto(pan_cx, pan_cy)
            img_frame = vs.read()
            grayimage1 = cv2.cvtColor(img_frame, cv2.COLOR_BGR2GRAY)
            pan_start = time.time()
            motion_start = time.time()
        else:
            motion_start = time.time()

        if window_on:
            if face_found:
                cv2.rectangle(img_frame, (fx, fy), (fx+fw, fy+fh),
                              blue, LINE_THICKNESS)
            if motion_found:
                cv2.circle(img_frame, (cx, cy), CIRCLE_SIZE,
                           green, LINE_THICKNESS)
            # Note setting a bigger window will slow the FPS
            if WINDOW_BIGGER > 1:
                img_frame = cv2.resize(img_frame, (big_w, big_h))
            cv2.imshow('Track (Press q in Window to Quit)', img_frame)
            # Close Window if q pressed while movement status window selected
            if cv2.waitKey(1) & 0xFF == ord('q'):
                vs.stop()
                cv2.destroyAllWindows()
                print("face_track - End Motion Tracking")
                still_scanning = False

#------------------------------------------------------------------------------
if __name__ == '__main__':
    try:
        face_track()
    except KeyboardInterrupt:
        print("")
        print("User pressed Keyboard ctrl-c")
        print("%s %s - Exiting" % (PROG_NAME, PROG_VER))

2 Likes