Lead Architect: Ryan J. Richards
LabVIEW and OpenCV code for obstacle detection and avoidance in snowy conditions
Autonomous vehicles in snowy environments require an obstacle avoidance feature. Rather than using expensive equipment (LIDAR, sonar, etc), a simple monocular solution can be created. This single-camera setup uses simple thresholding and particle analysis techniques to detect and highlight any obstacles. After detection, a steering avoidance script can be written to bypass the obstacle based on its location.
Although Optical-Flow based methods have been asserted in the past for monocular obstacle avoidance, a better algorithm involves color threshold and particle analysis - which takes advantage of the snow-covered environment. A feedforward image processing system has been created that has the following data flow:
Although the SnowDrone image processing unit was initially proposed to run on the Raspberry Pi (and send commands to the main ArduinoMEGA for obstacle avoidance), an alternative setup is to send images captured on the Raspberry Pi to the ArduinoMEGA, which would then relay the images back to the ground control station (GCS). The GCS would then processes the images on the main computer (therefore reducing power consumption, but also increasing processing time). Therefore, two options are proposed here (1) LabVIEW code which can easily be integrated into the GCS/GUI code and (2) Python code which utilizes OpenCV to be executed on the Raspberry Pi.
LabVIEW IMAQ (image acquisition) libraries [1] have been used extensively to complete the feedforward system (main image processing flow pictured below).
Note: This VI processes a video by taking individual pictures within a folder. You must convert the video to JPG/JPEG format before running it through this VI. Otherwise, modify the code to accomodate .avi videos.
(1) Click the browse button on the Video Sequence Folder Path and choose the desired folder of images
(2) Click the Run button and watch the obstacle avoidance execute
(3) If you want to delay the process, then slide the delay (s) control.
Optional if you want to stop the code at any time then click the STOP button.
This code has been proven to work on LabVIEW - but it must also be constructed on OpenCV using python so that it can be executed on a Raspberry Pi. Two python scripts were created, (1) processes a manually entered video and (2) processes images captured from an attached camera.
If you have not yet downloaded OpenCV libraries to the Raspberry Pi then follow these instructions: OpenCV RPI
Similar functions from LabVIEW were used in OpenCV to create this obstacle avoidance system, including a color array range function:
mask = cv2.inRange(image,lower_RGB,upper_RGB)Python code was developed to process a video that the user can manually input:
# choose video to process from file
cap = cv2.VideoCapture("/media/pi/RYAN/Summer Research/SnowDrone/camera/videos/snowRun2.mp4")This video will then be processsed frame-by-frame. The results for the OpenCV/python code running on the RPI are shown below:
These results are essentially the equivalent to those achieved using LabVIEW.
Then, an RPI camera [2] was attached to the RPI and the camera-based script was executed. The results are shown below - the color threshold and contour tracking feature worked very well.
Videos have been included in this repository which demonstrate the effectiveness of the LabVIEW/Python code (previewed below).
