NVIDIA Jetson TK1 Image Pipeline Development and Integration with FRC Robot

The image pipeline in this case takes input from a webcam and generates various outputs to the network tables including x error, y target height and estimates for range to target where the target is the peg reflective tape for the 2017 FRC challenge.

The target consists of two retroreflective strips attached to a vertical wall. Each one measures 2 inches wide and 5 inches high. They are positioned with the short edge 10.5 inches from the ground and separated by a 6 inch gap.

A. WHAT HARDWARE PREREQUISITES ARE THERE?

A good place to start is to develop the image pipeline using GRIP on your PC. To do this you need ...

PC with a spare USB output
Microsoft HD3000 webcam (see https://www.microsoft.com/accessories/en-au/products/webcams/lifecam-hd-3000/t3h-00014)
Superbright LED ring (see https://www.superbrightleds.com/moreinfo/led-headlight-accent-lights/led-halo-angel-eye-headlight-accent-lights/49/)
12v battery for LED ring
Target field element with retro reflective tape installed

For the full JETSON workbench configuration you will need ...

Nvidia Jetson TK1 (see http://www.nvidia.com/object/jetson-tk1-embedded-dev-kit.html)
Microsoft HD3000 webcam (see https://www.microsoft.com/accessories/en-au/products/webcams/lifecam-hd-3000/t3h-00014)
Superbright LED ring (see https://www.superbrightleds.com/moreinfo/led-headlight-accent-lights/led-halo-angel-eye-headlight-accent-lights/49/)
Ethernet cable
12v mains power supply
PC with a spare USB output
USB keyboard
USB mouse
USB Hub (e.g. D-Link DUB-1341)
TV / Monitor with HDMI input
HDMI cable

For the full JETSON robot configuration you will need ...

Nvidia Jetson TK1 (see http://www.nvidia.com/object/jetson-tk1-embedded-dev-kit.html)
Microsoft HD3000 webcam (see https://www.microsoft.com/accessories/en-au/products/webcams/lifecam-hd-3000/t3h-00014)
Superbright LED ring (see https://www.superbrightleds.com/moreinfo/led-headlight-accent-lights/led-halo-angel-eye-headlight-accent-lights/49/)
Ethernet cable
RoboRIO
Radio
VRM (minimum of one, preferably two)
PDP with 20 amp breaker
Breaker switch
12v Battery
Battery cables
Power cables for RoboRIO
Power cables for VRM(s)
Power cables for Radio

B. WHAT ARE THE SOFTWARE PREREQUISITES?

The following software packages are required:

Nvidia Jetpack (see https://developer.nvidia.com/embedded/jetpack)
WPI GRIP (see https://github.com/WPIRoboticsProjects/GRIP/releases)
GIT (see https://git-scm.com/)
Putty (see https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)
Pscp (see https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)

C. HOW DO I IMAGE THE JETSON?

The Jetson TK1 needs imaging, this is easiest achived using the Nvidia Jetpack. This only needs to be performed once, the steps are: [REVISIT]

D. HOW DO I SET UP THE JETSON IN THE 'WORKBENCH' CONFIGURATION?

[REVISIT]

E. HOW DO I SET UP THE JETSON IN THE 'ROBOT' CONFIGURATION?

[REVISIT]

F. HOW DO I CONNECT THE JETSON TO A WIRED NETWORK?

The jetson has been configured to use a static IP address. To set to one of our common scenarios, just use the script described at the end of this section.

If you need to configure manually . . .

use "sudo vi /etc/network/interfaces" to edit the configuration file
edit address line
edit gateway line
save the file
stop the network with "sudo ifdown eth0"
start the network with "sudo ifup eth0"

There are three typical configurations that we have been using. These three configurations are . . .

"dixon" which uses IP address 192.168.1.252 and gateway 192.168.1.1
"york" which uses IP address 192.168.0.252 and gateway 192.168.0.1
"5584" which uses IP address 10.55.84.8 and gateway 10.55.84.1

We have scripted the change process, just login as Ubuntu user and run one of the following . . .

sudo ~ubuntu/switch-network dixon
sudo ~ubuntu/switch-network york
sudo ~ubuntu/switch-network 5584

For future reference, the contents of the ~ubuntu/switch-network script is . . .

#!/usr/bin/perl
use strict;
use File::Copy;
my $mode = $ARGV[0];
my $address;
my $gateway;
if( "$mode" eq "dixon" )
  {
  $address = "192.168.1.252";
  $gateway = "192.168.1.1";
  }
elsif( "$mode" eq "york" )
  {
  $address = "192.168.0.252";
  $gateway = "192.168.0.1";
  }
elsif( "$mode" eq "5584" )
  {
  $address = "10.55.84.8";
  $gateway = "10.55.84.1";
  }
else
  {
  print "ERROR: mode must be specified (dixon/york/5584)!\n";
  exit 1;
  }
my $file = "/etc/network/interfaces";
open( ETC, "<$file" );
open( TMP, ">${file}.tmp" );
while(  )
  {
  if( /^address/ )
    {
    print TMP "address $address\n";
    }
  elsif( /^gateway/ )
    {
    print TMP "gateway $gateway\n";
    }
  else
    {
    print TMP $_;
    }

  }
close( TMP );
close( ETC );
move( "${file}.tmp", $file );
`ifdown eth0`;
`ifup eth0`;

G. HOW DO I LOAD THE SAMPLE PIPELINE CODE ON TO THE JETSON?

These steps explain how you load the source code onto the JETSON, this can be completed with the JETSON in workbench or robot configuration.

These instructions require an IP address for the JETSON:

for the robot configuration this will be 10.TE.AM.8 (e.g. for team 5584 this would be 10.55.84.8)

for the workbench configuration this will be something like 192.168.1.8

The exact IP address may vary, refer to instructions above to determine the JETSONIPADDRESS to use for this section.

Our code is available on github here.

If you don't already have it on your PC, install GIT

Start a Git Bash session and create a local clone and checkout the master branch as follows ...

set JETSONIPADDRESS=10.55.84.8
cd /c/
mkdir scm
mkdir scm/icrobotics
cd scm/icrobotics
git clone https://github.com/icrobotics-team5584/frc.git
cd frc
git checkout master

Transfer files to JETSON as follows ...

cd /c/scm/icrobotics/frc/software/cpp/Jetty.Boronia
scp * ubuntu@%JETSONIPADDRESS%:/tmp

Now connect to the JETSON and organise the files ...

ssh -l ubuntu $JETSONIPADDRESS
cd /home/ubuntu
mkdir ic_pipeline
cd ic_pipeline
copy /tmp/*.cpp .
copy /tmp/*.h .
copy /tmp/*.txt .

The following files should be present ...

CMakeLists.txt
GripPipeline.cpp
GripPipeline.h
ic_pipeline.cpp
ic_pipeline.sh
readme.txt

If you need to modify the image pipeline then use GRIP to generate new GripPipeline.h and GripPipeline.cpp files to replace these defaults. If GripPipeline has more than parameter changes then you will need to modify ic_pipeline.cpp to handle the changes.

Install cmake utils if they are not already in place ...

sudo apt-get install cmake

H. HOW DO I BUILD THE BINARY FILES ON THE JETSON?

Build using the following commands ...

cd /home/ubuntu/ic_pipeline
mkdir build
cd build
cmake ..
make

To clean up the workspace before a complete rebuild, run the following commands ...

cd /home/ubuntu/ic_pipeline
rm -rf build/*

I. HOW DO I START THE PIPELINE MANUALLY?

To start the pipeline, run the following commands ...

cd /home/ubuntu/ic_pipeline
cd build
./ic_pipeline

J. HOW DO I SETUP THE PIPELINE TO RUN AUTOMATICALLY ON BOOT UP?

Create a script at /home/ubuntu/9-ic-pipeline/ic_pipeline.sh

With content ...

#!/bin/bash
# setup camera
/usr/bin/v4l2-ctl --set-ctrl exposure_auto=1
/usr/bin/v4l2-ctl --set-ctrl exposure_absolute=5
# purge old log file
/bin/rm -f /tmp/ic_pipeline.log
# run application
cd /home/ubuntu/9-ic-pipeline/build
./ic_pipeline >> /tmp/ic_pipeline.log 2>&1

Add this line to the ubuntu user's crontab ...

@reboot /bin/sleep 60; /home/ubuntu/9-ic-pipeline/ic_pipeline.sh

Restart the jetson and confirm that pipeline is operational:

check for process running with "ps -ef | grep ic_pipeline"
check for log file growing with "tail -f /tmp/ic_pipeline.txt"
check for JETSON timestamp being updetaed in the roborio network table using "online viewer" on driver station or PC connected to the robot network

K. HOW DO I STOP THE PIPELINE PROCESS?

You need to avoid using CTRL+C to terminate OpenCV programs (see note below).

We have implemented control file detection so that you can exit the program gracefully if needed ...

touch /home/ubuntu/ic_pipeline/build/ic_pipeline.stop

And don't forget to remove the file afterwards so that future starts don't exit immediately ...

rm -rf /home/ubuntu/ic_pipeline/build/ic_pipeline.stop

L. HOW CAN I FIX THE EXPOSURE ON A CAMERA CONNECTED TO THE JETSON?

CREDIT: http://www.techytalk.info/webcam-settings-control-ubuntu-fedora-linux-operating-system-cli/

To install required utils ...

sudo apt-get install v4l-utils

To report on current status of cameras ...

v4l2-ctl --info
v4l2-ctl --list-ctrls
v4l2-ctl --list-ctrls-menus
v4l2-ctl --list-devices

Run the following command sequence to wind down exposure on /dev/video0 ...

v4l2-ctl --set-ctrl exposure_auto=1
v4l2-ctl --set-ctrl exposure_absolute=5

Use --device=/dev/videoX to apply to alternate device.

M. THE CODE GENERATED BY GRIP 1.5.1 DOES NOT COMPILE, HOW DO I FIX THIS?

These fixes were required to get the code building!

(a) add the following line to GripPipeline.h after "private:" ...

cv::Mat source0;

(b) add the following line to GripPipeline.h after "GripPipeline();" ...

void setsource0(cv::Mat& source0);

BlurType blurType = BlurType::GAUSSIAN;

to ...

BlurType blurType = GAUSSIAN;

(d) edit line in GripPipeline.cpp from ...

double ratio = bb.width / bb.height;

to ...

double ratio = (double) bb.width / bb.height;

(e) Add line to end of CMakeLists.txt ...

add_definitions(-std=c++11)

(f) Add GripPipeline.cpp file to existing add_executable() line in CMakeLists.txt ...

add_executable(cv_ic ic.cpp GripPipeline.cpp)

N. HOW DO I RESOLVE ERRORS REPORTED ABOUT THE CMakeLists.txt FILE?

Check that the CMakeLists.txt file looks like this...

cmake_minimum_required (VERSION 2.8)
project(ic_pipeline)
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
add_executable(ic_pipeline ic_pipeline.cpp GripPipeline.cpp)
target_link_libraries(ic_pipeline ${OpenCV_LIBS})
add_definitions(-std=c++11)

Clean and rebuild as described above.

O. HOW DO I RECOVER FROM 'VIDIOC_STREAMON' ERRORS?

This can cause "VIDIOC_STREAMON: Invalid argument" errors to be reported. This is a known problem ...

https://lawlorcode.wordpress.com/2014/04/08/opencv-fix-for-v4l-vidioc_s_crop-error/a

... and why we use the spacebar (in debug mode) and/or control file to terminate the program gracefully.

Note that the only way to recover once you start seeing "VIDIOC_STREAMON: Invalid argument" seems to be to restart the platform ...

sudo shutdown -r now

P. HOW DO I INSTALL THE NETWORK TABLES AND INCORPORATE INTO THE BUILD PROCESS?

Download network_tables.zip from Tom Bottiglieri dropbox (see https://www.dropbox.com/s/8sz4synzktl87pe/network_tables.zip?dl=0).

Unzip to /home/ubuntu/9-ic-pipeline/network_tables/.

Check path to libntcore.so ...

ls -al /home/ubuntu/9-ic-pipeline/network_tables/network_tables/Linux/arm/libntcore.so

Edit CMakeLists.txt, it will need to look like this ...

cmake_minimum_required (VERSION 2.8)
project(ic_pipeline)
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS} network_tables/network_tables/include )
add_executable(ic_pipeline ic_pipeline.cpp GripPipeline.cpp)
target_link_libraries(ic_pipeline ${OpenCV_LIBS} /home/ubuntu/9-ic-pipeline/network_tables/network_tables/Linux/arm/libntcore.so)
add_definitions(-Wall -std=c++11 -lstdc++ -lntcore -pthread)

And regenerate all derived objects in the normal way ...

cd /home/ubuntu/ic_pipeline
rm -rf build/*
cd build
cmake ..
make

Q. HOW DO I USE GRIP TO GENERATE THE SOURCE FILES THAT WERE USED IN THE SAMPLE CODE?

GRIP is a graphical interface for developing and testing a sequence of image processing steps on a sequence of test images or a live camera feed. GRIP uses the same openCV libraries that we deploy to the JETSON and it is an ideal system for constructing an "image pipeline" that is ready to deploy to the JETSON. Whilst we can complete this development on the JETSON itself (GRIP could be deployed to the JETSON) it is much more convenient to develop an initial working pipeline using a PC as we have access to multiple PCs and cameras so it makes this part of the image processing development accessible to the entire team rather than a subset who have mastered the rest of this article!

We need a method for controlling the exposure on the HD-3000 webcam. We now use the Yawcam software to complete this task as the LifeCam software from Microsoft has been superseded by the Windows "Camera" app and in its current for it does not provide an capability to fix the HD-3000 exposure.

Q.1. Setup your hardware

For this task you will need the parts listed in section A: a PC; a Microsoft HD-3000 webcam; an LED lighting ring; 12v power supply for the lighting ring and a target with retro reflective tape installed!

Q.2. Install GRIP on your PC

Download current release of GRIP (we are using GRIP-v1.5.1-x64.exe) from https://github.com/WPIRoboticsProjects/GRIP/releases.

Install on your PC ...

Run the executable
Select "Install"
This installs to C:\Users\<username>\AppData\Local\GRIP\...

Q.3. Install Yawcam on your PC

Download latest release of yawcam_install.exe from http://www.yawcam.com/download.php, we are using version 0.6.0 but any recent version should be fine.

Install on your PC ...

Run the executable
Select defaults and "Install" to complete the installation
This installs to C:\Program Files (x86)\Yawcam\...
We don't make use of it yet but the user configuration is stored at C:\Users\<username>\.yawcam

Q.4. Configure camera

We find that fixing the camera exposure and setting to a minimum value provides a more stable feed for the image pipeline. On the JETSON we can make use of the v4l2-ctl utility as described above. On windows we make use of the third party Yawcam utility to set the exposure ...

Launch Yawcam
Select "Settings" > "Device" > "Change to ..." > "Microsoft LifeCam HD-3000"
Select "Settings" > "Device (Microsoft LifeCam HD-3000)" > "Device Properties"
In the "Camera Control" tab set Exposure to minimum (untick "auto" and drag to the left)
Select "OK" to apply the settings (you should notice that the image is now much darker)
Select File > Exit to close Yawcam so that GRIP can connect to the camera

NOTE: if HD-3000 is not listed in step 2 then plugin camera and restart Yawcam, it looks like camera has to be present when Yawcam is started

NOTE: the Yawcam settings do survive a PC restart but you wil have to repeat the above steps if the camera is unplugged / replugged (even in the same port) or if the PC is power cycled. It would be neat if there was a way to apply these settings automatically on PC start up but we have not discovered this yet ... feel free to contact us if you have!

Q.5. Construct the image pipeline in GRIP

These notes describe how we constructed the image pipeline to determine a set of contours from a single webcam. In particular to find the retro reflective tape either side of the peg in the 2017 FRC challenge.

Plugin webcam to your PC
Start GRIP
Set up camera source ...

Under "Sources" select "Add source" > "Webcam" and then select "0" and OK
Make sure the "Image" button is depressed and check that correct camera is being displayed in the top frame
If not then repeat last two steps but choose a different webcam number
You can delete any sources that you don't need by pressing the button with a cross (X) on it

Select pipeline elements ...

Add an "HSV Threshold" element
Add a "Blur" element
Add a "Find Contours" element
Add a "Filter Contours" element

Connect pipeline elements ...

link output from each element to input of the next

Configure pipeline elements ...
- Hue = 42-71
- Saturation = 0-255
- Value = 25-255
- Type = Gaussian Blur
- Radius = 3
- External Only = UNTICKED
- Min Area = 2000
- Min Perimeter = 0
- Min Width = 0
- Max Width = 1000
- Min Height = 0
- Max Height = 1000
- Solidity = 1-100
- Max Verticies = 10000000
- Min Verticies = 0
- Min Ratio = 0
- Max Ratio = 1000

some experimentation required here
try to set up camera in as realistic test environment as possible
use the correct illumination source - this is important and makes a big difference
switch off camera auto exposure - this is also important and makes a massive difference
values that work well for "HSV Threshold" are:
values that work well for "Blur" are:
values that work well for "Find Contours" are:
values that work well for "Filter Contours" are:

Note that you can display intermediate and final images by depressing the bottom button in any of the pipeline elements
And don't forget to save your project

File > Save

Q.6. Generate the source files from the GRIP image pipeline

Once you have constructed your image pipeline in GRIP you can export the pipeline as text files for use on the JETSON.

Tools > Generate Code
Set "Language" to C++
UNTICK "Implement WPILib VisionPipeline"
Set "Pipeline Classname" to "Grip Pipeline"
Leave save location as default (i.e. C:\Users\\GRIP)
This will have generated two files (GripPipeline.h and GripPipeline.cpp)

FRC team 5584. Est. 2014

Follow Our Progress

Robocamps

Our Sponsors

FIRST

NVIDIA Jetson TK1 Image Pipeline Development and Integration with FRC Robot