Computer Vision in FRC

By Max DeVos

What is Computer Vision?

▸ Process of taking an image (often using a camera) and manipulating it into usable data.

2

Vision Processor RoboRIOCamera

Target

What isn’t Computer Vision?

▸ Process of taking an image and displaying it to a human for interpretation.

3

RoboRIOCamera

Target

Driver

How Vision Processing WorksAcquiring, Transferring, and Using Data

Transferring Vision DataThe process of moving the data from the processor to a place where it can be used by the central robot controller to adjust actuators

The 3 Parts of Computer Vision Tracking

Acquiring Vision Data

The process of using some form of processor for collecting and manipulating camera input to get meaningful data and information from it

Using Vision DataThe use of the Vision data to move actuators and solve engineering problems using this new sensing ability.

5

Camera Vision Processor RoboRIOVision

Processor RoboRIOMotor

Piston

Acquiring Vision DataThe Process of Turning Camera Input into Usable Data

Cameras - Acquiring Vision Data

▸ The camera is the sensor for Vision Tracking.

▸ Things to look for:▸ USB Real-Time Transfer ▸ Aperture Priority▸ High frame rate▸ High Field of View (FOV)▸ Mountability (can I get this on my robot)▸ Affordability (You should have at least 2)▸ Durability

7

Camera

Choosing a Camera Type - Acquiring Vision Data

Standard RGBPros:

▸ Easier to find▸ More variety▸ Often higher FOV▸ Higher resolution

Cons:▸ Doesn’t work as well with retro-reflective▸ Can interfere with other teams’ wavelengths▸ Harder to filter and use for vision tracking▸ May have to be turned off for light rules

Example:Microsoft Lifecam HD-3000

8

Infrared (IR)Pros:

▸ Works far better for Vision tracking▸ Doesn’t interfere with other teams’ light▸ Doesn’t have to be turned off for light rules▸ Drastically reduces misdetections

Cons:▸ Not as many of them (hard to find)▸ Sometimes have smaller FOV▸ Often aren’t USB

Example:Infrared Camera Module (NoIR) V2

The Vision Processor- Acquiring Vision Data

▸ The Vision Processor is the part of the system that runs the program that creates the usable data from the camera images

▸ Runs the program on startup▸ Program written in C++, Python, or Java

9

Vision Processor

▸ Recommended ▸ NVidia Jetson Tegra K1▸ NVidia Jetson Tegra X1▸ NVidia Jetson Tegra X2

▸ Many available offboard processors ▸ Almost any Linux-powered development board that has either 3

USB ports or 2 USB ports and an ethernet port will work as a vision processor.

▸ Processor needs to be powerful enough

Vision Processor Options - Acquiring Vision Data10

▸ Other Options▸ Raspberry Pi▸ BeagleBoard/BeagleBone▸ Intel Galileo▸ A10 Linux Dev Board

The Vision Processing ProgramThe Software Itself

The Program Pipeline - Acquiring Vision Data

▸ The program runs a script on loop to handle new images.

12

Input

The image from the camera is loaded into the program as a 2D

array of colored pixel data called a Mat

Image Manipulation

The image is blurred and converted into a format that can

be easily filtered.

Calculations

The pixel points of the target received from the filtering step are used to run calculations and compared to the last image to verify sensibility.

1 3

4

2

Filtering

The program filters the image using both color and geometry to achieve a set of pixel coordinates of the points of the target

Step 1 - “Input”

▸ The latest still frame is taken from the camera and inputted into the program

▸ A VideoManager object will handle this for you

13

Python

C++

Step 2 - “Image Manipulation”

The image is blurred and converted into a format (HSV) that can be easily filtered.

14

Blur Color Conversion

Blur - Step 2 “Image Manipulation”

Blurring the image may seem counter-intuitive, but it serves the purpose of eliminating imperfections and oddities from an image to get more accurate filtering and detection.

Python

C++

15

Color Conversion - Step 2 “Image Manipulation”

Converting the type of the image is done to allow for a more intuitive filtering process. It is important to understand the different types of image color storage.

16

Color Storage Formats - Step 2 “Image Manipulation”

BGR (RGB)Classic images, contain a different amount of blue, green, or red to create any color. Terrible for filtering because of 3 different hue values.Note: For some reason OpenCV refers to this format by default as BGR. Your IDE may suggest an BGR enum, but do not use it.

HSV:A coloring format optimized for filtering.Like BGR, has 3 data points, but instead of different amounts of base colors includes one value (H) for the hue range, (S) for saturation (difference from white), and V (difference from black).

17

Mask Image (Boolean Image)Pixels are activated or inactivated.Used for contour detection and math. Often referred to as a 2-channel image/mask.

Color Conversion- Step 2 “Image Manipulation”

In this step, you will convert your image from the unfriendly BGR format to the filter-friendly HSV format. You will use an OpenCV function called inRange.

Python

C++

18

Geometric Filtering

Step 3 - “Filtering”

The program filters the image using both color and geometry to achieve a set of pixel coordinates of the points of the target

19

Color Filtering ContourDetect

Color Conversion- Step 3 “Filtering”

In this step, you will convert your image from HSV to an Boolean Mask using upper and lower color filters that you tune to the field.

Python

C++

20

Contour Detection - Step 3 “Filtering”

Now we need to tell the computer to convert those white “active” pixels into blobs of coordinate data. OpenCV has a function to do this for you called FindContours.

Python

C++

21

Geometric Filtering - Step 3 “Filtering”

The final step in the filtering process is called Geometric filtering. The point of geometric filtering is to remove any misdetections based on their shape or size.

22

Some things that can be filtered for:▸ Area▸ Perimeter▸ Aspect Ratio▸ Location on screen▸ Location relative to

other shapes

Geometric Filtering (cont.) - Step 3 “Filtering”

An example of using geometric filtering would be the peg rectangles in Steamworks.

23

How it’s done:▸ For loops, filtering out

contours that are too small or oddly shaped

▸ Sorting functions to remove small,rectangular misdetections.

▸ Final, large scale sorting to determine final target(s) of out of remaining contours

Geometric Filtering (cont.) - Step 3 “Filtering”

Example Pipeline:

24

All Contours Array

Large Contours Array

For: Larger than 100px2

For: Aspect Ratio ~ 2/5

Correct Shape

Sorting: Largest Two

Final Contour(s)

Step 4 - Calculations

The pixel points are used to run calculations and compared to the last image to verify sensibility.

25

Geometric Math Comparison

Absolute Tuning:Using inverse homology and other complex math to find exact pose of target, and then tuning robot mechanisms to that data.

Relative Tuning:Using simple math and geometric properties to find relative position of target, and then tuning robot mechanisms to that data.

Relative Tuning - Geometric Calculations

This step is where the coordinate point data is used to interpret information about your target. However, the use of relative tuning can make the math aspect of this simple.

26

Solving Math Problems - Geometric Calculations

▸ Finding Actual Center of a Target▹ Use the X system

▹ Draw lines from opposing corners and find point of intersection▹ Solves perspective problem by cancelling corners

27

Note:

To determine which corner is which, sort them by slope from the origin, take most extreme two and least extreme two and sort those by their X or Y value to ensure correct corners.

Important Math Functions - Geometric Calculations

Yaw

28

Pitch

▸ Finding Yaw/Pitch Offset to Target▹ Used for alignment to target around an axis

TargetX = The X value of the center of your targetTotalX = Total Width of screenThis will return a value -1 to 1 of how much the robot needs to move/rotate-1 is left, 1 is right

TargetY = The Y value of the center of your targetTotalY = Total Height of screenThis will return a value -1 to 1 of how much the robot needs to move1 is up, -1 is down

Comparison- Step 4 “Calculations”

Due to the nature of misdetections, it’s important to verify that data makes sense. This can be done through comparing data to previous data.

29

Store

If the data is good, store in in the place of the previous data for future comparisons.

Timestamp

Use your language’s time system to assign a variable to the time of the frame for latency comparison

Compare

Compare your current data to your most recent data to verify that it is possible. If not, disregard that data and wait for next data.

Transferring Vision DataMoving Vision Data from the Processor To Your Robot Controller

Methods of Communication - Moving Vision Data

Serial Communication

▸ Pros▹ Faster

▸ Cons▹ More difficult▹ Third Party Serial

Manager

31

Network Communication

▸ Pros▹ Easier▹ Built in support for

multiple values▸ Cons

▹ Slower▹ Requires a radio port

▸ NetworkTables Library▹ Built into WPILib▹ Can be compiled

independently▹ PyNetworkTables for

Python▹ System of sending

multiple values over radio

Network Communication - Moving Vision Data32

▸ Network Configuration▹ Use RoboRIO as Server▹ Use Coprocessor as Client▹ Send data over radio

Vision Processor RoboRIORadio

Using Vision DataMoving Vision Data from the Processor to the Robot Controller

Using Vision Data to Actuate Actuators

▸ This is where your Vision data will be used to move the robot.

▸ The values you have available:▹ Yaw Offset (-1 to 1)▹ Pitch Offset (-1 to 1)▹ Average height of contour for distance

34

Relative Tuning:Using simple math and geometric properties to find relative position of target, and then tuning robot mechanisms to that data.

We can use relative tuning to find a function that tells the robot specifically how to behave based on sensor data, in this case, Vision data.

Example:Sensor Data: Height of contour for distance Feedback: RPM of flywheelGoal: Adjust RPM of flywheel to match distance

Relative Tuning - Using Vision Data35

Vision Processor

RoboRIO PID Loop

Motor

Piston

1. Create a data table2. Use a dynamic variable as your X

axis (height of target contour)3. Use your robot adjustment for

your Y axis (flywheel RPM to get to goal)

4. Repeat a lot of times at different X values

5. Plot the data & create a line of best fit

6. Use that equation in your programming for interpreting vision data.

Relative Tuning - How To36

Thanks!Any questions?You can email us at programming@kingtec2169.comOr me personally at devosmaxwell@gmail.comOr see me after the presentation :)Presentation can be found on https://kingtec2169.com/resources/documents/