Processing with OpenCV

Previously, I mentioned that I plan to use OpenCV. Therefore, I will focus on processing images from the camera using this library.

The brown and blue spots are the handles on the wall. The projector projects the circles with numbers. The goal is to detect obscuration, specifically, to spot the collisions. The first step is to prepare the image for analysis.

I will use an image from a camera pointed at the wall for detection. Processing the differences between successive image frames will give me information about what has changed. That is where the movement occurs. To perform all necessary operations, I will use available functions from OpenCV. The library was created in the Russian branch of Intel and is currently maintained by the Itseez group. Its primary purpose is real-time image processing and analysis. Ports for the most important languages are available, perfect for my applications.

First, I must install it; again, I will use Contribution Manager.

From this point, the benefits of this colossus are available. The first step in the analysis will be to eliminate the background. The most primitive solution is to subtract the previously-stored background from each frame. The resulting difference will make it easier to detect those elements that appear in the background. For now, my background will be a piece of wall. I want to catch that something has occurred in this image.

The colour image won’t be of much use in the analysis, so I’ll convert it to binary. We get this conversion using the threshold() function. The argument defines the cutoff level. It is marked in the figure – the blue line. In red, what we reject, in green what is left and will be white. If the image is dark, there will be a lot on the left and not much on the right on the histogram. It’s the case with my pictures. That’s why I set the cutoff factor relatively low, that is, at 50. Otherwise, I wouldn’t be able to pull anything out of the image. Interestingly, what Processing makes available is extremely poor in capabilities. The original OpenCV is more complex.

The result of the threshold() operation

The operation responsible for determining the difference is diff(). The getSnapshot()call returns the current state in OpenCV, resulting from the previous operations. Which, to tell you the truth, is not so good. You have to remember what was set there recently in the object all the time. Therefore, I added the getGrayImage function for convenience, which creates loads and transforms the appropriate image.

import gab.opencv.*;
import org.opencv.imgproc.Imgproc;

OpenCV opencv;

PImage src, bkg, diffed;

final static int W = 960;
final static int H = 540;


PImage getGrayImage(String fileName)
{
  OpenCV cv = new OpenCV(this, loadImage(fileName));
  cv.gray();
      
  PImage img = cv.getSnapshot();
  img.resize(W/2, H/2);
  
  return img;
}

void setup() 
{
  size(960, 540);

  opencv = new OpenCV(this, W/2, H/2);
  opencv.gray();

  bkg = getGrayImage("frame-bkg.jpg");
  src = getGrayImage("frame-reka-na-tle.jpg");
 
  opencv.loadImage( src );
  opencv.diff( bkg );
  
  diffed = opencv.getSnapshot();

  image(bkg, 0, 0);
  image(diffed, W/2, 0);
  
  opencv.threshold(50);
  image(opencv.getSnapshot(), 0, H/2);
  
  opencv.loadImage( diffed );
  opencv.threshold(60);
  image(opencv.getSnapshot(), W/2, H/2);


  noLoop(); // nie potrzebuję wywoływać draw(), bo już wszystkie operacje zostały wykonane.
}

It already looks pretty promising, but it would be good to process it a bit more. I want to get a more regular shape and remove the noise. Here, the morphological transformations – erosion and dilation – come to the rescue. They change the image and allow the data preparation for further analysis. As far as more details are concerned, I suggest looking in some more severe places. Google is spilling links left and right.

The erosion of an image makes objects smaller, thus separating them from others in the picture. It also eliminates small pins, points, etc. These appear due to camera inaccuracy, sensor noise, vibration and other inaccuracies. Consequently, I get an effect similar to a reduction in detail, or something has eaten elements in the image.

After eroding, it would be nice to fill in the resulting gaps and sharp edges. To do this, I will use an expansion joint. The effect is similar to growing dough around the elements.

More dilate calls generate this effect:

The code that generates the above image:

import gab.opencv.*;

OpenCV opencv;

PImage src, bkg, diffed;

final static int W = 960;
final static int H = 540;


PImage getGrayImage(String fileName)
{
  OpenCV cv = new OpenCV(this, loadImage(fileName));
  cv.gray();
      
  PImage img = cv.getSnapshot();
  img.resize(W/2, H/2);
  
  return img;
}

void setup() 
{
  size(960, 540);

  opencv = new OpenCV(this, W/2, H/2);
  opencv.gray();

  bkg = getGrayImage("c:/workspace/climbing/#media/frame-bkg.jpg");
  src = getGrayImage("c:/workspace/climbing/#media/frame-reka-na-tle.jpg");
 
  opencv.loadImage( src );
  opencv.diff(bkg);
  diffed = opencv.getSnapshot();

  image(bkg, 0, 0);
  image(diffed, W/2, 0);
  
  opencv.threshold(80);
  
  image(opencv.getSnapshot(), 0, H/2);

  opencv.erode();
  opencv.dilate();

  opencv.dilate();
  opencv.dilate();
  opencv.dilate();
  opencv.dilate();
  
  opencv.dilate();
  opencv.erode();
 
  image(opencv.getSnapshot(), W/2, H/2);

  noLoop();
}

To experiment, I added some more blur and raised the contrast. It all turned out dark. I also added the following processing steps.

In this example, I’m using the first captured frame as the background right after starting the program. I may have chosen the environment slightly wrong, and the effect is not spectacular. However, I think it shows at least a little of what’s going on. It’s undoubtedly better to run it and see it live. The code generating the following six steps looks like this:

import gab.opencv.*;
import processing.video.*;

OpenCV opencv;
Capture video;
PImage src, bkg, thresh, dilated, eroded, diff, both;

final static int W = 960;
final static int H = 540;


void setup() 
{
  size(1440, 540);
  frameRate(30);

  opencv = new OpenCV(this, W, H);

  video = new Capture(this, W, H);
  video.start();
}

void update()
{
  if ( video.available() )
  {
    video.read();
    
    // pierwsza klatka traktowana jest jako tło
    if ( bkg == null )
      bkg = video.copy();
    
    opencv.loadImage(video);
    src = opencv.getSnapshot();
    
    opencv.diff(bkg);
    opencv.blur(10);
  
    diff = opencv.getSnapshot();
    opencv.threshold(50);
    
    thresh = opencv.getSnapshot();
  
    opencv.erode();
    eroded = opencv.getSnapshot();

    opencv.loadImage(thresh);
    opencv.dilate();
    dilated = opencv.getSnapshot();
    
    opencv.loadImage(thresh);
    
    //opening
    opencv.erode();
    opencv.dilate(); 
    
    opencv.dilate(); 
    opencv.dilate(); 
    
    //closing
    opencv.dilate();    
    opencv.erode();

    both = opencv.getSnapshot();
  }
}

void draw() 
{
  update();

  if (src == null)
    return;
    
  image(bkg,       0,   0, W/2, H/2);
  image(src,     W/2,   0, W/2, H/2);
  image(diff,      W,   0, W/2, H/2);
  image(eroded,    0, H/2, W/2, H/2);
  image(dilated, W/2, H/2, W/2, H/2);
  image(both,      W, H/2, W/2, H/2);

  fill(255, 0, 0);
  text("bkg",       0 + 10,   0 + 10);
  text("src",     W/2 + 10,   0 + 10);
  text("diff",      W + 10,   0 + 10);
  text("eroded",    0 + 10, H/2 + 10);
  text("dilated", W/2 + 10, H/2 + 10);
  text("both",      W + 10, H/2 + 10);
}

You can read more about filtering here: http://sun.aei.polsl.pl/ and http://etacar.put.poznan.pl/.

So far, the whole thing looks like preparation for Computer Vision labs. Some pretty ugly code came out. The effects are okay, but I can not see that it will look significantly better. The functions are there, it’s easy to use, but there is a lack of control over various operations, including eroding and dilating. Of course, I can run the given process repeatedly in a loop, but it is far from an optimal solution. I also did not find the possibility of extensive parameterization of transformations.

Additionally, I don’t like the editor in Processing. I could probably plug in something external, but I don’t particularly feel like searching for a solution. Unfortunately, I’m lazy, and I like to have a convenient editor with a debugger at hand, so I’m jumping to C++. I’m currently on Windows, so I’ll use VisualStudio 2013 because that’s what I have installed. As for libraries, I choose pure OpenCV, OpenFrameworks or Cinder. I don’t have the faintest idea which will be optimal.

I choose Cinder; it has a cool icon – there must always be a reason.

Indeed, the first steps in Processing have sped up the experiments considerably, but now it’s time for a more serious solution. The following entry is a rewrite of what I have for Cinder+OpenCV.