C++ CRF Code
blankness
imageCRF
HOME

 

C++ CRF Code for Images (imageCRF)

This package is useful for crfs for image segmentation problems using 2D images, 3D volumes of images (for example in medical imaging) and video data (containing frames). The code was written during my graduate school. The program heavily uses the MRF stereo package and optical flow package and some other packages for support. Please look into the README.txt for more information about all packages and citation. I will be trying to add more documentation with time. Email me at lastname AT cs.rochester.edu if you need more help [lastname = bhole].

imageCRF code (v1.0) :

Sample data for crf (v1.0) :

 

Features of the crf package

  1. Support for logistic regression and crf for 2D images and 3D image volumes.
  2. For 3D volumes, connections based on optical flow for video are also supported.
  3. Configuration for arbitrary connections between neighboring nodes.
  4. Support for use of different inference algorithms like max-product belief propagation, graph-cuts, variational message passing. Additional support for other inference algorithms like TRWS, sum-product belief propagation etc can easily be added since code exists in the included MRF package.
  5. The package is designed to work with input features that are represented as histograms. Please see the section on 'Input Features' and the paper[1] for more details.
  6. Uses gradient descent for parameter learning.
    l-bfgs is implemented but works only for logistic regression.
  7. Limited handling of unknown label nodes during training and interactive mode.

 

Quick Installation and Running Instructions

You need to make sure you install openCV and libpng on your machine. The Notes.txt file in the imageCRF folder explains how to install openCV on Mac OS X. You also need to make sure the correct file paths for these libraries are mentioned in the Makefile(s) of all the packages included.

Use the following script to compile all the packages in the imageCRF.

$ ./compileall.sh

Run the modelmain program. You need to pass the parameter file and a temporary debug file. The following are sample runs for different parameter files provided in the data folder. More details about the usage will follow soon. If you need more details before then, contact me at lastname AT cs.rochester.edu [lastname = bhole].

For 2D images :
$ ./modelmain ../data/2Dimages/parameters/parlrI.cfg ../data/tempdebug.txt
$ ./modelmain ../data/2Dimages/parameters/parcrfI.cfg ../data/tempdebug.txt
For videos with frames :
$ ./modelmain ../data/video/case1/parameters/parlrI.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case1/parameters/parcrfI.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case2/parameters/parlrI_oplocal.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case2/parameters/parlrI.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case2/parameters/parcrfI.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case2/parameters/parlrI_final.cfg ../data/tempdebug.txt
$ ./modelmain ../data/video/case2/parameters/parcrfI_final.cfg ../data/tempdebug.txt

 

Detailed Installation Instructions on Linux

  1. Install libpng, lame, faac, faad2
  2. Install opencv
  3. I have disabled the use of openmp but just in case you see an error pointing to it, you might need to install openmp and find where omp.h is.
  4. Occasionally, i have also seen a problem with libconfig++.9.so not being able to be found. In that case you can to make sure the 'so' file is in the LD_LIBRARY_PATH.
  5. In the main crf package, open the file compileall.sh.
    In compileall.sh, you need to pass either linux32 or linux64 depending on whether you are using a 32 bit machine or a 64 bit machine. Note that there are two places you need to change this.
  6. In the makefile of crf package, Makefile, you need to set the OpenCV and lib (libm) paths correctly depending on where you have the libraries installed. Look at the first few lines of Makefile to set the paths correctly.
  7. Run
    $ ./compileall.sh
    

 

Detailed Installation Instructions on Mac

  1. Install libpng
  2. You might want to install macports and curl if they are not already installed. (these might be needed for installing opencv). The port command is installed in /opt/local/bin .If not on your path, you will need to use the full path while using it later on.
  3. Install OpenCV (Instructions that follow worked for OpenCV-2.4.3 on Mac Lion (10.7))
    1. I used instructions from here but had some errors. Many instructions on the website mentioned are quite useful.
    2. Install lame, faac, faad2 (these might be needed for opencv)
    3. If you don't get opencv installed using curl, download the opencv tar.gz or tar.bz2 (here) and in the source directory (as mentioned in the link above) change modules/highgui/CMakeLists.txt and add just before
      if (HAVE_FFMPEG) (on line 158) the following text:

      if(APPLE)
          list(APPEND HIGHGUI_LIBRARIES ${BZIP2_LIBRARIES} -lmp3lame -lfaac -lbz2)
      endif(APPLE)

    4. Install cmake using macports
      sudo port install cmake
    5. mkdir build
      cd build
      cmake -G "Unix Makefiles" ..
      make
      sudo make install
      opencv gets installed in /Users/${USERNAME}/libraries/OpenCV-2.4.3/
  4. I have disabled the use of openmp but just in case you see an error pointing to it, you might need to install openmp and find where omp.h is. On my mac omp.h was in /usr/lib/gcc/i686-apple-darwin11/4.2.1/include/
  5. Occasionally, i have also seen a problem with libconfig++.9.dylib not being able to be found. In that case you can try the following:
    cp libconfig++.9.dylib to /usr/local/lib
    OR
    DYLD_LIBRARY_PATH=/Users/${USERNAME}/Documents/imageCRF.v1.0/crf/libconfig-1.4.6/lib/.libs/
    export DYLD_LIBRARY_PATH
  6. In the main crf package, open the file compileall.sh.
    In compileall.sh, you need to pass either macosx, linux32 or linux64. So use macosx in this case. Note that there are two places you need to change this.
  7. In the makefile of crf package, Makefile, you need to set the OpenCV and lib (libm) paths correctly depending on where you have the libraries installed. Look at the first few lines of Makefile to set the paths correctly.
  8. Run
    $ ./compileall.sh
    

 

Input Features

The main features supported are intensity for grayscale images, RGB (actually Lab color space) for color images, pairwise features (based on labels or gradient), location, HOG, optical flow, appearance and motons. Intensity, RGB, pairwise, optical flow can be used directly. Note that HOG, appearance and motons will need the user to generate these from external packages, cluster them using kmeans or other clustering software. More instructions will follow.

 

Input Parameters

The data files have examples of parameter files. The path of this file is set in the config file under the featureparams section with the title name 'filename'.

There is a different character for each feature. For e.g. 'u' for intensity, 'v' for pairwise features, 'd' for pairwise features along the depth (or z direction), 'l' for location parameter values, 'f' for optical features, 's' for appearance, 'q' for motons, 'h' for HOG features.

'u', 'v', 'd', 'l' and 'f' are special features which are handled specific to the features.
's', 'q' and 'h' could potentially be used for any other feature if they are represented in the form mentioned in the paper[1] i.e. they are clustered and there is one parameter for each cluster and class label pair.

The number of times each parameter is entered is the number of feature bins * number of class labels for datacost features and number of feature bins * number of class labels * number of class labels for the pairwise features.
For example, if the grayscale intensity (range of 0 to 255) is broken into bins {0-51, 52-102, 103-154, 155-205, 207-255}, then there are 5 bins. If we consider there are 3 class labels, then the number of 'u' parameters that need to be entered are 15.

The reason for not using the short hand notation of entering only the number of parameters needed and having all set to the same initial value is that it is typically a good idea to learn the datacost parameters using a logistic regression first before using the crf. These logistic regression will assign different values to these parameters. These can be transfered to the crf parameter file as initial values and the crf can then train the datacost and pairwise parameters jointly.

 

Configuration File

commented.cfg is an example of a configuration file with detailed explanation of the flags.
Remember that only either logreg (logistic regression) or crf should be enabled, not both.

 

IO files/directories

This pacakage was mainly designed to work on 3D volumes of images and videos. The artifact is that you need a folder for each training instance. That means each set of 3D images for a training instance needs to have a separate folder.
For example, if you look at the data, video/case1/crf/input will contain folders, one for each training instance. For the over-simplified example I provide, I only have one training instance and one test instance. In the case of the 2Dimages/case1/crf/input, there are 5 training instances and 2 test instances each having only one image because it is 2D input.

The video/case1 folder typically contains the following folders and files:

  • lr - logistic regression input folder
  • lr_out - logistic regression output folder
  • lr_out.txt - logistic regression output file containing training accuracy and test accuracy for each iteration and also dumps the parameter values at each iteration and is useful while debugging
  • lr_outparam.txt - logistic regression parameters learnt after completion
  • crf - crf input folder
  • crf_out - crf output folder
  • crf_out.txt - crf output file containing training accuracy and test accuracy for each iteration and also dumps the parameter values at each iteration and is useful while debugging
  • crf_outparam.txt - crf parameters learnt after completion
  • parameters - contains the config files

'lr' or 'crf' folders in turn contain the following folders and files:
  • input - contains one folder for each training instance and test instance
  • gtruth - corresponds to the ground truth labeling for training of the model and contains one folder for each training instance and test instance
  • intern - corresponds to handling unknown labels during training and interactive mode andcontains one folder for each training instance and test instance
  • loc - corresponds to images for the location feature, contains one folder for each training instance and test instance (more on this later)
  • cache - contains the optical flow calculated values and caches the files. In case the same data is run, using the cache files saves significant time.
  • b???????.txt - the parameters with their values are included in this file. The ? indicate whether a feature is enabled or not. (more on this later)
Note that the folder names must correspond to one another i.e. input/cub1 must be the same training instance for gtruth/cub1.

The test instances need to have a name starting with 'test' and the test flag in the config file must be set.

 

References

  1. Bhole, Chetan and Pal, Christopher and Rim, David and Wismüller, Axel (2013)
    3D segmentation of abdominal CT imagery with graphical models, conditional random fields and learning.
    Machine Vision and Applications, pp. 1-25 (Springer Online First Article).
more on this later