PCA + SVM using C++ Syntax in OpenCV 2.2

Question

I'm having problems getting PCA and Eigenfaces working using the latest C++ syntax with the Mat and PCA classes. The older C syntax took an array of IplImage* as a parameter to perform its processing and the current API only takes a Mat that is formatted by Column or Row. I took the Row approach using the reshape function to fit my image's matrix to fit in a single row. I eventually want to take this data and then use the SVM algorithm to perform detection, but when I do that all my data is just a stream of 0s. Can someone please help me out? What am I doing wrong? Thanks!

I saw this question and it's somewhat related, but I'm not sure what the solution is.

This is basically what I have:

vector<Mat> images; //This variable will be loaded with a set of images to perform PCA on.
Mat values(images.size(), 1, CV_32SC1); //Values are the corresponding values to each of my images.

int nEigens = images.size() - 1; //Number of Eigen Vectors.

//Load the images into a Matrix
Mat desc_mat(images.size(), images[0].rows * images[0].cols, CV_32FC1);
for (int i=0; i<images.size(); i++) {
  desc_mat.row(i) = images[i].reshape(1, 1);
}

Mat average;
PCA pca(desc_mat, average, CV_PCA_DATA_AS_ROW, nEigens);

Mat data(desc_mat.rows, nEigens, CV_32FC1); //This Mat will contain all the Eigenfaces that will be used later with SVM for detection

//Project the images onto the PCA subspace
for(int i=0; i<images.size(); i++) {
  Mat projectedMat(1, nEigens, CV_32FC1);
  pca.project(desc_mat.row(i), projectedMat);

  data.row(i) = projectedMat.row(0);
}

CvMat d1 = (CvMat)data;
CvMat d2 = (CvMat)values;

CvSVM svm;
svm.train(&d1, &d2);
svm.save("svmdata.xml");

Solution 1

What etarion said is correct.

To copy a column or row you always have to write:

Mat B = mat.col(i);
A.copyTo(B);

The following program shows how to perform a PCA in OpenCV. It'll show the mean image and the first three Eigenfaces. The images I used in there are available from http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html:

#include "cv.h"
#include "highgui.h"

using namespace std;
using namespace cv;

Mat normalize(const Mat& src) {
    Mat srcnorm;
    normalize(src, srcnorm, 0, 255, NORM_MINMAX, CV_8UC1);
    return srcnorm;
}

int main(int argc, char *argv[]) {
    vector<Mat> db;

    // load greyscale images (these are from http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html)
    db.push_back(imread("s1/1.pgm",0));
    db.push_back(imread("s1/2.pgm",0));
    db.push_back(imread("s1/3.pgm",0));

    db.push_back(imread("s2/1.pgm",0));
    db.push_back(imread("s2/2.pgm",0));
    db.push_back(imread("s2/3.pgm",0));

    db.push_back(imread("s3/1.pgm",0));
    db.push_back(imread("s3/2.pgm",0));
    db.push_back(imread("s3/3.pgm",0));

    db.push_back(imread("s4/1.pgm",0));
    db.push_back(imread("s4/2.pgm",0));
    db.push_back(imread("s4/3.pgm",0));

    int total = db[0].rows * db[0].cols;

    // build matrix (column)
    Mat mat(total, db.size(), CV_32FC1);
    for(int i = 0; i < db.size(); i++) {
        Mat X = mat.col(i);
        db[i].reshape(1, total).col(0).convertTo(X, CV_32FC1, 1/255.);
    }

    // Change to the number of principal components you want:
    int numPrincipalComponents = 12;

    // Do the PCA:
    PCA pca(mat, Mat(), CV_PCA_DATA_AS_COL, numPrincipalComponents);

    // Create the Windows:
    namedWindow("avg", 1);
    namedWindow("pc1", 1);
    namedWindow("pc2", 1);
    namedWindow("pc3", 1);

    // Mean face:
    imshow("avg", pca.mean.reshape(1, db[0].rows));

    // First three eigenfaces:
    imshow("pc1", normalize(pca.eigenvectors.row(0)).reshape(1, db[0].rows));
    imshow("pc2", normalize(pca.eigenvectors.row(1)).reshape(1, db[0].rows));
    imshow("pc3", normalize(pca.eigenvectors.row(2)).reshape(1, db[0].rows));

    // Show the windows:
    waitKey(0);
}

and if you want to build the matrix by row (like in your original question above) use this instead:

// build matrix
Mat mat(db.size(), total, CV_32FC1);
for(int i = 0; i < db.size(); i++) {
    Mat X = mat.row(i);
    db[i].reshape(1, 1).row(0).convertTo(X, CV_32FC1, 1/255.);
}

and set the flag in the PCA to:

CV_PCA_DATA_AS_ROW

Regarding machine learning. I wrote a document on machine learning with the OpenCV C++ API that has examples for most of the classifiers, including Support Vector Machines. Maybe you can get some inspiration there: http://www.bytefish.de/pdf/machinelearning.pdf.

Solution 2

data.row(i) = projectedMat.row(0);

This will not work. operator= is a shallow copy, meaning no data is actually copied. Use

cv::Mat sample = data.row(i); // also a shallow copy, points to old data!
projectedMat.row(0).copyTo(sample);

The same also for:

desc_mat.row(i) = images[i].reshape(1, 1);

Comments

[Roland Nasr]

Does it matter that it's only a shallow copy? It will just update the pointer to that set of data, which still should be usable. Correct?

[etarion]

@rsnj: It does matter because that statement is essentially a no-op. data.row(i) creates a new matrix header that points to the same data, now operator= modifies this new returned matrix header and does nothing else, and at the end of the expression the temporary matrix header (that you modified) gets destructed. data is completely unmodified. the opencv manual explicitely states that this "doesn't work".

[Roland Nasr]

I've been looking at the tests and I think I'm more confused than I was before. I see they use a Random Number Generator to construct their Matrix to perform PCA on. Am I correct to assume that my approach on reshaping my Image Mat's and adding them to the rows of a new Mat is the correct approach? The thing that really confuses me between the C and C++ version is the C accepts an array of images (or Matrix) and the C++ version only accepts a single Matrix. Thanks!