Ensemble of different local descriptors, codebook generation methods and subwindow configurations for building a reliable computer vision system

2.1 SIFT descriptor

The SIFT¹ descriptor (Lowe, 2004) is a 3D histogram that takes the gradient locations (in our case quantized into a 4 × 4 location grid) and orientations (quantized into eight values) and weighs them by the gradient magnitude and a Gaussian window superimposed over the region. The SIFT descriptor, which is obtained by concatenating the orientation histograms over all bins and normalized to unit length, shows how the local gradients around a point are aligned and distributed at different scales.

2.2 Local ternary patterns (LTP)

LTP is a recent variant (Tan and Triggs, 2007) of LBP. The LBP operator is rotation invariant and evaluates the binary difference between the gray value of a pixel x and the gray values of P neighboring pixels on a circle of radius R around x. A problem with conventional LBP is its sensitivity to noise in the near-uniform image regions. The three value encoding scheme of LTP overcomes this problem. The implementation of LTP used in our experiments is a modification of the original LBP Matlab² code. It includes three value encodings and a normalized histogram.

In our experiments, we used both the rotation invariant bins and the uniform bins, with each descriptor used to train a different classifier. The final descriptor was obtained by concatenating the features extracted with (R = 1, P = 8) and (R = 2, P = 16). We tested both LTP with uniform bins (LTP-u) and LTP with rotation invariant uniform bins (LTP-r).

2.3 Local phase quantization (LPQ)

LPQ³ is a texture descriptor (Ojansivu and Heikkila, 2008) that uses the local phase information extracted from the 2-D short-term Fourier transform (STFT) computed over a rectangular neighborhood of radius R at each pixel position in an image. Only four complex coefficients, corresponding to the 2-D frequencies, are considered and quantized using a scalar quantizer between 0 and 255. The final descriptor is the normalized histogram of the LPQ values. Different LPQ descriptors were evaluated in our experiments, with two selected for our final system. Both of these were extracted by varying the parameter R (specifically, R = 3 and R = 5), and each descriptor was used to train a different classifier.

2.4 GIST

The GIST descriptor (Oliva and Torralba, 2001) computes the energy of a bank of Gabor-like filters evaluated at 8 orientations and 4 different scales. The square output of each filter is then averaged on a 4 × 4 grid.

2.5 The histogram of oriented edges (HOG)

One way of looking at HOG (Dalal and Triggs, 2005) is as a simplified version of SIFT. HOG calculates intensity gradients from pixel to pixel and selects a corresponding histogram bin for each pixel based on the gradient direction.

The HOG features extracted in our experiments used a 2 × 2 version of the HOG. The HOG features were extracted on a regular grid at steps of 8 pixels and stacked together considering sets of 2 × 2 neighbors to form a longer descriptor with more descriptive power.

2.6 Daubechies wavelets (DW)

As explained in Huang et al. (2003), DW is a feature extraction method where the average energy of the three high-frequency components is calculated up to the Lth level decomposition using both the scaling and the wavelet functions of the selected wavelet. In our experiments we use decomposition L = 10 coupled with Daubechies 4 wavelet function.

2.7 Laplacian features (LF)

LP, as proposed in Xu et al. (2012), is based on a SIFT-like descriptor extracted at different window sizes. A descriptor called the multifractal spectrum (MFS) then extracts the power-law behavior of the local feature distributions over the scale. Finally, to improve robustness to changes in scale, a multi-scale representation of the multi-fractal spectra under a wavelet tight frame system is proposed.

2.8 Local derivative pattern (LDP)

As detailed in Zhang et al. (2010), LDP is a general framework that encodes directional pattern features based on local derivative variations. The LDP templates extract high-order local information by encoding various distinctive spatial relationships contained in a given local region.

2.9 Speeded up robust features (SURF)

SURF Xiao et al., 2010 is an improvement of the famous SIFT features (Lowe, 2004). SURF extract features starting from interest points detected by a method based on the Hessian matrix. A set of features based on Haar wavelet response around a point of interest is then extracted. To speed the feature extraction step, the integral image is used.

3.1 Cloud of features

In our experiment we also tested a novel method for object recognition based on cloud of features (Lai et al., 2004). A cloud represents image i as M_i feature vectors, storing the information on M_i single points, or patches in the image. Each point/patch is a “subwindow” of fixed size (we run three methods with size = {8, 10, 12}. The cloud points, C_i, of an image, I_i, are formed by these patches. From each point/patch (in this paper we extracted {150, 250, 350} points/patches from each image), we extracted a given descriptor, giving us 9 classifiers (three size × the number of retainer patches). These nine classifiers were combined by sum rule.

In our experiments, we fit a one-class SVM (Maneivitz and Yousef, 2002), implemented as in libSVM (www.csie.ntu.edu.tw/∼cjlin/libsvm/) around the cloud of points C_i and enclose the data by a hypersphere H_i of minimal volume. We can describe this process formally as follows:

Let the hypersphere be described by the center a and the radius R. For a vector x, coming from the cloud of points C_j, representing the image I_j, we define: $${\text{C}}^i(\mathbf{x})=\text{Q,}$$ where x is accepted by the one-class classifier H_i that describes the cloud points of the image pair I_i and where Q is the indicator function (for instance, Q(A) = 1, if the condition A is true and 0 otherwise). In our experiments the acceptance of the vector x is defined as follows:

• 1, if x is accepted by the one-class classifier;

• 0, if x is rejected by the one-class classifier.

An image I_j could be classified by taking into account the fraction of vectors from the cloud representation C_j, which are rejected by the classifier H_i: $${\text{S}}_i({\text{I}}_j)=\left(\frac{1}{{\text{M}}_j}\right){\displaystyle \sum _{x∊{\text{C}}_j}}(1-{\text{C}}^i(\mathit{x}))$$ where M_j is the size of the cloud C_j .

Notice that if only one classifier is used, the performance may suffer from a large overlap between individual clouds of points. For instance, if a cloud subsumes another originating from a different class, the percentage of outliers (vector x rejected) can still be zero. Such a situation lowers the performance of the whole system. To prevent this from happening, the information given by all the classifiers is combined. The relations among the images of the training set can be evaluated using a “classifier profile,” which expresses the dissimilarities among a given image from all the images of the training set. The classifier profile of a test image is now classified by SVM trained using the classifier profile of the images that belong to the training set. We assign each image of the testing set to the class of the nearest neighbor (for a better mathematical description of classifier profile, see Lai et al., 2004).

4.1 Datasets description

4.1.1

4.1.1 Scene dataset (Oliva and Torralba, 2001)

This dataset has the following fifteen categories (we set NIMG = 50): coast (360 images), forest (328 images), mountain (274 images), open country (410 images), highway (260 images), inside city (308 images), tall building (356 images), street (292 images), bedroom (216 images), kitchen (210 images), living room (289 images), office (215 images), suburb (241 images), industrial (311 images), and store (315 images). Images are approximately 300 × 250. Fig. 2 shows a few samples.

Close

Figure 2

Samples from the scene dataset.

Export to PPT

The testing protocol established by other papers in the literature for the scene dataset, which we followed, requires 5 experiments, each using 100 randomly selected images per category for training and the remaining images for testing. The performance indicator is the accuracy, which is averaged across the experiments.

4.1.2

4.1.2 Caltech-256

This dataset includes a challenging set of 256 object categories containing a total of 30,607 images with at least 80 images for each category. The images in Caltech-256 were collected by choosing a set of object categories, and then by downloading examples from Google Images. The final dataset was produced by manually screening out all images that did not fit the chosen category. According to a widely used protocol, we ran 5 split tests using 40 images per class for training (we set NIMG = 40) and 25 for testing. The performance indicator was the accuracy, which was averaged on the 5 experiments. Fig. 3 provides samples of some images contained in the Caltech-256 dataset.

Close

Figure 3

Samples from the Caltech dataset.

Export to PPT

4.1.3

4.1.3 Building recognition

This dataset (Amato et al., 2010) contains 1227 photographs crawled from Flickrs of landmarks located in Pisa. The dataset is divided into 12 classes having a minimum of 46 images per class. According to the official testing protocol (Amato et al., 2010), the dataset should be divided into a training set of 921 photos (we set NIMG = 50), or approximately 80% of the dataset and a testing set of 226, or approximately 20% of the dataset. For comparison purposes these two sets are provided on the web page of one of the authors of Amato et al. (2010). The performance indicator used in our experiments was accuracy. Some sample images are shown in Fig. 4.

Close

Figure 4

Samples from the landmark dataset.

Export to PPT

The VOC2012 dataset (Everingham et al., 2012) includes twenty classes of images from four main categories:

• Person: people;

• Animal: bird, cat, cow, dog, horse, and sheep;

• Vehicle: airplane, bicycle, boat, bus, car, motorbike, and train;

• Indoor: bottle, chair, dining table, potted plant, sofa, and tv/monitor.

According to the official protocol, we use Trainval images for training and the Testing set of images for testing. Several example images of the VOC2012 dataset are available at http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2012/examples/index.html. The official performance indicator is the average precision (AP). For a given task and class, the precision/recall curve is computed from a method’s ranked output. Recall is defined as the proportion of all positive examples ranked above a given rank. Precision is the proportion of all examples above that rank which are from the positive class. The AP summarizes the shape of the precision/recall curve. It is defined as the mean precision at a set of eleven equally spaced recall levels $[0\text{,}0.1\text{,}\dots \text{,}1]$ (Everingham et al., 2010).

Due to restraints on computation time, we focused only on the person image classification contest (we set NIMG = 250).

4.2 Empirical results

The first experiment, reported in Tables 1–3, was aimed at comparing variants of the steps in our proposed approach. In this table performance of the descriptors detailed in Section 2 is reported.

In particular, we evaluated the performance obtained by considering different global descriptors (STEP2 in Section 3). In these tests we used only a stand-alone SVM, with the radial basis function kernel and no dataset parameter tuning⁶ for each dataset. We compared the following different approaches for the global descriptors:

• B: the features were extracted from the whole image.

• B1: the image was divided in four equal regions without overlap and a central region of size 1/2 of the original image. For each region a different SVM was trained; the five descriptors were then used to train four SVMs combined by sum rule.

• M1: as in B1, but instead of the original image, we used the foreground region extracted method using the saliency map with TH = 0.15.

• M2, as in B1, but instead of the original image, we used the foreground region extracted using the saliency map with TH = 0.25.

The following fusions among descriptors were also compared:

• Fus1: sum rule among the descriptors based on B1 and M1;

• Fus2: sum rule among the descriptors based on B1, M1, and M2;

• ALL1: sum rule among the descriptors of Fus1;

• ALL2: sum rule among the descriptors of Fus2.

In the Scene dataset (see Table 1), B1 outperforms M1. This is due to the fact that the background region discarded by the saliency map contained important information for classifying a given image in a given scene class. The fusion, Fus1 (sum rule between B1 and M1), outperforms B1 because different descriptors are extracted from the two different images (the original image when B1 is applied and the foreground region when M1 is used).

In the building recognition task, it is typically important to discard background regions. For this reason, M1 outperforms B1 in the Building dataset (see Table 2). The best results are also obtained in this dataset by combining different approaches, with Fus2 outperforming the other approaches. LPQ(5) works better than ALL1 or ALL2 in the Building dataset but not on others: when using this approach, it would be desirable to test it using the training data to determine whether it is suited to that specific classification.

In the Caltech-256 dataset (see Table 3), the saliency approaches do not outperform B1, but both Fus1 and Fus2 outperform B1, M1, and M2.

From the results reported in Tables 1–3, the following conclusions can be drawn:

• In the building dataset, it is clear that global descriptors work well. Notice that several of our approaches outperform the SIFT based method reported in Amato et al. (2010), which obtains an accuracy of 92%.

• It is interesting to note how differently the same descriptor works in each of the datasets. In the scene/landmark datasets, e.g., GIST works poorly with respect to the object classification dataset. Fusion, it should be noted, works well in all the tested datasets.

• For a statistical validation of our experiments, we have used the Wilcoxon signed rank test (Demsar, 2006) to compare FUS2 and B1 (considering the different datasets and the different descriptors); we found a statistical difference with a p-value of 0.05.

• Finally, we should note that some state-of-art stand-alone approaches obtain a performance that is similar to our stand-alone best methods. The SIFT based method tested in Xiao et al. (2010) (which was the best method using an ensemble approach), for example, obtained an accuracy of 81.2% in the scene dataset, while the salient coding approach (Huang et al., 2011), which is a better performing variant of LLC (the winner of VOC2009), obtained 82.6% in the scene dataset.

In the BoW approach proposed in this work, we used a stand-alone SVM as our classifier with the histogram intersection kernel (without any parameters tuning for each dataset⁷: the same settings were used in all the datasets and with all the descriptors). Using the same kernel with the same parameters for all the datasets and all the descriptors is very useful for practitioners since they can use this same set of features in their datasets.

In our second set of experiments, reported in Table 4, we compare some variants in the steps of our proposed approach using the scene dataset. We tested several descriptors derived from the original image − all those described in Section 2 and their fusion by sum rule. We label the methods as follows:

• −1: a version of our system based on a single codebook creation (only one PCA projection) with patches with ps = 8% and only the first method of STEP6 for codebook creation is used.

• M−1: a version of our system based on using all the PCA projections for building patches (see Section 3 – TRAINING2:PCA) with ps = 8% and only the first method of STEP6 for codebook creation.

• −2: as in −1, but the second method of STEP6 is used.

• M−2: as in M−1, but the second method of STEP6 is used.

• Local⁸: a complete version of our system based only on a local descriptor; we combine, by sum rule, the scores obtained by the SVMs trained using the textons vocabulary obtained both with ps = 12.5% and ps = 8%.

• Fusion: the fusion of all the Local descriptors by sum rule.

As can be seen in Table 4, our ensemble approach improves the performance of stand-alone approaches (compare −1 with Local), thus gaining a performance similar to state-of-the-art stand-alone approaches: the SIFT based method tested in Xiao et al. (2010) obtained 81.2%, while salient coding based method proposed in Huang et al. (2011) obtained 82.6%.

In Table 5 we report the results obtained by our system compared to the best methods reported in the literature for the scene dataset. In the following tables we named the whole system detailed in Section 3 as ensemble of different local descriptors (EDL), i.e. global descriptors combined with the local descriptors.

Due to computational time factors, we used fewer descriptors for our bag of feature approach in the Caltech and Building datasets. Results are reported in Table 6. In the Building dataset, we used images extracted from the foreground region with the saliency map TH = 0.15, since this method using global descriptors produced the best results. We used the original images in the Caltech dataset because this produced the best results with this dataset.

In the Building dataset, the second approach for building the codebook works very poorly, so in Local this approach was not considered in this dataset. It is clear in Table 6 that the ensemble outperforms the stand-alone approach (compare Local with −1).

In Table 7 we compare results obtained by our complete system with the best methods reported in the literature. It should be noted that in Demsar (2006), Gehler and Nowozin (2009) and Yang et al. (2009) 45 images in Caltech-256 dataset were used for training each class, making their training sets larger than those used in our system. In Torresani et al. (2010) several approaches are compared (see Fig. 2 of that paper⁹), and only LPbeta (i.e. Gehler and Nowozin, 2009) among the approaches based on 40 training images obtained a performance higher than 40%.

In Table 8, we report the results obtained by the cloud of feature approach (Cloud), comparing it with the local approach detailed in Section 2 (Local) and their fusion by weighted sum rule (SUM1) where the weight of Local is three and the weight of Cloud is 1. Cloud is applied separately in the five regions (B1), and then these five scores are combined by sum rule.

To reduce the computational time, we used only three descriptors and ran tests on only two datasets. These preliminary results show that the method proposed here could be considered a new approach for object classification.

In Table 8, each cell of Cloud reports two values. The first is the performance obtained considering the fusion among the nine approaches detailed in Section 3.1. The value between the brackets is the performance obtained by a standalone method, with DIMsize = 10 and 150 patches retained in each image. It is clear in these experiments that Cloud is very well suited for building an ensemble of classifiers.

Finally, in Table 9, we report the results on the VOC2012 contest. We ran our approach on only the person classification dataset. Since our ensemble approach needs bounding boxes that contain a given object to classify, we used the method proposed in Bourdev and Malik (2009) for extracting subwindows that might contain a person (we simply retained the 15 regions with higher similarity to the person template used (Bourdev and Malik, 2009). Since each subwindow was of a different dimension, we resized each so that the minimum size was 40 pixels. Each subwindow was classified as person/non-person using our approach.

The results reported in Table 9 are a subset of the validation set (500 person images and 1500 non-person images) and only the B1 approach is used for the global descriptor (the saliency map is not considered). Moreover, the TRAINING2 step is iterated only four times (two times retaining 99% of the variance and two times retaining 98% of the variance). These restrictions were due to constraints in computation time. It should be remarked that this dataset is built by images with complex backgrounds that contain no information about the class of the images. Increasing the dimension of the ensemble would likely boost the performance of our system. Examining the results in Table 9, it is clear that the ensemble once again improves performance.

Since a validation set is available, we also ran some experiments for optimizing the approach:

• F1, the best fusion by sum rule of LPQ R = 3, GIST, and LTP-u among the global approaches, obtained an average precision (ap) of 73.7;

• F2, the fusion among the local approaches, obtained an ap of 73.9;

• F1 + F2, the fusion between F1 and F2, obtained an ap of 78.9;

• Fp, the fusion by weighted sum rule of F1, F2, and the score obtained by poselet (Bourdev and Malik, 2009) (weight of poselet = 4), obtained an ap of 87.4.

We submitted our best approach as a competitor for the VOC2012 contest. The result obtained in the VOC2012 contest was 88.7%. Notice that in this paper we have not used the images used in the test set of VOC2012, since their labels are unavailable, in this work we have used a validation set. Since the VOC2012 classification dataset is the same as that used in 2011 and no additional data has been annotated, we can fairly compare our approach with both the competitors of VOC2011 and VOC2012 (http://pascallin.ecs.soton.ac.uk/challenges/VOC). Among the 26 competitors of VOC2011/VOC2012 contests, we rank in the 10th position. Except for Panasonic and the National Laboratory of Pattern Recognition, Institute of Automation Chinese Academy of Science, we obtained a performance that was similar to the other state of the art approaches. Moreover, since we simplified our approach for the competition by reducing the number of components due to restrictions in the computation time. We would expect to obtain an even better performance using the complete system described in this paper.