Dermoscopic image classification using CNN with Handcrafted features

4.1 Segmentation

Segmentation is the process of obtaining required image object which gives the location of object and boundaries like lines and curves. Before extracting the feature values, the dermoscopic images are subjected to a segmentation process. In this work, a region growing technique is used to extract the region of interest. In region growing technique initially a seed point will be taken, and the neighbouring pixels of the initial seed point are examined to determine whether the neighbouring pixels falls into that region or not. The neighbouring pixels which are having the properties similar to the seed will grow on and the segmentation of the required tumour part can be acquired. The segmented dermoscopic images are shown in Fig. 3.

Close

Fig. 3

Segmentation of Dermoscopic Image.

Export to PPT

4.2 Feature extraction

Feature extraction is the process of making a raw data into some numerical value features which can be processed by preserving the actual data. Simply an image feature is a characteristic which helps in the discrimination of one image from another image.

Various methods exist for feature extraction such as GLCM, Bag of Features, Local Binary Patters etc., (Paci et al., 2013) proposed multi local quinary patterns and multi local phase quantization with ternary coding for extracting texture features which are classified using Support Vector Machine and Gaussian Process Classifier achieving good classification accuracy with different texture datasets such as Broadatz, KTH-TIPS, OUTEX TC 00,000 dataset etc., (Tahir, 2018) proposed analysis of patterns in the images of proteins using GLCM and concluded that it is efficient for analysis of patterns in the biomedical area. (Xu and Li, 2020) used GLCM features for extraction of features for efficient detection of breast tumours.

In the present work the handcrafted features that are to be considered are Texture, Shape and colour features. From GLCM the texture features attained are autocorrelation, correlation, contrast, energy, dissimilarity, entropy, variance, homogeneity and normalized inverse difference moment. In addition to these texture features, the coefficients of Scattered Wavelet Transform obtained from the decomposed dermoscopic images are also considered. The shape features such as area, perimeter, form factor, major axis length and minor axis length is obtained from segmented dermoscopic images. Colour features such as mean, standard deviation, kurtosis, skewness, and smoothness index are calculated from each channel (RGB) of the colour histogram for color features.

4.3 Scattered wavelet transform

Wavelets techniques are an effective tool for feature extraction and an adequate data. Scattered wavelet transform proposed by (Bruna and Mallat, 2013), is capable to generate definitive features that are locally steady to small deformations. This transform builds a stable invariant representation by computing the cascade of wavelet modulus operator with filter coefficients. To construct a wavelet scattering transform of an input image (I) three successive important operations such as Convolution, Nonlinearity, and Averaging need to be performed as depicted in Fig. 4.

Close

Fig. 4

Construction of Scattered Wavelet Transform.

Export to PPT

Dilated mother wavelets with different scaling levels are the wavelet functions used in the scattering transform (Pandey et al., 2016). Morlet wavelet denoted by $\psi$ is considered as a mother wavelet. A two dimensional Image I considered to be a real number which can be represented as $I∊{\mathbb{R}}^2$ and a low pass averaging filter in the frequency domain controlled by scaling factor J can be represented by ${\varnothing }_J\left(f\right)=2^{-2J}\varnothing \left(2^{-J}f\right).$ Convolving dermoscopic image with an averaging filter results in locally affine transformation given by

Majority of the high frequency components of the image are rejected and Eq. (1) produces exactly same representation up to $2^J$ pixels. To acquire high frequency components of the image a set of constant high frequency bandpass filters can be applied to an image. Morlet filters or wavelets ${\psi }_{\lambda }\left(f\right)$ with altering scales $2^J$ and rotations $\theta$ are constituted with a set $\wedge$ , and is given as

The constant set of wavelet bank ${\wedge }_0$ can be given a

Where ${\lambda }_i=\left\{j,\theta \right\};$ with $j∊\left\{0,1,\cdots ,J-1\right\}$ and $\theta =\frac{k\pi }{\mathrm{K}},\mathrm{k}∊\left\{0,1,\cdots ,K-1\right\}$ and $w=JK$

Applying average filter, high frequency bandpass filters or wavelets over an image is done with the wavelet modulus $W(I)$ of an image $I$ given by

For the higher frequency, locally translational invariant descriptors, $\left|I\ast {\psi }_{\lambda }\left(f\right)\right|$ can be calculated by attaining the space average over the wavelet modulus coefficients,

Which are the coefficients of 1st order scattering calculated by using modulus of 2nd wavelet transform $\left|W_2\right|$ that is applied to $\left|I\ast {\psi }_{\lambda }\left(f\right)\right|$ for each ${\psi }_{\lambda }\left(f\right)∊{\wedge }_0\left(J,K\right)$ . It also gives complimentary wavelet coefficient of high frequency

Where ${\psi }_{{\lambda }^{\text{'}}}\left(f\right)∊{\wedge }_0\left(J,K\right)$ and ${\psi }_{{\lambda }^{\text{'}\text{'}}}\left(f\right)∊{\wedge }_1\left(J,K\right)$ .

Here ${\wedge }_0$ and ${\wedge }_1$ represents two different filter banks and by using a low pass filter ${\varphi }_J$ the coefficients are averaged, which defines second order scattering coefficients and can be represented as

By applying a third wavelet modulus transform, the averages of all these are computed by

$\left|W_3\right|$ to $\left|\left|I\ast {\psi }_{{\lambda }^{\text{'}}}\left(f\right)\right|\ast {\psi }_{{\lambda }^{\text{'}\text{'}}}\left(f\right)\right|$ for each ${\psi }_{{\lambda }^{\text{'}}}\left(f\right)∊$ ${\wedge }_0\left(J,K\right)$ and ${\psi }_{{\lambda }^{\text{'}\text{'}}}\left(f\right)∊{\wedge }_1\left(J,K\right)$ . By using third wavelet modulus the wavelet modulus coefficients are computed by convolving with a new set of wavelets ${\psi }_{{\lambda }^{\text{'}\text{'}\text{'}}}$ which consists of scaling and orientation parameters defined by ${\lambda }^{\text{'}\text{'}\text{'}}$ such that ${\psi }_{{\lambda }^{\text{'}\text{'}\text{'}}}∊$ ${\wedge }_3\left(J,K\right)$ where ${\wedge }_3\left(J,K\right)$ can be the new wavelet bank. By repeatedly applying this process further layers can be computed for defining a scattering wavelet transform network, which is able to extend for any order m, with the help of ${\wedge }_{m-1}$ at each step. A robust feature representation can be achieved with the help of higher order scattering coefficients.

At the m^th layer the scattering wavelet coefficients can be defined as

${\varphi }_{{\lambda }_i}\left(f\right)∊\wedge$ for I = 1,2,…,m, and $p=\left\{{\varphi }_{{\lambda }_1}\left(f\right),{\varphi }_{{\lambda }_2}\left(f\right),\cdots .,{\varphi }_{{\lambda }_m}\left(f\right)\right\}$ is an ordered set of wavelets, where $p∊{\wedge }^m$ and

Here X represents Cartesian product. During the scattering process the loss of information in one stage will be recovered in the next stage. The energy of the scattering coefficients decreases in proportion with the increasing level of scattering; hence an infinite level of decomposition is not required due to the decrease in the energy. Visualization of first order scattering disc on a dermoscopic image is shown in Fig. 5. Decomposing a dermoscopic image using scattered wavelet transform is shown in Fig. 6.

Close

Fig. 5

Visualization of first order Scattering Disk on Dermoscopic Image.

Export to PPT

Close

Fig. 6

Decomposition of Dermoscopic Image At Different Scales Using Scattered Wavelet Transform.

Export to PPT

4.4 Feature extraction using Scattered Wavelet Transform

The Statistical Texture features are extracted by applying the Scattered Wavelet Transform to the dermoscopic image along with the GLCM, colour and shape features for Melanoma as well as various Skin Lesions. The GLCM features are calculated by the mathematical relations shown in Supplementary File Table SI and the Statistical Texture features are calculated by the mathematical representations from the coefficients of different multiresolution transforms shown in Supplementary File Table SII. Supplementary File Table SIII–SVIII shows the features obtained for different skin Lesions.

4.5 Convolution neural network

Convolution Neural Networks are a kind of deep feed-forward artificial neural networks which can be able to analyse visual imagery (LeCun and Bengio, 1998). A CNN is a combination of local receptive fields, shared weights and spatial or temporal subsampling. A CNN is constructed by combining convolutional, pooling and fully connected architectures. The neural networks which are having one or more convolutional layers are said to be convolutional neural networks. The convolution layers in the CNN consist of filters with a capability of learning.

A convolution layer processes a 2-dimensional convolution on the inputs. CNN perform feature extraction and classification. Feature extraction involves three phases, Convolution, Activation and Pooling. The features from the raw images are extracted automatically which yields better accuracy. The convolution and the pooling layers in the CNN makes the model computational efficient. CNN has the flexibility to handle large data sets which makes it superior to other methods. According to the size of the input, each and every filter is having a small equal height, width and depth. Convolution can be done by applying appropriate filters. Convolution is a mathematical operation which produces a third function (z) from two functions x and y.

CNN uses a hierarchical model to get a fully connected layer, in this all the neurons are connected to each other to process the output. A simple CNN model consists of two hidden layers (convolution layers). To create feature maps CNN uses various kinds of kernels to convolve the input image which results in the output matrix of a convolution layer.

Where the input image I is convolved with kernel ${\mathrm{K}}_{\mathrm{m},\mathrm{j}}$ . The bias ${\mathrm{b}}_{\mathrm{j}}$ is added to each element obtained to the sum of all convoluted matrices. The size of the input image I should be increased by padding with zeros to maintain the output image same as that of the input image. The output of the convolution consists of both negative and positive values. The negative values are removed using Activation or Nonlinear functions such as Hyperbolic, Softmax, Rectified Linear Unit (ReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Unit (SELU) which are given in the Eqs. (12)–(16) respectively.

The convolution layer is followed by pooling layer which is used to reduce the size of the feature maps and parameters of the network that increases computational speed. Max pooling is used widely to select the maximum of the value. The hyper parameters of pooling layer are filter size, stride and max pooling. Regularization methods need to be used to introduce dropouts not only on the training data but also on the new entries to reduce the problem of over fitting. Loss function gives the difference between the output and target variable helps in the connection of various inputs to the next layer independently. The basic architecture of a CNN is shown in Fig. 7.

Close

Fig. 7

Basic Architecture of CNN.

Export to PPT

4.6 Convolution neural network with Handcrafted features

The architecture of CNN considered in this work is shown in Fig. 8. The proposed architecture consists of two convolution layers followed by two max pooling layers and dropouts to randomly discard the nodes during the training phase which eventually reduces the overfitting and improves generalization error. Next to dropout, the 2-Dimensional features are flattened to 1-Dimensional array and given to fully connected layer consisting of 128 neurons. The output layer consists of one neuron per class.

Close

Fig. 8

Architecture of CNN with Handcrafted features.

Export to PPT

In the training process of the proposed architecture Skin Lesions are resized to 64X64 and given as input to the CNN with RGB channels. The filter should be 3-dimensional if the input image is a color image. Filters can occur in various sizes. The job of a filter is to identify an image specific feature. When a trained filter gets multiplied with a portion of the image, the area of the image containing that feature will produce very high values in the feature maps. All the values in the filter start with random initialization, just like the weights in the neural networks. First few convolution layers consist of filters that will identify common features such as lines, edges and curves which are part of Skin Lesion. Subsequent layers contain filters that can identify complex and image specific features.

The first convolution layer is assigned with 32 filters with a 5X5 kernel, producing 32 channels with a size of 60X60. Pooling can be performed after activation to reduce the dimensionality of the dataset. It reduces the height and width but keeps the depth. After applying the first convolution on the input images a maxpooling layer with a size of 2X2 is applied on the convolved output to select the maximum elements with prominent features using a stride of 1. First max pool layer consists of a size of 2X2 which is applied on first convolution layer output gives 32 channels with a size of 30X30. The second convolution layer is assigned with 64 filters with a 5X5 kernel, size which produces 64 channels with a size of 26X26. On applying the maxpool to the second convolved output, it gives 64 channels with a size of 13X13. Batch normalization is performed at the output of each maxpool layer to standardize the inputs. An activation function Rectified Linear Unit (ReLu) is used in this architecture to introduce nonlinearities to the network.

A dropout of 0.25 is applied before the flattening of 2-Dimensional to 1-Dimensional to reduce the complexity and increase the computation time by discarding some nodes. The discarded nodes weights and bias values are not considered during the training phase. The 1-Dimensional features are given to the fully connected layer with 128 neurons. In addition to the 1-Dimensional features which are obtained by the convolution process, hand-crafted features which are acquired by using Scattered Wavelet are also given as additional features to the fully connected layer. These additional features will be concatenated in the fully connected layer with the features obtained by the convolution layer, leads to a better accuracy when compared with the single input convolution neural network model.

The proposed architecture was trained with 80% data and tested with 20% data. The number of images used in the training and test data for Melanoma recognition and different skin lesion classification are listed in Supplementary File Table SIX. Sample Images given to CNN are shown in Fig. 9.

Close

Fig. 9

Sample input images to CNN.

Export to PPT