Enhanced skin cancer classification using a combination of bag of words and deep Q-network features with ReliefF feature selection

1. Introduction

One of the cancers that has increased the most in recent decades is skin cancer. Discriminative features are necessary to differentiate between unusual alterations in the color, texture, or structure of the skin for precise detection and classification. Cells that grow out of control, divide quickly into one part of the body, invade other human parts, and spread throughout the body are the precursors of cancer (Bibi et al., 2023). The ultraviolet (UV) rays from the sun have the capacity to damage skin cell DNA. Additionally, abnormal enlargements of the human body can potentially result in skin cancer.

The third most prevalent of skin cancers, melanoma, is regarded as extremely deadly. The condition known as melanoma occurs when cells that produce skin pigment malfunction and change color. In order to achieve successful treatment, it is necessary to increase the chances of early detection and treatment (Dillshad et al., 2023). The condition is caused by an accumulation of melanin granules and their distribution to the top layer of the skin (Dugonik et al., 2025). Therefore, developing a computer-assisted diagnostic (CAD) system for classifying skin lesions is essential (Khattar & Kaur, 2022). Additionally, early detection of skin cancer is greatly aided by artificial intelligence (AI), machine learning (ML), and deep learning (DL) (Ahmad et al., 2023).

Artificial neural networks (ANNs), a cutting-edge method in the field of AI, have been increasingly popular recently in several fields, such as computer vision, digital image processing, and image classification methods (Almasoud et al., 2022; Sedik et al., 2025; Tounsi et al., 2025). ANNs have shown remarkable efficacy in solving a variety of complex problems, including medical imaging, object detection, and image classification, which benefits patients and healthcare systems. This study presents a hybrid feature extraction method that combines the Bag of Words (BoW) feature function and Deep Q-Network (DQN) to improve the accuracy of multi-class skin cancer classification. The BoW method captures essential local image features, while the Deep Q-Network enhances feature representation by extracting deep hierarchical features from skin images. Additionally, the Oriented FAST and Rotated BRIEF (ORB) technique is used to enhance feature reliability. The extracted features are then fed into a Narrow Neural Network (NNNs) model for precise classification of different skin cancer types. The primary goal of this approach is to improve classification performance by leveraging both handcrafted and deep learning-based features.

By integrating a hybrid feature extraction method, this study aims to enhance the efficiency and accuracy of skin cancer classification, ultimately assisting dermatologists and healthcare professionals in early diagnosis and treatment planning. The following are the main contributions:

• a.

  Introduced a BoWs feature function to effectively capture key local image features.

• b.

  Integrated a DQN feature extraction to obtain deep features from skin images.

• c.

  Employed the ORB technique to enhance the reliability and effectiveness of extracted features.

2. Related Works

An overview of earlier research on the diagnosis of skin cancer is given in this section. The use of ANNs for the diagnosis and classification of skin cancer has been the subject of substantial research in recent years by a large number of researchers (Shah et al., 2023). (SM et al., 2023) proposed a skin cancer categorization using a DL method. The suggested model’s performance was assessed using the ISIC-2019 and ISIC-2020 datasets. The deep convolutional neural networks (DCNN) model obtained the highest area under the curve (AUC) score of 96.81%. A healthcare system, proposed by (Hoang et al., 2022), utilizes first-order cumulative moment entropy-based weighting and first-order cumulative moment (EW-FCM) and weighting-based entropy for segmentation, along with wide-ShuffleNet for classifying skin cancer. While (Jasil S. et al. 2023) presented a novel method of classifying skin cancer by fusing residual network designs with Densenet. Their approach achieved a 95% accuracy rate. Their study has drawbacks, including the computational cost, potential for overfitting, and bias in the dataset. (Keerthana et al., 2023) utilized MobileNet and DenseNet-201 transfer learning models for feature extraction. An accuracy of 88.02% was achieved by their method when tested on the ISBI 2016 dataset. The main limitation of this study, however, was that Support vector machines (SVMs) perform well on small datasets but scale poorly with large feature sets, making them useless for real-time classification in clinical settings. In a study by (Xu et al., 2020), cancer detection was addressed within a two-step procedure, which first involved applying a median filter and then utilizing an improved convolutional neural network (CNN) for image segmentation. The performance achieved a 95% precision rate. However, the main limitation of this study was the soft computing methods, particularly ANNs and fuzzy logic. Additionally, (Popescu et al., 2022) proposed a DL based skin lesion classification system that was trained on the HAM10000 dataset. The primary drawback of this study was the high computational complexity and resource demands. Moreover, (Viknesh et al., 2023), suggested two techniques to detect skin cancer. The drawbacks of this approach were its sensitivity to noise and image quality variations. In their tests with the balanced ISIC-2019 dataset, (Zhao et al., 2021) classified dermoscopy images using StyleGAN and DenseNet201, with a classification accuracy of 93.64%. The complexity and diversity of real dermoscopy images are not adequately represented by StyleGAN-generated images. (Houssein et al., 2024) proposed the DCNN model to extract intricate features from skin lesion images, which achieved high accuracy rates of 98.5% for the HAM10000 dataset and 97.1% for the ISIC-2019 dataset. This approach has drawbacks, including high computational costs and resource requirements. Skin cancer classification is being significantly impacted by significant developments in human-AI collaboration and reinforcement learning. A class-weighted reinforcement learning framework was presented by the authors of (Mayanja et al., 2025); it particularly resolves class imbalances during training and shows notable improvements in diagnosis accuracy. By using a DQN for feature extraction rather than classification and ReliefF for feature selection, our study, in contrast, makes indirect use of reinforcement learning. When it comes to clinical AI systems, (Tschandl et al., 2020) demonstrated that human-computer collaboration can achieve or even surpass dermatologist-level performance, emphasizing interpretability and dependability. By enhancing transparency, our feature rating and Bag-of-Words descriptors fit well with this paradigm. Through reinforcement learning, (Barata et al., 2023) proposed advanced decision support by integrating uncertainty estimation into clinical workflows. Our approach operates at the feature level, providing a new complementary perspective on the use of RL in medical imaging, whereas their model was more focused on dynamic policy learning. By simulating global context and long-range dependencies, Vision Transformers have shown impressive performance on skin lesion classification challenges. For instance, ViT-based models that were trained on ISIC datasets have outperformed conventional CNNs in terms of accuracy, with rates above 96%. Swin Transformers are ideal for high-resolution dermoscopic images because they introduce shifting windows and hierarchical feature maps. Research indicates that Swin Transformer models perform better in lesion segmentation and classification than ResNet and DenseNet. CNNs and transformers have been combined in recent studies to take advantage of both global semantic information and local texture. These models frequently employ transformer blocks for contextual refining after CNNs for initial feature extraction.

3. Materials and Methods

This study proposed hybrid features of the bag of words+deep q-network (BoWs+DQN) skin cancer classification model from skin images to increase the efficacy and accuracy of skin cancer classification. The primary stages of the proposed flowchart have been illustrated in Fig. 1.

Close

Fig. 1.

Flowchart of the proposed hybrid BoWs–DQN skin cancer classification model.

Export to PPT

The following are the main stages of the proposed model:

• 1.

  Data Collection: Collect skin lesion images from datasets

• 2.

  Preprocessing: (Clean the images):

• -

  Resize images.

• -

  Normalize pixel values.

• -

  Denoise/Augmentation.

• -

  Hair removal from the lesion area.

• 3.

  Feature Extraction: (Extract important patterns):

• -

  Texture features (gray-level co-occurrence matrix) GLCM, Local Binary Patterns (LBP)).

• -

  Shape features (lesion borders).

• -

  Color features.

• -

  Deep features via convolution neural network (CNN) models like Residual Network (ResNet), visual geometry group (VGG).

• 4.

  Feature Selection: (Select best features to reduce noise):

• -

  Use principal component analysis (PCA), least absolute shrinkage and selection operator (LASSO), or mutual information

• -

  Keep only discriminative features.

• 5.

  Classification: (Predict the skin lesion type):

• -

  Traditional ML: Tree, Logistic Regression, Naive Bayes, SVM, K-Nearest Neighbors (KNN), Neural Network

• -

  Deep Learning: CNNs

• 6.

  Skin Cancer Classes

• -

  Multi-class: melanoma, nevus, keratosis, etc.

3.1 Skin image dataset

This study used datasets of dermatoscopy images to assess the proposed method. The international skin imaging collaboration (ISIC) provided the ISIC-2019 dataset (Houssein et al., 2022; Skin Lesion Images for Melanoma Classification: ISIC 2019 challenge, 2019), which consists of dermatoscopy images used to categorize skin lesions into eight distinct diagnostic groups. The dataset includes 25,331 images, each of which corresponds to a specific group of symptoms indicating a type of skin injury. The ISIC-2019 dataset is considered one of the largest freely available datasets for classifying skin injuries. The reason ISIC-2019 was chosen for this work is that, unlike earlier versions of ISIC, which concentrated on binary classification (e.g., benign versus malignant), ISIC-2019 supports multiclass classification, which is in line with the aim of this study in differentiating between various types of skin cancer. Fig. 2 illustrates the different classes present in the ISIC-2019 dataset. The dataset ISIC-2019 has a more balanced class distribution and consistent image acquisition protocols to reduce bias and increase model generalization. Furthermore, the dataset has been commonly used in most recent benchmarking studies, allowing a fair and uniform comparison with existing methods. The diversity of lesion types and expert-verified annotations render ISIC-2019 as an appropriate and dependable resource for evaluating the proposed hybrid BoWs-DQN model.

Close

Fig. 2.

ISIC-2019 Dataset samples from eight various classes.

Export to PPT

3.2 Skin image preprocessing

Medical image problems are often characterized by low contrast, noise, blur, artifacts, and unrealistic colors. To achieve extremely precise results, the proposed model integrates the following preprocessing methods: adaptive histogram equalization, median filtering, sharp filtering, and image reshaping. Adaptive histogram equalization (AHE) is used to bring the input image’s low contrast into the normal pixel intensity value range. This method makes all images from the dataset suitable and gets more details for the feature extraction stage. By using noise reduction and a sharp filter after applying adaptive histogram equalization to correct and make pixel values appear with more details. In addition, all images in the dataset are adjusted to a uniform dimension of 200 pixels × 200 pixels, as shown in Fig. 3. In addition to preserving uniformity in image dimensions, this resizing maximizes memory usage and computational effectiveness. For feature extraction to be smooth and capture pertinent patterns and information across the model. Standardizing image sizes is necessary because irregular sizes can result in errors and mismatched dimensions during training.

Close

Fig. 3.

Preprocessed sample images demonstrating resizing, noise removal, and contrast enhancement steps.

Export to PPT

3.3 Features extraction

The feature extraction algorithm’s objective is to extract the most relevant and usable feature set from the preprocessed skin images. These features include enhanced BoWs features and deep learning features. Both low-level details (edges, colors) and high-level patterns (tumor structures, lesion boundaries) can be captured by using a hybrid feature extraction method. Combining the BoWs feature function with DQN features for skin cancer classification results in improved feature representation. The BoWs effectively capture localized texture, color, and structural details, whereas the DQN extracts deeper contextual and morphological representations of the entire lesion. When combined, these complementary features form a richer and more discriminative feature space, leading to a more robust and accurate classifier performance.

3.3.1 Bag of words (BoWs)

The visual components of skin cancer images typically include several key features that help in diagnosis and classification. These components can be assessed manually by dermatologists or extracted using computational techniques such as texture analysis and deep learning. The main visual components include the color features, texture features, shape and border characteristics, and structural patterns. The BoVW method reduces an image from raw pixel data or highly complex features to a histogram of visual word occurrences, BoWs. This is because, especially in large datasets, it simplifies the data by condensing important image features. The method uses histogram-based image representation, visual vocabulary construction, and key feature extraction. The SURF features are extracted from the feature selection point locations using the grid method, followed by block width [32 64 96 128] and grid step [8 8]. Using clustering features to create a visual vocabulary of 500 words. Algorithm 1 provides an illustration of the study’s pseudo-algorithm.

Algorithm 1. Pseudo-algorithm of skin image detection using BoWs

• 1.

  Input: skin images.

• 2.

  Output: Descriptors and keypoints. For each image in the dataset:

  • Detect keypoints using (SURF).

  • Extract descriptors around each keypoint (SURF descriptors).

• 3.

  Building the Visual Vocabulary:

  • Aggregate all descriptors from the dataset into a single pool.

  • Use k-means clustering on the pooled descriptors to create K clusters.

  • The cluster centroids represent the visual words in the vocabulary.

• 4.

  Feature Vector Representation:

  • For each image in the dataset:

  • Assign each descriptor to the nearest cluster.

  • Create a histogram of visual word occurrences (Bag-of-Visual-Words representation) based on skin image detection and feature extraction process for non-expert readers.

The number of visual words that the bag of features object created is reflected in the length of the histogram. An image’s feature vector is created from the histogram, as shown in Fig. 4.

Close

Fig. 4.

Feature vector from visual word index.

Export to PPT

Close

Fig. 4.

Feature vector from visual word index.

Export to PPT

3.3.2 Deep Q-network (DQN)

To get around the limitations of traditional Q-Learning in large or continuous state spaces, deep learning is utilized to extract high-level features using a deep network technique called DQN. DQN uses NNs to approximate the Q-values rather than keeping Q-values for each state-action pair. The state serves as the network’s input, and the Q-values for every action that could be taken make up its output. DQN uses the Bellman equation evolving from the Q-Learning algorithm to update Q-values.

(1)

$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha (r+\gamma ma{x}_{a\text{‘}}Q\left(s^{\prime },a^{\prime };{\theta }^{-}\right)-Q\left(s,a;\theta \right)$

Where:

s is the current state,

a is the action taken,

r is the reward received,

s’ is the next state,

a’ is the next taken,

θ are the weights of the main Q-network,

θ^- are the weights of the target Q-network,

$ma{x}_{a\text{'}}Q\left(s^{\prime },a^{\prime }\right)$is the maximum Q-value for the next state.

Deep Q-Networks, or DQNs, provide extensive feature representations with potentially transferable learnt representations, which is the rationale behind their use as feature extraction for image classification. To extract features, each image is first pre-processed (resized, normalized, etc.) before being run through the DQN’s convolutional layers. To estimate the Q-values for every potential action, these features are then flattened and put into fully connected layers. The network is trained to maximize cumulative reward, and the agent chooses actions based on the highest Q-value. To calculate the target Q-values, DQN incorporates a second NN known as the target network. Using a sequence of convolution and pooling layers, DQN processes the given skin images. The suggested DQN structure for obtaining deep features from skin image data has been displayed in Fig. 5. As a result, each image has two features extracted: one by DQN and one by BoWs.

Close

Fig. 5.

Architecture of the proposed DQN feature extraction.

Export to PPT

To construct a unified feature representation from the BoW and DQN outputs, the following fusion method (BoW + DQN) was employed step-by-step:

• 1.

  Feature Extraction:

  • BoW features: Extracted as histogram vectors representing visual word occurrences (dimension: 500).

  • DQN features: Extracted from the final fully connected layer of the DQN architecture (dimension: 1024).

• 2.

  Normalization:

  • Both feature sets were normalized using z-score normalization to ensure comparable scales and prevent dominance of higher-magnitude features.

• 3.

  Concatenation:

  • The normalized BoW and DQN vectors were concatenated into a single feature vector of dimension 1524.

  • No weighting was applied, as ReliefF feature selection was used post-fusion to rank and filter the most discriminative features.

• 4.

  Dimensionality Reduction:

  • ReliefF was applied to the fused vector to reduce redundancy and retain only the most relevant features for classification.

• 5.

  Classifier Input:

  • The final selected feature vector was fed into various classifiers, with the NNN yielding the highest performance.

4. Results and Discussion

The deep features are extracted using MATLAB 2024b from skin images after resizing skin lesion images into 200x200 pixels prior to implementation of the DQN. The proposed model was tested using MATLAB 2024b, which was installed on a workstation with an Intel Core i7 (10th Gen) CPU, an NVIDIA RTX 2070 GPU, and 8 GB of VRAM. It was running Windows 11 and had 16 GB of RAM.

The ISIC-2019 dataset was split into 80% training, 10% validation, and 10% testing subsets at the patient level to guarantee equal evaluation and prevent data leaks. To ensure rigorous separation, each subset’s preprocessing, data augmentation, and feature extraction processes were carried out independently.

In this study, three validations are used to evaluate the performance of a proposed skin cancer classification. The process of cross-validation (CV) divides a dataset into 5-fold cross-validation. This process is repeated multiple times, and the final performance metric is averaged.

The ISIC-2019 dataset was split into five folds using the stratified k-fold cross-validation technique, which attempts to maintain data integrity and prevent data leaking. Four of the folds served as training, and each one served as validation once. Preprocessing was carried out after the split to ensure that the training would not be impacted by any information from the hold-out set. BoWs and DQN features were extracted separately for every fold, while ReliefF features were only selected within the corresponding training subset to prevent bias. Feature extraction and selection were done independently for each fold. The same preprocessing, feature extraction (BoWs + DQN), ReliefF selection, and classification pipeline were used for each fold’s model during independent training. Table 1 displays the results for each fold.

Table 2 lists the hyperparameters that were utilized in the DQN model. The proposed DQN network exhibits superior performance in extracting deep features from skin images, as illustrated in Fig. 6, the training and loss progress plot of the DQN.

Close

Fig. 6.

Training plot of the proposed DQN.

Export to PPT

Five feature selection algorithms: MRMR, Chi2, ReliefF, ANOVA, and Kruskal-Wallis are compared in the classification section to show how well the suggested ReliefF feature selection algorithms identify distinguishing features from skin images. Using the same hyperparameters under the NN model, Table 3, illustrates which of these algorithms will perform the best with the proposed method. When compared to another algorithm after the trained features were set, the ReliefF algorithm finally obtained the highest accuracy score.

Table 4 presents the validation accuracy of different classifiers using three feature configurations: BoWs-only, DQN-only, and the hybrid combination (BoWs+DQN). Among the single-feature models, BoWs-based classifiers generally achieved higher accuracy than DQN-only models, indicating that handcrafted texture and shape descriptors remain informative for this dataset. However, DQN features alone performed less effectively, particularly with traditional classifiers such as Tree (29.8%) and Kernel (32.6%), suggesting that deep features extracted through DQN require complementary information for optimal discrimination. When combining BoWs and DQN features, a significant performance boost was observed across all classifiers. The highest validation accuracy (99.6%) was achieved by the Narrow Neural Network, followed closely by SVM (96.7%) and Kernel-based models (92.4%). This demonstrates that the fusion of handcrafted and deep-learned features effectively captures both local texture patterns and high-level semantic representations, resulting in superior classification performance. These findings confirm the effectiveness and generalization capability of the hybrid BoWs+DQN framework, highlighting its ability to leverage the strengths of both feature domains. The consistent improvement across all classifiers further validates the robustness and adaptability of the proposed approach. Fig. 7 and Table 5 illustrated the performance results of the proposed method on the ISIC-2019 dataset through a confusion matrix.

Close

Fig. 7.

Confusion matrix of the proposed NNN model on the ISIC-2019 dataset.

Export to PPT

The reliability of the model’s classification results is further supported by the low standard deviations and tight confidence intervals (CI), usually within 2% that show the model’s strong generalization and minimal variability across validation folds. Multi-source dermatoscopic images of common skin lesions known as the HAM10000 dataset was used to expand evaluation beyond the ISIC-2019 database (The HAM10000 dataset, 2018). The results have been shown in Table 6 with the confusion matrix in Fig. 8.

Close

Fig. 8.

Confusion matrix of the proposed Narrow Neural Network (NNN) model on the HAM10000 dataset.

Export to PPT

The results suggest that the proposed approach achieved a balanced and highly discriminative performance across diverse lesion categories. High sensitivity, specificity, and AUC values showed that the model successfully reduces false positives and false negatives, which makes it ideal for automated dermatological diagnostics and clinical decision support.

To evaluate the impact of the proposed method, several transfer learning algorithms were employed to extract features from the same preprocessed images. The resulting features were refined using the standard ReliefF selection method and classified with an NNN. The comparative classification results have been presented in Table 7. The results show that the proposed method is superior in terms of accuracy and robustness, which makes it a viable option for clinical decision support applications and real-world skin lesion categorization.

Based on the stratified 5-fold cross-validation findings, statistical significance tests were performed to determine whether the suggested hybrid BoWs+DQN feature extraction approach performs noticeably better than baseline models (e.g., BoWs-only, DQN-only, residual network with 50 layers (ResNet50), and visual geometry group with 16 layers (VGG16)).

• Test Setup

• Metric Used: Accuracy across folds

• Comparison Groups:

  • Proposed Method (BoWs+DQN)

  • Baseline CNNs (ResNet50, VGG16)

  • Individual Feature Extractors (BoWs-only, DQN-only)

• Tests Applied:

  • Paired t-test: Assumes normal distribution of fold-wise accuracy differences

  • Wilcoxon signed-rank test: Non-parametric alternative for robustness

Table 8 presents the statistical validation of model improvements using both paired t-test and Wilcoxon signed-rank test. The results demonstrate that the proposed BoWs+DQN model consistently outperforms all comparative methods, including BoWs-only, DQN-only, ResNet50, and VGG16. The mean accuracy differences range between +3.2% and +4.8%, indicating a notable enhancement in classification performance when the BoWs feature descriptors are integrated with DQN-based learning. Furthermore, the p-values obtained from both the paired t-test and the non-parametric Wilcoxon test are all below 0.01, confirming that these improvements are statistically significant at the 1% significance level. These findings validate that the proposed hybrid BoWs+DQN approach provides a more robust feature representation and generalization capability compared to traditional deep and handcrafted feature-based models. The statistical significance of the results supports the reliability and reproducibility of the observed performance gains.

5. Limitations

The DQN training and feature fusion are computationally intensive, potentially limiting their deployment in low-resource environments. Another limitation lies in the dependence on hand-crafted BoW features within the hybrid framework, which may reduce generalization to unseen data. Ultimately, further validation through cross-dataset experiments and DQN hyperparameter optimization is needed to demonstrate the robustness and scalability of the proposed approach.

6. Conclusions

Skin cancer is one kind of skin lesion that is deemed malignant and needs to be detected and treated as soon as possible. With their ability to process and interpret complex visual input, artificial intelligence (AI) technologies are making significant strides in dermatological diagnostics and are suitable for tasks such as image classification. This study introduced a novel technique for automatically identifying different skin types in images by using the ReliefF feature selection on a combination of DQN deep features and BoWs features. In this study, the BoWs model effectively extracted the texture patterns by encoding visual features into a dictionary of key patterns, while the DQN model extracted the deep the complex patterns in skin images. According to the results, the ISIC-2019 dataset’s detection accuracy increased to 99.6% when hybrid features were used. With the ReliefF feature selection, the results showed the advantages of the suggested feature extraction model. Future research is focused on enhancing the skin cancer classification system through expanding datasets, enhancing pre-processing methods, and utilizing optimization algorithms for hyperparameter tuning. The proposed approach might also be further tested to address other issues with medical imaging, like lighting, artifacts, and variations in image quality.