Diabetic Retinopathy (DR) is a progressive and vision-threatening complication of diabetes,
posing significant challenges for early detection and timely intervention. Automated systems
for DR severity grading has shown promising results; however, several challenges, including
data scarcity, class imbalance, variability in fundus image quality, and the detection of
subtle abnormalities in early stages. This dissertation presents a comprehensive deep
learning framework to address these challenges through novel contributions that enhance
classification accuracy, generalization, and clinical relevance. The first contribution
introduces an ensemble framework that integrates the best performing pre-trained deep
learning models and enhances feature extraction using soft attention mechanism. Transfer
learning is leveraged to extract robust features from fundus images, and a soft attention
mechanism is incorporated to focus on critical fundus regions, ensuring accurate detection
of DR-related abnormalities. This approach is particularly beneficial for small datasets
like IDRiD, where training deep learning models from scratch would lead to overfitting.
Additionally, by combining multiple pre-trained models, the ensemble approach improves
generalization across datasets, ensuring that the model performs well on diverse populations,
including DDR, APTOS, and EyePACS. The second contribution proposed a hybrid
system integrating Discrete Wavelet Transform (DWT) with Convolutional Neural Networks
(CNNs) for multi-resolution feature extraction. This approach captured both fine and
coarse details in fundus images, allowing CNNs to extract meaningful features at different
resolutions. Furthermore, Contrast Limited Adaptive Histogram Equalization (CLAHE)
is used as a pre-processing step to enhance the contrast of fundus images, improving
the visibility of the clinical features. Since DR datasets suffer from class imbalance,
oversampling techniques are applied to ensure balanced training, preventing the model
from being biased toward majority classes. The third contribution focuses on improving
the detection of mild-stage DR, which is particularly challenging due to the subtle nature
of the microaneurysms. The proposed approach pre-processes fundus images using
resizing, augmentation, and oversampling to ensure a diverse and balanced dataset for
training. A hybrid feature extraction strategy has been proposed where Local Binary
Pattern (LBP) is applied to enhance texture features, while CLAHE-enhanced blood
vessel structures are extracted using CNNs. By fusing these complementary features,
the model improves sensitivity to mild DR cases while maintaining efficiency with fewer
parameters. The framework is validated across multiple datasets to ensure its robustness. The fourth contribution investigates the impact of AI-generated images on DR grading
and introduces a custom loss function to handle dataset imbalance. Balanced Generative
Adversial Network (GAN) and Attention enhaced Balanced GAN are used to generate
synthetic high-quality fundus images. While geometrical transformations proved effective
in improving model performance, GAN-based augmentation fell short due to limitations
in synthetic image realism. Additionally, a Performance Aware Weighted Loss (PAWL)
function is designed to mitigate class imbalance, ensuring equitable learning across all
DR severity levels. This approach significantly improves classification accuracy, across
all DR severity classes, enhancing the model&rsquos real-world applicability. The findings of
this dissertation hold significant implications for the field of automated DR detection.
By addressing class imbalance, leveraging multi-resolution and hybrid feature extraction
techniques, and integrating novel loss functions, this research advances the capabilities
of AI-driven diagnostic systems. Future work may explore explainable AI methods,
semi-supervised learning, and real-time deployment on edge devices to further enhance
clinical applicability and trustworthiness.