ViT-BT: Improving MRI Brain Tumor Classification Using Vision Transformer with Transfer Learning
Article Sidebar
Main Article Content
Abstract
This paper presents a Vision Transformer designed for classifying brain tumors (ViT-BT), offering a novel methodology to enhance the classification of brain tumor MRI scans through transfer learning with Vision Transformers. Although traditional Convolutional Neural Networks (CNNs) have demonstrated significant capabilities in medical imaging, they often need help to grasp the global contextual information within images. To address this limitation, we utilize Vision Transformers, which excel at capturing long-range dependencies due to their self-attention mechanism. In the case of ViT-BT, the Vision Transformer model undergoes pre-training followed by fine-tuning on specific MRI brain tumor datasets, thereby improving its capability to classify various brain tumor types. Experimental results indicate that ViT-BT outperforms other CNN-based methods, delivering superior accuracy and resilience. Evaluations were performed using the BraTS 2023 dataset, comprising multi-modalMRI images of brain tumors, including T1-weighted, T2-weighted, T1CE, and Flair sequences. The ViT-BT model showcased remarkable performance, achieving precision, recall, F1-score, and accuracy rates of 97%, 99%, 99.41%, and 98.17%, respectively. This advancement is anticipated to significantly enhance diagnostic accuracy in clinical settings, ultimately leading to improved patient outcomes. The research underscores the potential of transfer learning with Vision Transformers in medical imaging as a promising avenue for future exploration across various medical domains.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
M. Al-ordain, A. Khan, Sliman A., et al. “Combining the transformer and convolution for efficient brain tumor classification using MRI images.” Applied Science, MDPI, 2023. https://doi.org/10.3390/app13063680
P., M.; Anbumani, G.; Theivendren, P.; Gopal, M. “An Overview of Brain Tumor. In Brain Tumors”, IntechOpen: London, UK, 2022.
L. H. Enjun Z. Long Chen at el. “A transformer-based generative adversarial network for brain tumor synthetics”. Front NeuroSci, 30 Nov. 2022, Volume 16, 2022
Ostrom, Q.T.; Patil, N.; Cioffi, G.; Waite, K.; Kruchko, C.; Barnholtz-Sloan, J.S., “CBTRUS Statistical Report: Primary Brain and Other Central Nervous System Tumors Diagnosed in the United States in 2013–2017”, Neuro Oncol. 2020.
Park, J.; Y.G., “Brain Tumor Rehabilitation: Symptoms, Complications, and Treatment Strategy”, Brain Neurorehabilit. 2022.
Z.li, Y. Cong, Xiu Chen, et al. “Vision transformer-based weakly supervised histopathological image analysis of primary brain tumors,” Science volume 26, issue 1, 2023.
Bosman, F.T., “Integrative Molecular Tumor Classification: A Pathologist’s View”, In Encyclopedia of Cancer, 3rd ed.; Boffetta, P., Hainaut, P., Eds.; Academic Press: Oxford, UK, 2019; pp. 279–285.
Deng, J.; Hua, L.; Bian, L.; Chen, H.; Chen, L.; Cheng, H.; Dou, C.; Geng, D.; Hong, T.; Ji, H.; et al., “Molecular diagnosis and treatment of meningiomas: An expert consensus”, Chin. Med. J. 2022, 135, 1894–1912.
La Rosa, S.; Uccella, S., “Pituitary Tumors: Pathology and Genetics. In Encyclopedia of Cancer”, 3rd ed.; Boffetta, P., Hainaut, P., Eds.; Academic Press: Oxford, UK, 2019; pp. 241–256.
Zhang, L.; Liu, Y.; Huang, H.; Xie, H.; Zhang, B.; Xia, W.; Guo, B., “Multifunctional nano theranostics for near-infrared optical imaging-guided treatment of brain tumors,” Adv. Drug Deliv. Rev. 2022.
Brindle, K.M.; Izquierdo-García, J.; Lewis, D.; Mair, R.; Wright, A.J. Brain tumor imaging. J. Clin. Oncol. 2017.
Jose B., Kaiser K., Daniel S., et al. “Deep convolutional neural networks for brain image analyses on magnetic resonance imaging: a review,” Volume 95, April, pages 64-81, Science Direct, Elsevier, 2019.
A. Arkinya, F. Zaccayn, James, et al. “Brain tumor diagnosis using medical learning, convolutional neural networks, capsule neural networks, and vision transformers, applied to MRI: a survey,” Journal of Imaging, volume 8, issue 8, 2022
Kaiming He, Xiangyu Z., Shaoqing R., Jian S., “Deep residual learning for image recognition. Computer vision and pattern recognition”, doi.org/10.48550/arXiv.1512.03385, 2015.
Philip M., Harsh S. M., “Transfer learning with convolutional neural networks for classification of abdominal ultrasound images”, National Library of Medicine, 30 (12), 2017.
Bieza, A.; Krumina, G., “The value of magnetic resonance spectroscopy and diffusion tensor imaging in the characterization of gliomas growth patterns and treatment efficiency,” J. Biomed. Sci. Eng. 2013, 6, 518–526. 2017.
Abd-Ellah, M.K.; Awad, A.I.; Khalaf, A.A.; Hamed, H.F., “A review on brain tumor diagnosis from MRI images: Practical implications, key achievements, and lessons learned,” Magn. Reson. Imaging 2019.
Hemanth, G.; Janardhan, M.; Sujihelen, L., “Design and implementing brain tumor detection using machine learning approach”, In Proceedings of the 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, 23–25 April 2019.
Salmon, E.; Ir, C.; Hustinx, R., “Pitfalls and Limitations of PET/CT in Brain Imaging,” Seminars in Nuclear Medicine; Elsevier: Amsterdam, The Netherlands, 2015; pp. 541–551.
Abdullah A. Asiri L., Ahmad Shaf, Tariq Ali, et al.;” Exploring the Power of Deep Learning: Fine-Tuned Vision Transformer for Accurate and Efficient Brain Tumor Detection in MRI Scans”, MDPI, Diagnostics 2023, 13, 2094.
Swati, Z.N.K.; Zhao, Q.; Kabir, M.; Ali, F.; Ali, Z.; Ahmed, S.; Lu, J., “Brain tumor classification for MR images using transfer learning and fine-tuning”, Comput. Med. Imaging Graph 2019, 75, 34–46.
AlBadawy, E.A.; Saha, A.; Mazurowski, M.A., “Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing”, Med. Phys. 2018, 45, 1150–1158.
Hongwei Bran Li, Gian Marco Conte, Syed Muhammad Anwar, et al.; “The Brain Tumor Segmentation (BraTS) Challenge 2023: Brain MR Image Synthesis for Tumor Segmentation (BraSyn)”, arXiv:2305.0901August 23 2023
Thaha, M.M.; Kumar, K.P.M.; Murugan, B.S.; Dhanasekeran, S.; Vijayakarthick, P.; Selvi, A.S., “Brain tumor segmentation using convolutional neural networks in MRI images”, J. Med. Syst. 2019, 43, 294.
Havaei, M.; Davy, A.; Warde-Farley, D.; Biard, A.; Courville, A.; Bengio, Y.; Pal, C.; Jodoin, P.-M.; Larochelle, H., “Brain tumor segmentation with deep neural networks,” Med. Image Anal. 2017, 35, 18–31.
Sharif, M.I.; Li, J.; Amin, J.; Sharif, A., “An improved framework for brain tumor analysis using MRI based on YOLOv2 and convolutional neural network”, Complex Intell. Syst. 2021, 7, 2023–2036.
Zein Eldin, R.A.; Karar, M.; Coburger, J.; Wirtz, C.; Burgert, O. DeepSeg: “Deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images”, Int. J. Comput. Assist. Radiol. Surg. 2020, 15, 909–920.
Hatami Zadeh, A.; Nath, V.; Tang, Y.; Yang, D.; Roth, H.; Xu, D., “Swin transformers for semantic segmentation of brain tumors in MRI images. In Brain lesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries:”, 7th International Workshop, 2021.
Jia, Q.; Shu, H. Bitr-Unet, “A CNN-transformer combined network for MRI brain tumor segmentation. In Brain lesion: Glioma, Multiple Sclerosis, Stroke, and Traumatic Brain Injuries”, 7th International Workshop, Brain Les 2021.
Wang, W.; Chen, C.; Ding, M.; Yu, H.; Zha, S.; Li, J., “Trans bts: Multimodal brain tumor segmentation using transformer”, in Proceedings of the Medical Image Computing and Computer, France, 27 September–1 October 2021.
Peiris, H.; Hayat, M.; Chen, Z.; Egan, G.; Harandi, M., “A robust volumetric transformer for accurate 3D tumor segmentation”, In Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2022: 25th International Conference, Singapore, 18–22 September 2022.
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I., “Attention is all you need,” in Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017.
Sharma, K.; Kaur, A.; Gujral, S., “A review on various brain tumor detection techniques in brain MRI images”, IOSR J. Eng. 2014, 4, 6–12.
Wiest, R.; Menze, B.; Reyes, M.; Porz, N.; Van Leemput, K., “The multimodal brain tumor image segmentation benchmark (BraTS)”, IEEE Trans. Med. Imaging 2014, 34, 1993–2024.
Park, N.; Kim, S., “How do vision transformers work?” arXiv 2022, arXiv:220206709.
Özyurt, F.; Sert, E.; Avci, E.; Dogantekin, E.,” Brain tumor detection based on Convolutional Neural Network with neutrosophic expert maximum fuzzy sure entropy”, Measurement 2019.
Polly, F.; Shil, S.; Hossain, M.; Ayman, A.; Jang, Y.M., “Detection and classification of HGG and LGG brain tumor using machine learning”. In Proceedings of the 2018 International Conference on Information Networking (ICOIN), Chiang Mai, Thailand, 10–12 January 2018; pp. 813–817.
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al., “An image is worth 16 × 16 words: Transformers for image recognition at scale”, arXiv 2020, arXiv201011929.
Steyaert, S.; Qiu, Y.; Zheng, Y.; Mukherjee, P.; Vogel, H.; Gevaert, O., “Multimodal data fusion of adult and pediatric brain tumors with deep learning”. Med arXiv 2022.
Deepak, S.; Ameer, P. Brain tumor classification using deep CNN features via transfer learning. Computing. Biol. Med. 2019.
Tummala, S.; Kadry, S.; Bukhari, S.; Rauf, H.T., “Classification of Brain Tumor from Magnetic Resonance Imaging Using Vision Transformers Ensembling”. Curr. Oncol. 2022, 29, 7498–7511.
Shaik, N.S.; Cherukuri, T.K., “Multi-level attention network: Application to brain tumor classification”. Signal Image Video Process 2022, 16, 817–824.
Ahmad, B.; Sun, J.; You, Q.; Palade, V.; Mao, Z., “Brain tumor classification using a combination of variational autoencoders and generative adversarial networks”. Biomedicines 2022, 10, 223.
Asiri, A.A.; Aamir, M.; Shaf, A.; Ali, T.; Zeeshan, M.; Irfan, M.; Alshamrani, K.A.; Alshamrani, H.A.; Alqahtani, F.F.; Alshehri, A.H.D., “Block-Wise Neural Network for Brain Tumor Identification in Magnetic Resonance Images”. Comput. Mater. Contin. 2022, 73, 5735–5753.
Asiri, A.A.; Shaf, A.; Ali, T.; Aamir, M.; Usman, A.; Irfan, M.; Al-Shamrani, H.A.; Mehdar, K.M.; Alshehri, O.M.; Alqhtani, S.M. “Multi-Level Deep Generative Adversarial Networks for Brain Tumor Classification on Magnetic Resonance Images”. Intell. Autom. Soft Computing. 2023, 127–143.
Asala Z., Khawla H., “Brain MRI Images segmentation based on U-Net architecture”. Iraqi Journal for Electrical and Electronic Engineering (IJEEE). Volume 18, Issue 1, 2022
Fabian I, Philip K., Wolfgang at el., “Brain tumor segmentation and radiomics survival predictions: Contribution to the BraTS2017 challenge”, Lecture notes in computer science, pp 287-297, 2018.
Ayesh Younis, Li Qiang, Charles O., et al., “Brain tumor analysis using deep learning and VGG16 ensembling learning approaches”. MDPI, Applied Science, 12(14), 2022.
Abu Bakr; Shadman Sakib; Mohammed M. et al., “Deep Convolutional Neural Networks Model Based Brain Tumor Detection in Brain MRI Images.” Fourth International Conference on I-SMAC, India, 2020.
Heba Mohsen; El-Sayed E.; El-Syed M.; “Classification using deep learning neural networks for brain tumors”. Future Computing and Informatics Journal, Vol. 3, Issue 1, Science Direct 2018.
M. O. Khairandish, M. Sharma, V. Jain, et al., “A Hybrid CNN-SVM Threshold Segmentation Approach for Tumor Detection and Classification of MRI Brain Images”. IRBM, ELSEVIER, Vol. 43, Issue 4, 2022.
Ruria, S., Gautam, P., Raj, A., & Pandey, G. (2024). Brain Tumor Detection System using Deep Learning. In International Journal of Innovative Technology and Exploring Engineering (Vol. 13, Issue 3, pp. 23–27). https://doi.org/10.35940/ijitee.h9678.13030224
Agrawal, Y., & Birchha, V. (2020). Classification of Brain Tumour in MRI Images using BWT and SVM Classifier. In International Journal of Recent Technology and Engineering (IJRTE) (Vol. 8, Issue 6, pp. 3662–3667). https://doi.org/10.35940/ijrte.f7958.038620
Shetty, S., & Shetty, J. (2020). Classification of Brain Tumor using Convolutional Neural Networks. In International Journal of Engineering and Advanced Technology (Vol. 9, Issue 3, pp. 2841–2845). https://doi.org/10.35940/ijeat.c5995.029320