CNNFORMER interpolation: a demosaicing framework for single-chip color–NIR1 imaging in fluorescence-guided cancer surgery

Yang, Jiankun

Permalink

https://hdl.handle.net/2142/129754 Copy

Description

Title

CNNFORMER interpolation: a demosaicing framework for single-chip color–NIR1 imaging in fluorescence-guided cancer surgery

Author(s)

Yang, Jiankun

Issue Date

2025-05-09

Director of Research (if dissertation) or Advisor (if thesis)

Gruev , Viktor

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Machine Learning
• In-vivo Imaging
• Computer Vision

Abstract

Single-chip multispectral imaging sensors that integrate vertically stacked photodiodes with pixelated spectral filters offer significant promise for advanced, real-time visualization during image-guided cancer surgery. However, the inherent design trade-offs of these sensors lead to a substantial reduction in spatial resolution, which can limit their clinical utility. Conventional demosaicing algorithms, such as bilinear interpolation and convolutional neural network (CNN)-based approaches, have been developed to mitigate these limitations by improving the effective resolution and reducing common image artifacts. While these methods have achieved meaningful improvements, they still exhibit deficiencies in accuracy and often introduce smaller, yet clinically relevant, reconstruction artifacts, particularly in regions with high spatial frequency content. To address these persistent challenges, this work presents a CNN–Transformer-based demosaicing algorithm specifically optimized for reconstructing high-resolution color and near-infrared (NIR) images acquired by a hexachromatic multispectral imaging sensor. Transformer architectures, originally developed for natural language processing to capture long-range dependencies within sequential data, have recently shown remarkable success when adapted to vision tasks due to their ability to model complex, global relationships across an image. By integrating convolutional layers for local feature extraction with Transformer modules for capturing long-range context, the proposed hybrid model was developed and trained on extensive color-image datasets and rigorously evaluated on both color and NIR imagery. Quantitative evaluation demonstrates that the CNN–Transformer demosaicing method achieves an average mean squared error (MSE) reduction of approximately 85% for color images and 76% for NIR images, along with improvements in structural dissimilarity (DSSIM) of roughly 72% and 79%, respectively, when compared to state-of-the-art CNN-based demosaicing algorithms on preclinical datasets. In clinical datasets, the model similarly achieves significant reductions in MSE and DSSIM, particularly excelling at reconstructing fine image details critical for accurate intraoperative visualization. These results underscore the capability of Transformer-based models to surpass the limitations of traditional CNN-only approaches by capturing more comprehensive spatial information. Leveraging recent advances in GPU computing, the proposed CNN–Transformer demosaicing framework offers a practical, real-time solution for enhancing spatial resolution and image fidelity in multispectral imaging applications, ultimately advancing the effectiveness of image-guided cancer surgery.

Graduation Semester

2025-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129754

Copyright and License Information

Copyright 2025 Jiankun Yang

Owning Collections