Towards efficient and powerful machine learning vision systems for mobile: unifying interactive segmentation and matting

Wu, Mingyuan

Permalink

https://hdl.handle.net/2142/129245 Copy

Description

Title

Towards efficient and powerful machine learning vision systems for mobile: unifying interactive segmentation and matting

Author(s)

Wu, Mingyuan

Issue Date

2025-04-28

Director of Research (if dissertation) or Advisor (if thesis)

Nahrstedt, Klara

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Machine Learning
• Segmentation
• Matting

Abstract

Recent advancements in hardware have significantly enhanced the capabilities of smartphones, tablets and head-mounted devices, transforming these mobile devices from mere communication tools into powerful multimedia and creative platforms. This transformation has created unprecedented demand for advanced image editing functionalities running behind mobile applications. Today, users no longer like waiting until they return home to their desktop computers and launch software like Photoshop for sophisticated image manipulation. Instead, they expect equally powerful editing features that allow immediate modifications at the moment they capture or discover content worth sharing on their mobile devices. These features empower users across various contexts, from social media creators to professionals, profoundly influencing the way visual content is produced, consumed, and shared, and more importantly, the way how people interact, connect, and share their interests and cherished moments with those they love. These image editing functionalities are largely backed with recent innovations in efficient deep learning in computer vision areas. However, deploying heavy machine learning algorithms efficiently on mobile platforms remains challenging due to inherent constraints such as limited computation power, battery capacity, and the need for real-time respond. These resource limitations directly affect user experience, making it essential to develop highly optimized image editing algorithms specifically for the mobile environment. These algorithms must deliver high accuracy, high computational efficiency, low latency, real-time user interactivity, and robust generalization across diverse visual contents. In this thesis, we focus on segmentation and matting, which serve as foundational editing techniques for background replacement, portrait, and precise foreground object extraction, etc. We develop efficient neural network based algorithms designed for the resource-constrained mobile environment. Moreover, we tailor our editing functionalities to practical user scenarios, adding interactivity in segmentation tasks and reducing input requirements for matting processes, for better user experience and compatibility across wider range of mobile devices. Specifically, TraceNet ackles efficient and interactive instance segmentation by explicitly locating the user-selected instance via receptive field tracing, and thus significantly reduce computations in neural networks. Meanwhile, I-Matting addresses trimap-free image matting via a hierarchical adversarial training mechanism and a patch-rank component, and improves the matting accuracy while reducing computations. Collectively, these methods constitute a meaningful step towards efficient mobile-based image editing, demonstrating their effectiveness through accuracy and efficiency improvements in extensive datasets.

Graduation Semester

2025-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129245

Copyright and License Information

Copyright 2025 Mingyuan Wu

Owning Collections