Robust online video instance segmentation with track queries

Zhan, Zitong

Permalink

https://hdl.handle.net/2142/121199 Copy

Description

Title

Robust online video instance segmentation with track queries

Author(s)

Zhan, Zitong

Issue Date

2023-07-17

Director of Research (if dissertation) or Advisor (if thesis)

Lazebnik, Svetlana

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• computer vision
• video tracking

Abstract

With support of data and maturity of model, deep learning has been successful in high level understanding on videos. Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like Unidentified Video Objects (UVO) and Occluded Video Instance Segmentation (OVIS). We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (ROVIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multi-object tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos. Lastly, we extend ROVIS to the Refer-VOS task which is highly relevant to VIS, such that ROVIS can make segmentation predictions based on language input, and present our preliminary results.

Graduation Semester

2023-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/121199

Copyright and License Information

Copyright 2023 Zitong Zhan

Owning Collections