Video meets knowledge in visual question answering

概要

In this work, we address knowledge-based visual question answering in videos. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs that combines visual, textual and temporal coherence reasoning together with knowledge-based questions. Second, we propose a video understanding model by combining the visual and textual video information with specific knowledge about the dataset. We find that the incorporation of knowledge produces outstanding improvements for VQA in video. However, the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.

収録
画像の認識・理解シンポジウム(MIRU2019)論文集
Noa Garcia
Noa Garcia
特任助教

Her research interests lie in computer vision and machine learning applied to visual retrieval and joint models of vision and language for high-level understanding tasks.

Chenhui Chu
Chenhui Chu
招へい准教授
中島悠太
中島悠太
准教授

コンピュータビジョン・パターン認識などの研究。ディープニューラルネットワークなどを用いた画像・映像の認識・理解を主に、自然言語処理を援用した応用研究などに従事。

関連項目