Video meets knowledge in visual question answering

Abstract

In this work, we address knowledge-based visual question answering in videos. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs that combines visual, textual and temporal coherence reasoning together with knowledge-based questions. Second, we propose a video understanding model by combining the visual and textual video information with specific knowledge about the dataset. We find that the incorporation of knowledge produces outstanding improvements for VQA in video. However, the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.