To Decay or not to Decay: Modeling Video Memorability Over Time

From our paper: V. Casser*, C. Fosco*, A. Newman*, B. McNamara and A. Oliva: “To Decay or not to Decay: Modeling Video Memorability Over Time” Shared Visual Representations in Human and Machine Intelligence, NeurIPS’19.

Videos present a unique challenge for understanding visual memorability due to their motion and evolution across frames. To enable exploration of this field, we introduce Memento10k, the largest video memorability dataset to date. Based on our analysis of this data, we propose a new formulation for how video memorability decays that differs significantly from image memorability. Finally, we design a prediction network, Temporal MemNet, that achieves state-of-the-art performance by harnessing both visual and temporal content. The model accounts for 90% of human consistency, leaving room for improvement in terms of understanding and imitating how humans process motion in videos.

Supplementary Material