Video Annotation Tasks
This category contains annotation task designs for video understanding research, including action recognition, temporal localization, video summarization, and scene understanding.
Run these designs in Potato. See the Potato documentation to configure temporal and frame-level video annotation.
Subcategories
| Design |
Description |
Reference |
| activitynet-temporal-localization |
ActivityNet is a large-scale video benchmark for human activity understanding, with untrimmed YouTube… |
Caba Heilbron et al., CVPR 2015 |
| ava-atomic-actions |
AVA spatio-temporally localizes 80 atomic actions on people in movie clips, labeled at 1 Hz keyframes |
Gu et al., CVPR 2018 |
| breakfast-actions |
Fine-grained temporal action segmentation of breakfast preparation activities |
Kuehne et al., CVPR 2014 |
| charades-activity-segmentation |
Multi-label temporal activity segmentation in indoor home videos |
Sigurdsson et al., ECCV 2016 |
| epic-kitchens-egocentric |
Annotate fine-grained actions in egocentric kitchen videos with verb-noun pairs |
Damen et al., ECCV 2018 |
| finediving-procedure-aqa |
Fine-grained diving action quality assessment |
Xu et al., CVPR 2022 |
| finegym-action-segments |
FineGym is a hierarchical video dataset (CVPR 2020) for fine-grained gymnastics action recognition:… |
Shao et al., CVPR 2020 |
| fisv-figure-skating-scoring |
Automatic figure skating scoring as video regression |
Xu et al., IEEE TCSVT 2020 |
| soccernet-action-spotting |
SoccerNet-v2 defines 17 action spotting classes (goal, corner, card, foul, substitution, and more) with… |
Giancola et al., CVPR 2018 |
| thumos14-action-localization |
Temporal action localization in untrimmed sports videos |
Jiang et al., ECCV 2014 Workshop |
| Design |
Description |
Reference |
| ami-meeting-annotation |
Multi-tier ELAN-style annotation of multi-party meeting recordings |
Carletta et al., MLMI 2005 |
| Design |
Description |
Reference |
| moviescenes-detection |
Detect and annotate scene boundaries in movies |
Rao et al., CVPR 2020 |
| scene-boundary-detection |
Identify scene boundaries in documentary and narrative videos |
Sidiropoulos et al., IEEE TCSVT 2011 |
| shot-boundary-detection |
Detect shot boundaries and classify transition types in broadcast video |
Smeaton et al., 2010 |
| Design |
Description |
Reference |
| childes-child-language |
Multi-tier ELAN-style annotation of child-adult interaction videos for language acquisition research |
MacWhinney, 2000 (Lawrence Erlbaum, 3rd ed.) |
| Design |
Description |
Reference |
| cmu-mosei-multimodal-sentiment |
CMU-MOSEI is the largest multimodal dataset for sentiment and emotion analysis, with 23,453 annotated… |
Zadeh et al., ACL 2018 |
| Design |
Description |
Reference |
| dgs-corpus-sign-language |
Multi-tier ELAN-style annotation of German Sign Language (DGS) corpus videos |
Hanke et al., LREC 2020 |
| Design |
Description |
Reference |
| ego4d-episodic-memory |
Annotate egocentric (first-person) video for episodic memory tasks including activity segmentation, hand… |
Grauman et al., CVPR 2022 |
| Design |
Description |
Reference |
| how2sign-sign-language |
Multi-tier ELAN-style annotation of continuous American Sign Language videos |
Duarte et al., CVPR 2021 |
| Design |
Description |
Reference |
| iemocap-dyadic-emotion |
IEMOCAP is a 12-hour audiovisual corpus of two-actor emotional conversations with categorical and… |
Busso et al., Language Resources and Evaluation 2008 |
| Design |
Description |
Reference |
| howto100m-instructional |
Annotate instructional video clips with step descriptions and visual grounding |
Miech et al., ICCV 2019 |
| youcook2-instructional |
YouCook2 contains 2,000 cooking videos across 89 recipes (176 hours), each segmented into recipe steps… |
Zhou et al., AAAI 2018 |
| Design |
Description |
Reference |
| msad-anomaly-detection |
Video anomaly detection across multiple scenarios |
Zhu et al., NeurIPS 2024 |
| Design |
Description |
Reference |
| saga-gesture-speech |
Multi-tier ELAN-style annotation of co-speech gestures and their alignment with spoken language |
Lücking et al., Journal on Multimodal User Interfaces 2013 |
| Design |
Description |
Reference |
| movienet-scene-classification |
Scene classification labeling on MovieNet, a 1,100-movie dataset with scene, place, action, and cinematic-style annotations |
Huang et al., ECCV 2020 |
| Design |
Description |
Reference |
| lsmdc-keyframe-selection |
Select representative keyframes from movie clips for video description tasks |
Rohrbach et al., IJCV 2017 |
| summe-summarization |
Create video summaries by selecting key segments that best represent the content |
Gygli et al., ECCV 2014 |
| tvsum-summarization |
Frame-level importance scoring for video summarization |
Song et al., CVPR 2015 |
| youtube-highlights |
Detect highlight-worthy moments in domain-specific videos |
Sun et al., ECCV 2014 |
| Design |
Description |
Reference |
| activitynet-captions |
ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for… |
Krishna et al., ICCV 2017 |
| charades-sta-grounding |
Ground natural language descriptions to video segments |
Gao et al., ICCV 2017 |
| didemo-moment-retrieval |
Localizing natural language descriptions to specific video moments |
Hendricks et al., ICCV 2017 |
| qvhighlights-moment-saliency |
Query-based moment localization and highlight rating based on the QVHighlights dataset (Lei, Berg, Bansal,… |
Lei et al., NeurIPS 2021 |
| Design |
Description |
Reference |
| vbench-generation-quality |
VBench (CVPR 2024) breaks text-to-video quality into 16 evaluation dimensions, from temporal flickering to… |
Huang et al., CVPR 2024 |
| Design |
Description |
Reference |
| video-chatgpt-qa-display |
Video question answering evaluation based on the Video-ChatGPT benchmark |
Maaz et al., ACL 2024 |
| Design |
Description |
Reference |
| mvbench-video-understanding |
Comprehensive video understanding benchmark with multiple-choice questions, video segment annotation, and… |
Li et al., CVPR 2024 |
| nextqa-temporal |
NExT-QA is a video QA benchmark for causal and temporal action reasoning over 5,440 videos and ~52K questions |
Xiao et al., CVPR 2021 |
| Design |
Description |
Reference |
| vstar-video-dialogue |
Video-grounded dialogue annotation |
Wang et al., ACL 2023 |
Quick Start
# Navigate to a specific task
cd video/action-recognition/ava-atomic-actions
# Run with Potato
potato start config.yaml
Annotation Modes
The video_annotation scheme takes a mode setting. In segment mode annotators mark start and end times; in keyframe mode they pick representative frames; in tracking mode they follow an object across frames; and combined mode puts several of these on the same video.
Task Count
Total: 40 video annotation tasks across 21 subcategories