Papers With Code 2 | ML Benchmarks, SotA Results & Code

The ShareGPT4Video dataset is a large-scale resource designed to improve video understanding and generation¹. It features 1.2 million highly descriptive captions⁴ for video clips, surpassing existing datasets in diversity and information content⁴. The captions cover a wide range of aspects, including world knowledge, object properties, spatial relationships, and aesthetic evaluations⁴.

The dataset includes detailed captions of 40K videos generated by GPT-4V¹ and 4.8M videos generated by ShareCaptioner-Video¹. The videos are sourced from YouTube and other user-uploaded video websites, and they cover a variety of scenarios, such as human activities and auto-driving¹.

The ShareGPT4Video dataset also provides a basis for the ShareCaptioner-Video, an exceptional video captioner capable of efficiently generating high-quality captions for videos of a wide range of resolution, aspect ratio, and duration¹.

For example, the dataset includes a detailed caption of a video documenting a meticulous meal preparation by an individual with tattooed forearms¹. The caption describes the individual's actions in detail, from slicing a cucumber to mixing the dressing and adding croutons to the salad¹.

In addition to its use in research, the ShareGPT4Video dataset has been used to train the sharegpt4video-8b model, an open-source video chatbot². This model was trained on open-source video instruction data and is primarily intended for researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence².

(1) arXiv:2406.04325v1 [cs.CV] 6 Jun 2024. https://arxiv.org/pdf/2406.04325. (2) ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. https://arxiv.org/abs/2311.12793. (3) Lin-Chen/sharegpt4video-8b · Hugging Face. https://huggingface.co/Lin-Chen/sharegpt4video-8b. (4) ShareGPT4Video: Improving Video Understanding and Generation with .... https://www.aimodels.fyi/papers/arxiv/sharegpt4video-improving-video-understanding-generation-better-captions. (5) GitHub - ShareGPT4Omni/ShareGPT4Video: An official implementation of .... https://github.com/ShareGPT4Omni/ShareGPT4Video. (6) undefined. https://sharegpt4video.github.io/.