Arsha Nagrani
anagrani at google dot com

I am a senior research scientist at Google AI Research, where I work on machine learning for video understanding. I did my PhD with Andrew Zisserman in the VGG group at the University of Oxford, where I was fortunate enough to be funded by a Google PhD Fellowship. My thesis won the ELLIS PhD award.

Before that I did my undergrad at the University of Cambridge, where I worked with Roberto Cipolla and Richard Turner.

CV  /  Google Scholar  /  LinkedIn  /  Twitter  /  GitHub  /  Thesis

My research focuses on self-supervised and multi-modal machine learning techniques for video recognition, including the use of sound and text to learn better visual representations. Recently, I have also become interested in computer vision for wildlife conservation. For a full list of publications please see Google Scholar.

New dataset and video-language tasks for video chapterisation in long web videos.

Scaling up video and language models gives SOTA on 25+ VL benchmarks.

Image-text models like CLIP can be used for Moment Retrieval, Temporal Localization, and Action Segmentation

Describing visual content in movies (automatic audio descriptions!), focus on character recognition. This is part II, see below for AutoAD I @ CVPR23.

Generating better negatives using an LLM can produce a verb-focused video-text model with contrastive training.

LLMs are used to obtain emotion training data from speech.

LLMs are used to automatically create modular executable code, which is used to solve visual QA. CodeVQA gets SOTA on the COVR and GQA datasets.

SOTA, zero-shot audiovisual ASR is possible by using lightweight adaptors on top of large frozen unimodal models.

Describing visual content in movies (automatic audio descriptions!)

A single-stage, dense event captioning model pretrained on narrated videos at scale achieves SOTA for dense video captioning.

Creating short video summaries for instructional videos using simple heuristics. New test set released.

Visual context (objects, actions) improves ASR performance under challenging audio conditions.

Mining audiovisual clips for text captions by leveraging image-image similarity for SOTA video retrieval and captioning.

New unsupervised pretraining framework for multimodal video captioning that leverages future utterances in unlabelled videos.

Video encoder pretraining using appearance, sound, and transcribed speech, by masking out an entire modality and predicting it using the others.

Fully automated, audio-visual pipeline to detect and track two audio-visually distinctive actions in wild chimpanzees: buttress-drumming and nut-cracking.

Fully transformer based multimodal fusion model gets SOTA on video classification. Attention bottlenecks at multiple layers force cross-modal information to be condensed thereby improving performance at lower computational cost.

End-to-end encoder for visual retrieval that uses only self-attention blocks. This allows flexible training with variable length videos and images jointly.

Encoding augmentations along with data views gives SOTA on video self-supervised learning benchmarks.

Transformer model for audio-visual synchronization works well on non-speech classes in the wild.

Localizes sounding objects without any supervision using hard negative mining from within the image. Gives SOTA on Flickr SoundNet and a new VGG-SS dataset.

Uses a language model to learn a sequence of actions as temporal context for egocentric action recognition.

Predicting future utterances in a video based on previous dialogue and video frames without manual labels gives SOTA on standard QA datasets.

Two stream audio recognition models that gets SOTA on VGG-Sound and EPIC-Kitchens-100.

Investigate the performance of speaker recognition in movies, where often actors intentionally disguise their voice to play a character.

A large-scale story understanding dataset that contains the key scenes from movies with semantic captions. Basis of the CMD Challenge at ICCV 2021.

Breaking up multispeaker videos into "who spoke when". Based on this work we are hosting a new speaker diarisation track at the VoxCeleb Speaker Recognition Challenge.

Action localisation in movies using video level labels only.

Action recognition in movies using the speech alone.

Disentanglement of speech embeddings into content and identity with only accompanying facetrack as supervision. Based on this work we are hosting a new self-supervised track at the VoxCeleb Speaker Recognition Challenge.

Overview of the VoxCeleb1 and VoxCeleb2 datasets including various updates and splits, and new models for speaker recognition.

Recognition of wild chimpanzees using full body and full frame CNNs methods. We also release an 'in the wild' video chimpanzee recognition dataset.

Face detection, tracking, and recognition of wild chimpanzees from long-term video records using deep CNNs. We also show a brief application for social network analysis.

We propose a novel architecture for combining modalities in videos for action recognition, by using a temporal window to allow a range of temporal offsets.

We fuse the information from different embeddings experts for the task of video retrieval - achieving SOTA results on 5 different datasets. This work is also the basis for the Video Pentathlon at CVPR 2020.

A NetVlad layer in a deep CNN works well for speaker recognition on long noisy speech utterances.

We use the redundant (common) signal in both audio (speech) and vision (faces) to learn speech representations for emotion recognition without manual supervision.

Speaker Recognition in the Wild using deep CNNs. The VoxCeleb datasets are also used integrally in the VoxCeleb Speaker Recognition Challenge.

We learn joint embedding of faces and voices using cross-modal self-supervision from YouTube videos.

Can you recognise someone’s face if you have only heard their voice? Or recognise their voice if you have only seen their face?

We use face recognition and active speaker detection to automatically create a large scale speaker identification dataset from YouTube videos.

This guy is good at website design.