Automatically Labeling Offensive Formations in American Football Film Using Deep Learning

Abstract

Web services for storing, annotating, and sharing sports videos by high-school athletic teams have become more prevalent in recent years. However, most services lack the ability for coaches to automatically tag film, leading to many hours of manual annotation. For American football videos, coaches need to label formations, plays, and field positions in order to extract insights and create strategic game plans. This paper presents an end-to-end machine learning pipeline for automatically labeling American football offensive formations in videos. The pipeline includes pre-processing of videos, image classification, and a novel inference approach. The study compares a custom CNN model with pre-trained image classifier models using transfer learning. Specifically, CNN-based architectures (MobileNet, Inception, EfficientNet, etc.) and a transformer-based Vision Transformer (ViT) are compared. All models are trained on ~1400 images with the three most popular formation labels extracted from video clips of high-school football team games. The results show that several models, including the custom CNN model, achieved greater than 90% classification accuracy on the test dataset. The inference is performed by sampling multiple frames from a video clip, passing them through a trained image classifier, and taking a majority vote on the classification results to determine the final outcome. Our study found that using a sampling rate of 0.5 seconds, starting at 1 second, and taking five frames yields the highest inference accuracy of 95.4% using a trained customized CNN model. This system can assist all levels of football coaches in the analysis of game footage and formation identification.