SUMMARY

A global financial services firm partnered with Folio3 AI to build an AI-powered horse race tracking and analytics platform.

The goal: process live horse race video feeds, extract meaningful race insights, and convert complex race footage into structured JSON outputs for deeper analysis and prediction workflows. Folio3 AI built a deep learning pipeline to process live horse race videos, extract race insights, and deliver structured JSON outputs for scalable analytics and prediction workflows.

ABOUT THE CUSTOMER

The client is a technology-focused division of a large multinational financial services organization operating in a specialized market sector. Their division develops advanced analytical models, proprietary strategies, and algorithmic frameworks to support automated decision-making.

The client’s business depends on accurate, scalable, and data-driven race intelligence. They needed a way to analyze horse races at scale and extract meaningful insights from complex race video data.

Before partnering with Folio3 AI, the client had access to live race video feeds and domain-specific data, but lacked the complete AI engineering pipeline needed to convert that unstructured footage into structured outputs for advanced analysis.

THE CHALLENGE

Horse race intelligence has traditionally depended on manual video review, domain experts, and fragmented race data. Analysts had to watch race footage, interpret fast-moving visual patterns, and manually convert observations into usable insights for further modeling. This approach was time-consuming, difficult to scale, and exposed the client’s analytics workflows to inconsistency.

The client already had access to live race video feeds and domain-specific race data. What they lacked was the AI engineering layer required to transform that unstructured footage into structured, machine-readable race intelligence. Their platform needed to process actual race videos at scale, identify meaningful patterns from complex footage, and return standardized outputs that could support deeper analysis and prediction workflows.

Three specific technical challenges defined the project:

• High-speed multi-subject race tracking — Horse race videos include multiple fast-moving horses, dense subject overlap, shifting camera angles, motion blur, changing lighting, and frequent occlusions. The system needed to detect and track meaningful race activity without relying on frame-by-frame manual review.
• Scalable video-to-insight ML pipeline — The client did not need a standalone model. They needed a complete machine learning pipeline that could ingest race video feeds, preprocess footage, run deep learning inference, validate outputs, and prepare results for platform-level use at scale.
• Structured outputs for prediction workflows — Raw visual observations were not enough. The extracted race findings had to be converted into clean JSON outputs so the client’s analytics environment could consume them for reporting, modeling, automated decision-making, and prediction workflows.

OUR DELIVERY APPROACH

Folio3 AI developed a deep learning solution that takes race video inputs from the client’s platform, processes them using algorithms trained on client-provided data, extracts meaningful insights, and returns outputs in JSON format for further platform-level analysis.

The team designed the solution as a complete ML pipeline, not a standalone model. This allowed the client to process race video data at scale, fine-tune model outputs against ground truth, and integrate the results directly into their existing analytics environment.

TOOLS & TECHNOLOGIES

THE SOLUTION

The team designed and built an AI-powered horse race tracking and analytics platform that acts as the intelligence layer between live race video feeds and the client’s analytics environment. The solution ingests race footage, breaks it down into analyzable frames, detects and tracks horses across the race, and converts extracted race activity into structured JSON outputs for downstream analysis.

At its core, the platform processes high-speed race videos through a computer vision and deep learning pipeline trained on client-provided race footage.

The Four-Stage Pipeline

Folio3 built a four-stage pipeline that takes raw horse race video feeds and converts them into structured, platform-ready race intelligence for advanced analytics and prediction workflows.

1. Video ingestion and frame extraction — The client’s platform provides live horse race video feeds as the primary input. The system ingests the footage, standardizes video formats, extracts frames at defined intervals, and organizes them into model-ready sequences while preserving race timing and frame-level context.

2. Video preprocessing and object preparation — Race footage is processed to improve model performance before inference. This includes frame resizing, noise reduction, motion blur handling, visual normalization, and preparation of race segments where horses, track regions, and movement patterns are most relevant for analysis.

3. Computer vision and deep learning inference — The processed video sequences are passed through trained AI models designed to detect race participants, track movement across frames, and extract race-level facts from complex footage. The models analyze fast-moving horses, overlapping subjects, changing camera angles, and visual occlusions to identify meaningful race activity without manual review.

4. JSON insight generation and platform delivery — The extracted race insights are converted into structured JSON outputs that can be directly consumed by the client’s analytics platform. These outputs support deeper race analysis, reporting, prediction workflows, and automated decision-making across the client’s existing data environment.

SOLUTION ARCHITECTURE

RESULTS ACROSS RACE ANALYTICS WORKFLOWS