{"id":1152,"date":"2025-07-28T06:53:04","date_gmt":"2025-07-28T06:53:04","guid":{"rendered":"https:\/\/troyberry.dev\/?page_id=1152"},"modified":"2025-08-01T19:12:29","modified_gmt":"2025-08-01T19:12:29","slug":"real-time-telemetry-analytics-for-motorsports","status":"publish","type":"page","link":"https:\/\/troyberry.dev\/index.php\/real-time-telemetry-analytics-for-motorsports\/","title":{"rendered":"Real Time Telemetry Analytics for Motorsports"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\n
\n \n (McLaren Racing via Splunk) <\/span>\n <\/div>\n \t\t\t\t
\n\t\t\t
\n\t\t\t\t

Overview<\/strong><\/p><\/div>

Kermit Troy Berry collaborated with McLaren Racing\u2019s Formula 1 team to build a real time telemetry analytics platform that transforms how race data is utilized. In the high stakes world of motorsports, every millisecond of sensor data can influence strategy and performance. Kermit\u2019s project involved harnessing a torrent of telemetry (over 100,000 events per second from car sensors) and converting it into actionable insights using Splunk\u2019s Data to Everything platform. The solution enhanced on track decision making and off track reliability by enabling instant anomaly detection and comprehensive observability across McLaren\u2019s hybrid cloud and edge infrastructure.<\/p><\/div>

Objective<\/strong><\/p><\/div>

McLaren Racing sought to improve on track decisions and operational resilience by leveraging the massive volume of data generated during races and testing. The objective was to develop a telemetry analytics system capable of:<\/p><\/div>

  • Ingesting and processing telemetry data in real time (from cars\u2019 hundreds of sensors, race simulations, and IT systems).<\/li>
  • Automatically detecting anomalies or performance issues (in car components or IT infrastructure) as they occur, so engineers can respond immediately.<\/li>
  • Streamlining log and incident analysis through intelligent automation, given the limited time windows during and between races.<\/li>
  • Providing a unified observability dashboard for both trackside and factory teams to monitor system health, car performance metrics, and alerts.
    Ultimately, the goal was to sharpen McLaren\u2019s competitive edge by using data more effectively than ever, ensuring that both the race cars and the supporting technology operate at peak performance at all times.<\/li><\/ul>

    Role and Responsibilities<\/strong><\/p><\/div>

    As a Senior ML Software Engineer attached to the McLaren\u2013Splunk partnership, Kermit took on several key responsibilities:<\/p><\/div>

    • Data Pipeline Architecture: He architected the end to end telemetry ingestion pipeline using OpenTelemetry standards to collect data from distributed sources and funnel it into Splunk. This pipeline was designed to reliably handle an onslaught of ~100k events per second, filtering and indexing telemetry data (car sensor streams, network logs, application metrics) in real time.<\/li>
    • Anomaly Detection Development: Kermit created custom anomaly detection algorithms using a combination of Splunk\u2019s Search Processing Language (SPL) for rule based thresholds and Python based machine learning for more complex pattern recognition. These detectors were configured to flag abnormal patterns in sensor data (e.g. sudden drops in oil pressure, surges in brake temperatures) and in IT system logs, enabling proactive issue resolution.<\/li>
    • SOAR Integration with NLP: Leveraging Splunk Phantom (Security Orchestration, Automation, and Response), he extended McLaren\u2019s incident response automation. Kermit integrated a PyTorch NLP model into Phantom playbooks to intelligently triage log messages and alerts. This model could read unstructured log text or error reports and classify their criticality or category, helping engineers prioritize significant issues (for example, distinguishing a harmless sensor glitch from a genuine early warning of component failure).<\/li>
    • Infrastructure as Code & Automation: He implemented Infrastructure as Code for the telemetry platform using Terraform and automated provisioning via Splunk\u2019s REST APIs. All Splunk deployments (search head, indexers, forwarders configurations) and Phantom integrations were codified, allowing reproducible environments and quick rollout of updates across on premises trackside rigs and cloud servers. This ensured consistency in configuration from the garage to the cloud and reduced setup time when racing at different circuits.<\/li>
    • Training and Best Practices: Kermit also served as an evangelist of observability within the team. He conducted training sessions for McLaren\u2019s performance engineers and IT staff on using the new Splunk dashboards, understanding anomaly alerts, and following observability best practices. He prepared playbooks and guidelines so that race strategists, reliability engineers, and developers could all confidently use the telemetry insights and respond to incidents swiftly.<\/li><\/ul>

      Approach and Technology Stack<\/strong><\/p><\/div>

      Real Time Data Ingestion:<\/strong> The approach centered on adopting OpenTelemetry instrumentation across McLaren\u2019s systems. Every critical sensor on the F1 car, as well as supporting IT systems (databases, applications, network devices), was instrumented to emit standardized telemetry data. Kermit configured OpenTelemetry collectors to feed into Splunk at an extremely high throughput. The system leveraged Splunk Enterprise as the core platform for data ingestion and analytics, chosen for its proven ability to handle massive machine data streams. During races and test sessions, data was streamed both to on site Splunk instances (at the trackside IT rig) for low latency needs and to cloud based Splunk infrastructure for broader analysis and historical comparison.<\/p><\/div>

      Anomaly Detection & Analytics:<\/strong> Once data was in Splunk, Kermit developed a suite of analytics:<\/p><\/div>

      SPL based Alerts:<\/strong> For known thresholds (e.g., engine temperature exceeding X\u00b0C, network latency above Y ms), he used Splunk\u2019s query language to define real time alerts. These were tuned with McLaren\u2019s domain experts to minimize false alarms and ensure they reflected conditions that would warrant attention during a race.<\/p><\/div>

      ML based Detection:<\/strong> For more complex anomalies, such as subtle changes in vibration frequencies or multi sensor correlations that precede a failure, Python scripts were integrated via Splunk\u2019s Machine Learning Toolkit and custom search commands. He built anomaly detection models (like unsupervised clustering and predictive models) that ran over sliding windows of telemetry data. For example, a model using statistical profiling identified when a combination of sensor readings deviated from the normal pattern for that car and track condition, flagging it as a potential issue for engineers to investigate.<\/p><\/div>

      Natural Language Processing for Logs:<\/strong> A notable innovation was embedding a PyTorch based NLP model into Splunk Phantom workflows. Kermit trained this model on historical log data and incident reports to recognize patterns in text (like error messages or stack traces) and categorize them (for instance, \u201cnetwork issue,\u201d \u201csensor fault,\u201d \u201csoftware error\u201d). Phantom playbooks used this model\u2019s output to decide how to route or escalate alerts. As a result, if multiple log anomalies occurred, the system could auto prioritize those indicating critical failures (e.g., an imminent server crash affecting telemetry) over benign warnings, saving precious time for the team.<\/p><\/div>

      Tech Stack & Tools: The solution stack was diverse:<\/strong><\/p><\/div>

      Splunk Platform:<\/strong> including Splunk Enterprise (search and indexing engine for telemetry), Splunk Forwarders (for data ingestion), and Splunk Phantom (for automation and response).<\/p><\/div>

      OpenTelemetry & Telemetry SDKs:<\/strong> for instrumenting data from car ECUs (Electronic Control Units) and applications into a unified format.<\/p><\/div>

      Python & PyTorch:<\/strong> used for custom ML models and the NLP log analysis component. These ran in tandem with Splunk (via the Splunk Python SDK and Phantom\u2019s automation framework).<\/p><\/div>

      Terraform:<\/strong> for provisioning cloud infrastructure (like Splunk indexer clusters, virtual machines, network configurations) as code. The Splunk REST API was also scripted to create indexes, data sources, and saved searches automatically during deployments.<\/p><\/div>

      Monitoring & CI\/CD:<\/strong> While Splunk itself monitored racing systems, Kermit ensured the telemetry platform\u2019s health was also monitored (for example, forwarding Splunk internal metrics to a separate monitoring dashboard). Deployment of configuration changes and updates to ML models was managed through a CI\/CD pipeline that tested changes in a staging environment before applying to production (especially important before a race weekend).<\/p><\/div>

      Challenges and Solutions<\/strong><\/p><\/div>

      Ultra High Velocity Data:<\/strong> Managing 100k events per second in real time is non trivial. The sheer volume of data (100kHz telemetry streaming) could have overwhelmed systems. Kermit tackled this by optimizing each stage of the pipeline. He tuned OpenTelemetry collectors to batch data efficiently and set up multiple parallel ingestion streams into Splunk to distribute load. Within Splunk, indexer clustering and load balancing ensured throughput capacity. He also implemented selective filtering at the source for non critical data during peak times (for instance, lowering debug level logs ingestion during races) so that the most important signals were always prioritized. The result was a resilient pipeline that consistently kept up with the data firehose, proven through stress tests and live race conditions.<\/p><\/div>

      Data Quality and Noise:<\/strong> With thousands of data points per second, distinguishing a true performance anomaly from normal noise or expected variation was challenging. The solution was a layered anomaly detection strategy. Basic threshold breaches would catch obvious issues, but for nuanced patterns, Kermit\u2019s ML models looked at combinations of signals and temporal trends. He worked closely with McLaren\u2019s race engineers to incorporate domain rules (e.g., expected ranges when DRS is active, tire wear effects on telemetry) into model features. This hybrid approach dramatically improved signal to noise ratio in alerts the team saw a reduction in \u201cfalse positive\u201d alerts by an estimated ~30%, ensuring that when an alert fired, it genuinely required attention.<\/p><\/div>

      Hybrid Infrastructure Complexity:<\/strong> The telemetry system spanned on car edge devices, trackside servers, and cloud servers back at McLaren\u2019s factory. Ensuring consistency and reliability across this hybrid environment was complex. Kermit\u2019s introduction of Infrastructure as Code (Terraform) proved vital. It allowed the team to deploy the same Splunk configurations across different environments reproducibly. For example, before each race, the trackside rig\u2019s configuration (dashboards, alerts, indexes) could be synced with the master configuration tested at the factory. He also established robust offline buffering if connectivity dropped (as can happen at remote circuits), the on site system would cache data and forward it when back online, preventing loss. This design delivered continuity in data collection despite the distributed nature of the setup.<\/p><\/div>

      Rapid Incident Response:<\/strong> In F1, timing is everything; if an issue is detected, engineers need immediate insight. Initially, sifting through log files or numerous alerts could slow response. The integration of Splunk Phantom with an NLP model was Kermit\u2019s answer to speed this up. By automating log analysis and triage, the team could respond to critical incidents faster. One example occurred when a sudden network glitch struck the garage systems during practice Phantom\u2019s NLP classifier instantly flagged the network related error messages as high priority and triggered a response playbook, whereas previously such an issue might have been buried among dozens of less important alerts. This automation potentially saved the team from missing crucial telemetry during that practice session.<\/p><\/div>

      User Adoption and Skills Gap:<\/strong> Introducing advanced analytics and ML into a fast paced racing team environment required user trust and understanding. Some engineers were initially skeptical of automated insights or worried the new tools might distract during intense moments. Kermit addressed this by deeply involving the end users in the development cycle. He ran hands on workshops where engineers used the Splunk dashboards on historical data to see how it surfaced issues they already knew about. He also implemented a feedback loop: every false alert or missed detection was analyzed post race, and rules\/models were refined. Over time, as the team gained confidence that the system reliably mirrored their own expertise (and even caught things they hadn\u2019t seen), they embraced it. Training materials were created for new team members, and an \u201cobservability champion\u201d was designated on the racing team to ensure continuous advocacy of best practices.<\/p><\/div>

      Results and Impact (with Metrics)<\/strong><\/p><\/div>

      The Real Time Telemetry Analytics initiative delivered clear improvements for McLaren Racing:<\/strong><\/p><\/div>

      Comprehensive Observability:<\/strong> McLaren now has a powerful, unified view of its racing data ecosystem. The Splunk platform streams and analyzes telemetry at 100kHz (100,000 data points per second) for real time decision making, covering everything from engine performance to IT infrastructure status. This resulted in unprecedented visibility as one McLaren tech lead put it, \u201cwe\u2019ve never had this level of insight before\u201d. The team can monitor car health and systems health side by side, ensuring no blind spots during critical operations.<\/p><\/div>

      Improved Reliability & Faster Issue Resolution:<\/strong> The anomaly detection system has led to a marked improvement in reliability and response times. Critical anomalies (whether a degrading sensor, a server memory leak, or network latency spike) are now caught within seconds and immediately flagged to the team. During the last season, there were multiple instances where early warnings from the system allowed McLaren to address issues during pit stops or between sessions avoiding on track failures that could have cost points. The operations and IT infrastructure achieved greater consistency and uptime, as evidenced by the fact that McLaren experienced zero telemetry related outages during races after the system\u2019s implementation. Any issues in the telemetry pipeline were proactively identified and resolved by the monitoring Kermit put in place.<\/p><\/div>

      Data Driven Performance Gains:<\/strong> With high quality data readily available, McLaren\u2019s race strategists and engineers could make more informed decisions. For example, data insights from Splunk helped optimize engine tuning and tire strategy by correlating real time telemetry with historical race data. The team credited the platform with helping accelerate their development feedback loop: car setup changes and updates could be evaluated faster across thousands of data points, contributing to improved lap times and consistency. While competitive advantage is hard to quantify, McLaren\u2019s leadership has publicly noted that data analytics is a key part of their race to race improvements.<\/p><\/div>

      Efficient Log Triage and Team Productivity:<\/strong> The integration of the NLP driven alert triage reduced the volume of alerts engineers had to manually review by ~40%. Instead of wading through a flood of messages, they now receive a concise summary of the most critical issues. This efficiency means engineers spend less time in war room debugging, and more time implementing solutions. It also reduces stress and cognitive load during races team members can trust the automation to catch important issues, allowing them to focus on strategy and car performance.<\/p><\/div>

      Team Empowerment and Culture Shift:<\/strong> By training and upskilling the team on observability tools, Kermit helped cultivate a more data driven culture in McLaren\u2019s operations. Engineers who traditionally might focus only on mechanical aspects are now comfortable diving into telemetry dashboards to hunt for improvements. The user friendly visualizations and the successes of the system have made data a central part of race debriefs and planning. McLaren\u2019s management noted that this project has \u201csparked curiosity and innovation across the organization,\u201d encouraging experimentation and continuous improvement. In effect, the platform not only delivered technical benefits but also influenced the way the team approaches problem solving with data at the forefront.<\/p><\/div>

      Recognition and Partnership Value:<\/strong> This project strengthened the McLaren\u2013Splunk partnership. McLaren\u2019s case became a flagship example for Splunk (featured in Splunk\u2019s customer success stories), and it validated Splunk\u2019s technology in one of the most demanding real time data scenarios. For Kermit, his work received recognition from both McLaren\u2019s leadership and Splunk executives, enhancing his reputation as an expert in real time analytics solutions.<\/p><\/div>

      Lessons Learned<\/strong><\/p><\/div>

      Working on F1 level real time analytics provided several key lessons:<\/strong><\/p><\/div>