The automotive industry has taken a significant step forward in the development of a benchmark suite for AI systems in vehicles.
MLCommons and the Autonomous Vehicle Computing Consortium (AVCC) have announced the successful unveiling of the first phase of the MLPerf Automotive benchmark proof-of-concept (POC), aimed at enhancing the benchmark landscape for AI systems in vehicles. Developed by the Automotive benchmark task force (ABTF), this marks the initial stage in the creation of a comprehensive benchmark suite for AI systems in vehicles. Key industry players, including Arm, Bosch, Marvell, Samsung, and others, are actively partaking in this innovative initiative.
The increasing demand for AI-based systems in vehicles goes beyond fully autonomous cars, as auto manufacturers are integrating AI into various in-car systems to enhance the driving experience and ensure vehicle safety. These systems include speech-controlled infotainment, route guidance, collision avoidance, and driver monitoring, all of which require trained AI models, appropriate input sensors, and robust computing infrastructure to meet performance requirements.
The release of the POC is a significant milestone for the automotive industry, allowing stakeholders to provide feedback and establish a comprehensive benchmark suite. The full v1.0 MLPerf Automotive Benchmark Suite is scheduled for release by the end of 2024 and will cater to various AI components within automotive systems, ensuring industry requirements are met.
Additionally, the POC includes a fully operational reference implementation for a camera-based object detection capability, essential for collision-avoidance and autonomous driving systems. The model is tailored to recognize 8-megapixel images, aligning with the emerging camera-based systems in vehicles for long-term viability.
To validate the rigour and strength of the benchmark suite, the POC includes a small subset of the MLCommons Cognata dataset containing 120,000 8-megapixel images for training the model. The use of synthesized images allows for the inclusion of scenarios that are too dangerous to replicate in reality, emphasizing the commitment to delivering a robust benchmark suite for AI systems in vehicles.
The release of the POC presents an opportunity for further advancements in benchmarking AI systems in automotive environments. Stakeholders can download the Automotive POC, evaluate it, and contribute feedback through the ABTF working group, demonstrating a dedication to creating the best performance benchmark suite for AI systems in vehicles.
For further information on AVCC, please visit www.avcc.org. For additional details on MLCommons and membership inquiries, please visit MLCommons.org. For press inquiries, please contact Kelly Berschauer at [email protected] or Sarah Laliberte at [email protected].