The AI roadmap is taking shape

Drive Thor targets the 2025 production models, in which more automated and assisted driving will be included. It is Nvidia’s latest in-vehicle computing platform and the successor to Drive Orin.

The SoC uses the company’s next-generation AV processor, based on the Blackwell architecture that is designed to handle large (‘data centre-scale’) generative AI workflows, including large language models (LLMs) energy-efficiently.

It integrates the functionality of what would have been six separate SoCs – namely, parking, AV and activity safety, driver and occupant monitoring, cybersecurity management, a cluster display and an infotainment SoC. This brings greater efficiency in terms of development and energy use as well as faster software iteration, says Nvidia.

Among the milestones for this automotive-grade SoC is the inclusion of a transformer engine, a new component of the company’s GPU tensor core. By processing video data as a single perception frame, the transformer networks are able to process more data over time to contribute to automated or autonomous driving systems.

In addition to the new transformer engine, Nvidia has increased the speed and width of its NVLink interconnects. There is also a new decompression engine and Spark RAPIDs libraries for data analytics.

The SoC has an 8-bit floating point (FP8) precision for what Nvidia says is a new data type for automotive. “Traditionally, AV developers see a loss in accuracy when moving from 32-bit FP to 8-bit integer data formats. FP8 precision eases this transition, making it possible for developers to transfer data types without sacrificing accuracy.”

As a result, the SoC supports 1,000 INT8 tera operations per second (top/s) or 1,000 FP8 tera flop/s or 500 F16 tflop/s. Manufacturers can divide this performance in one of two ways – dedicated to the vehicle’s autonomous driving systems or divided between the in-cabin AI and driver assistance systems.

The AI cockpit

An AI cockpit enables software-defined, in-vehicle capabilities, which can learn and become smarter over time. An example is the MBUX AI system. It is powered by Drive AGX and Drive IX software. Mercedes-Benz has introduced this in the S-Class model, which was announced earlier this year.

“The Mercedes-Benz user experience of tomorrow will be hyper-personalised. With generative AI, our MBUX virtual assistant brings more trust and empathy to the relationship between car and driver. Thanks to our MB.OS chip-to-cloud architecture, our future vehicles will provide customers with exactly what they need when they need it,” says Magnus Östberg, chief software officer, Mercedes-Benz.

An intelligent system can be activated without having to say keywords (for example, “Hey, Mercedes”). This level of natural speech uses LLMs for conversation based on general knowledge and the predictive assistant can offer intelligent responses based on situational context, learnt behaviour and using generative AI.

There can also be visual feedback with 3D graphics to indicate if the assistant is thinking about a question, making a suggestion or issuing a warning.

The Surround Navigation feature takes information from sensors around the car to provide a real-world view of the vehicle’s surroundings with real-time interactions. 3D graphics in the driver display can show the driver when another car, a cyclist or pedestrian may be close by or approaching. The same graphics are used for navigation, superimposed on a rendering of the surrounding environment.

Interior cameras monitor the driver to track activity, head position and facial movements and analyse if the driver is paying attention. If it detects that the driver is drowsy or distracted it can provide an alert to redirect attention.

Sensors can also be used to protect passengers and other road users. For example, an intelligent system can sense if a passenger is preparing to get out and exterior sensors can monitor for, and warn about, oncoming traffic or pedestrians within range.

An intelligent system can also detect if a person is sitting properly in their seat in the event of an emergency and prevent an airbag being activated that could harm them.

Anticipating 2025 models

“Accelerated compute has led to transformative breakthroughs, including generative AI, which is redefining autonomy and the global transportation industry at large,” says Xinzhou Wu, Nvidia’s vice-president of automotive. Drive Orin is still available, but is more focused on AI in vehicles, whereas Drive Thor is being considered for AI-enabled vehicles, he explains.

Chinese EV manufacturer BYD says it will use the computing platform in its next models and GAC Aion, part of the GAC group, has announced that its Hyper EV models will feature a centralised AI computing system. The company is currently using Drive Orin in its flagship model, Hyper GT, which has Level 2+ automated driving capabilities in high-speed environments.

Another Chinese manufacturer, Xpeng, has also confirmed that it will use Drive Thor for AI-assisted driving, autonomous driving and parking and for driver and passenger monitoring (for tiredness, alert states and comfort).

Earlier this year, at CES, Li Auto, a Beijing-based EV manufacturer, announced that it will be using Drive Thor in its next-generation, extended range, EVs. The company currently uses two Drive Orin processors to power its assisted-driving system, AD Max, for its L-series models. These processors provide a combined 508 top/s, sufficient for real-time fusing and processing of sensor information to power autonomous driving for navigation and assistance driving functions such as lane change control, automated parking and active safety features, for example, automatic emergency braking.

Other manufacturers include the US autonomous delivery vehicle company Nuro, which has selected Drive Thor to power its Nuro Driver autonomous driving system. This uses the company’s proprietary AI-first software and sensors paired with Nvidia automotive-grade compute and networking hardware. The system will begin testing later this year.

For autonomous driving, Plus will deliver its Level 4 SuperDrive on Drive Thor for its trucks, and Canadian company Waabi is to use Drive Thor for its first generative AI-powered Waabi Driver, which combines its proprietary generative AI autonomy software stack with sensors and Drive Thor computing.

There is also WeRide, a Chinese autonomous driving technology company, which has partnered with Lenovo Vehicle Computing to develop Level 4 autonomous driving systems for commercial applications built on Drive Thor and integrated in Lenovo’s first autonomous driving domain controller AD1. Lenovo has also introduced a new AI acceleration engine, UltraBoost, which has an AI model engine and AI compiler tool chains that will be used to deploy LLMs within vehicles and which will run on Drive.

Learning generative AI

LLMs (large language models) are a form of generative AI. These deep- learning architectures, or transformer models, are neural networks that learn context and meaning. Traditional or multi-modal LLMs primarily process and generate text-based data, but there are also vision language models. These are also generative AI, but based on image processing and language understanding capabilities and are used to analyse and generate text via images or videos.

A third category is retrieval-augmented generation, which allows vehicle manufacturers to access knowledge from a specific database or the web to assist drivers.

Generative AI operates with Nvidia Avatar Cloud Engine, multi-modal language models and Drive to allow vehicle manufacturers to develop intelligent in-car virtual assistants. This can extend to designing avatars with customised voices and emotional attributes for a natural dialogue to provide real-time assistance and personalised interactions.

When combined with AI-enhanced surround visualisation using a 360° camera, such an intelligent assistant could provide local driving regulations or conditions to make driving safer.

The next step is anticipated by British startup Wavye. At GTC it announced a generative world model with an AI foundation model that learns to drive self-supervised “using AI end-to-end – from sensing, as an input, to outputting driving actions”.