To accelerate in one of the fastest “racing line” without losing control, race car drivers have to be able to brake, steer, and accelerate precisely scheduled sequences. The procedure is based on the frictional limits and is governed by the laws of Nature, which means that autonomous cars can learn to complete a race at the speed that is fastest (as many have already achieved). However, it becomes a difficult problem when the autonomous driver needs to share the space with other vehicles. Scientists have solved the problem by preparing an artificial-intelligence program that can outdo human players in the ultra-realistic race video game Gran Turismo Sport. The results could lead auto-driving vehicle researchers to different methods to help the technology work in real life.
Artificial Intelligence has dominated humans in a few video games, including Starcraft II and Dota 2. However, Gran Turismo differs from other games in several ways, according to Peter Wurman, director of Sony AI America and co-author of the study that is published in the journal Nature. “In most games, the environment defines the rules and protects the users from each other,” Wurman says. “But when racing the cars are close to one another, and there’s a sophisticated manner of conduct that needs to be mastered and implemented by the agents of AI. To win it is essential that they respect their opponents However, they have to maintain their own lines of driving and ensure they don’t let them surrender to the opposition.”
To help their program learn how to behave, Sony AI researchers used a method known as deep reinforcement. They gave rewards to their AI for certain behavior like keeping trackside as well as being at the helm of its car and adhering to the rules of racing. Then, they let the program free to explore different methods of racing, which would allow it to meet those objectives. It was Sony AI team trained multiple different versions of its AI, named Gran Turismo Sophy (GT Sophy) which are each focused on driving a specific type of car on a specific track. The researchers then pitted the AI against humans as Gran Turismo champions. In the initial test that was conducted in July of last year, humans were the top team overall score. In the second test, which took place in October 2021 the AI made its mark. It defeated its human adversaries both on its own as well as in a group and set the fastest times for laps.
Human players appear to take their losses in the right direction, and others enjoyed competing with the computer. “Some of the things that we also heard from the drivers was that they learned new things from Sophy’s maneuvers as well,” says Erica Kato Marcus, director of partnerships and strategies for Sony AI. “The algorithms that the AI used were so complicated that I would probably only be able to try them at least once. It was extremely difficult that I would never try it in an actual race.” adds Emily Jones who was a world-finalist at the FIA-Certified Gran Turmo Championships in 2020. She later competed in a GT Sophy race. While Jones claims that competing against the AI caused her to feel a bit insignificant, she describes the experience as awe-inspiring.
Cars in Gran Turismo Sport. Credit: Sony Interactive Entertainment
“Racing, like a lot of sports, is all about getting as close to the perfect lap as possible, but you can never actually get there,” Jones states. “With Sophy, it was amazing to be able to observe an event that seemed to be exactly the ideal lap. It was impossible to get faster.”
The Sony team is currently working to improve the AI. “We trained an agent, a version of GT Sophy, for each car-track combination,” Wurman declares. “And one of the things we’re looking at is: Can we train a single policy that can run on any car on any of the tracks in the game?” On the commercial side, Sony AI is also working with the developers of Gran Turismo, the Sony Interactive Entertainment subsidiary Polyphony Digital, to potentially include a variant that is based on GT Sophy into a future version of the game. To achieve this, the researchers will need to modify the AI’s performance so that it could be a tough opponent, but it is not invincible even for players who aren’t as skilled as the top players who have tried the AI so far.
Because Gran Turismo offers a realistic simulation of certain tracks and cars–as well as the specific physical parameters that govern them–this research could also be useful beyond video games. “I think one of the pieces that’s interesting, which does differentiate this from the Dota game, is to be in a physics-based environment,” claims Brooke Chan, a software engineer with the research firm for artificial intelligence OpenAI as well as co-author on the OpenAI Five project that beat human players in Dota 2. “It’s not out in the real world but still is able to emulate characteristics of the real world such that we’re training AI to understand the physical world a little bit more.” (Chan wasn’t involved in this GT Sophy study. )
“Gran Turismo is a very good simulator–it’s gamified in a few ways, but it really does faithfully represent a lot of the differences that you would get with different cars and different tracks,” claims J. Christian Gerdes, an Stanford University professor of mechanical engineering, who wasn’t involved in the latest study. “This is, in my mind, the closest thing out there to anybody publishing a paper that says AI can go toe-to-toe with humans in a racing environment.”
However, not everyone is in agreement but there is a consensus. “In the real world, you have to deal with things like bicyclists, pedestrians, animals, things that fall off trucks and drop in the road that you have to be able to avoid, bad weather, vehicle breakdowns–things like that,” says Steven Shladover who is a researcher engineer with the California Partners for Advanced Transportation Technology (California PATH) program at the University of California, Berkeley’s Institute of Transportation Studies, who was not part of this Naturepaper. “None of that stuff shows up in in the gaming world.”
However, Gerdes states that GT Sophy’s successes is still relevant since it challenges certain notions about how autonomous vehicles must be programmed. Automated vehicles can take decisions based on physical laws or it’s AI training. “If you look at what’s out there in the literature–and, to some extent, what people are putting on the road–the motion planners will tend to be physics-based in optimization, and the perception and prediction parts will be AI,” Gerdes states. In the case of GT Sophy, however, the motion-planning algorithm of the AI (such as the decision to maneuver around a corner in the highest level of its capabilities without creating crashes) was dependent on the AI part of the equation. “I think the lesson for automated car developers is: there’s a data point here that maybe some of our preconceived notions–that certain parts of this problem are best done in physics–need to be revisited,” he adds. “AI might be able to play there as well.”
