Third Place Solution and Experience in 2020 ROAR S1 Series Game
Written by Michael Wu, Flaviano Christian Reyes, James Cheney, Marleah Puckett, and Jonathan Wong describing their experience and solution for their EECS 106A/206A final project at UC Berkeley during Fall 2020.
Table of Contents
Introduction
This project is part of a larger research project called ROAR. ROAR stands for Robot Open Autonomous Racing and is Berkeley’s first AI race car competition. Our project is primarily concerned with making racing agents in the virtual environment. In the future, once we finalize our algorithms through computer simulations, we can port our algorithms to actual cars and try out strategies in reality. For a more detailed overview, please visit our project page.
For this project, we built four agents that use different control strategies. These controllers include:
- A traditional PID controller
- A gym-tuned PID controller where K values were determined by a learned policy with current state as input
- A controller with latitudinal control via PID and longitudinal control via roll calculated using depth camera data
- A traditional Stanley controller
We test these agents using Carla, an autonomous driving software that acts as a framework for testing driving algorithms, and our results are reported in the results section of this website. Furthermore, we were able to test the traditional PID controller agent in a physical car at Richmond Field Station.
Our project also implements and showcases a calibration of the ROAR car, which is ultimately helpful for the next step in the ROAR research pipeline since we would like our live trials to match up with our simulations (i.e. if we tell a car to go straight, it goes straight, just like in a virtual simulation).
In the end, our project not only contributes (1) different types of agents for students and researchers to compare to and (2) improves the simulation-to-reality transition, but we also:
- Set up the Vive Tracker system in Richmod Field Station for GPS/State messages
- Built a gym environment for researchers in ROAR to train their RL agents
These contributions were necessary for us to develop our virtual agents and eventually test them in real life.
Goal of Project
Our end goal was to achieve the fastest lap, whether that be in physical or virtual simulations. For the purposes of this project and the ROAR race this semester, we were primarily focused on getting the fastest lap time in the virtual race simulation that was open to students. We sent our RL PID agent to represent our group in the race because at the time, this was our fastest agent based on our experiments.
Motivation and Questions behind Project
This is an interesting project for a number of reasons. First of all, the field of autonomous driving is a hot topic in research today, and companies have invested a lot of time and money into creating reliable and performant systems that could be used in production level environments. The public as a whole has not embraced automous driving strategies due to (but not limited to) (1) saftey concerns and (2) economic impact (automous vehicles could take away jobs). Although our research group is not actively solving this latter systemic issue, we are aware of the consequences of self-driving vehicles.
What our project is primarily concerned with is speed and reliability, which relates to the safety concerns that the general public has about autonomous vehicles and the performance goals that car companies seek. We want to answer the following question: how do we drive as fast as possible and avoid obstacles or collisions? From this main question comes related subproblems such as:
- How do we know where to go? (A sensing and planning problem)
- How do we get from point A to point B? (A sensing, planning, and actuation problem)
For the purposes of our project, we drive around the virtual and physical environment and collect our vehicle’s coordinates along the way. These waypoints give us the path we need to traverse. And in regard to the second question, we implement our variety of controllers and see which one yields the fastest lap without colliding into other objects.
Real-World Robotics Application
Our project centers around autonomous vehicle research, which car companies have invested time and money in to develop production level systems. Our project work could be useful in figuring out which autonomous vehicle strategies to use in production, but utlimately, its up to others to figure out how they could use our work. For instance, our project could be useful in figuring out autonomous driving capabilities within personal cars, or it could be useful for deploying a feet of autonomous rideshare or delivery vehicles. In the end, it’s all about imagination!
Project Overview
For this project, we built four agents that use different control strategies. The path that each car is following is an alloted set of way points collected from driving a car around a track. For virtual simulations, we use the location data from Carla. For real life simulations, we installed a VIVE tracking system in Richmond Field Station to keep track of where the car is on the track.
Traditional PID Controller
The Real PID controller was difficult to implement, but works very well. Through testing, laptime was drastically decreased and the car didn’t run into any barriers (although there was one close call). Given the challenges we faced dealing with the physical car, we are very pleased with the output of the Real PID.
One interesting observation is that the car was found to speed up at the start of its lookahead length and then slow down until the end of its lookahead length. We implemented a short lookahead since it was beneficial around corners, reducing crashes and creating smoother movement. However, this behavior, combined with a short lookahead length made the car perform poorly on straight runs that can handle faster speeds. To improve the Real PID controller in the future, the lookahead could be improved so that there is a longer lookahead on straight paths that can handle increased speeds. Likewise, as a driver it is known that it is desirable to enter a curve slowly and then speed up while exiting the curve and, when done in reverse, this could be detrimental. Improving the system so that the car never enters a curve at its fastest speed would also be beneficial.
Results shown below. Here is a demo of our PID controller at Richmond Field Station.
Gym-Tuned PID Controller
The video demonstrates an agent using the model trained using observation of current steering, throttle, and speed. We made our own gym environment that adapts well to our existing ROAR codebase and the Carla autonomous driving software that we use for prototyping.
It is worth noting that the trajectory of the vehicle is obviously not smooth, however, it never collides. We think that it is due to the extremely high penalty that we have imposed on collision and therefore the model has learnt to not collide. We suspect that the shaking is due to the fact that the waypoint being read in is not smooth, or in other words, the trajectory we passed in is not exactly smooth.
It is also worth noting that the agent learnt to increase its speed on straight lines and decrease speed at turns. That is a desired behavior because as humans, we would also opt for such actions.
Here is a demo of our RL PID agent.
Roll Controller
Our roll controller agent uses a PID controller for latitudinal control and uses the roll of the car for longitudinal control. Our natural driving intution is that we don’t want to skid off the side of the road, so we slow down when approaching curves and accelerate when getting out of them. Using this intutiion, the car will slow when going around curves and speed up when going out of them by factoring in the roll of the car. Furthermore, we use a sparse flood fill algorithm on depth data to sense the ground plane, and based on the normal of the ground plane (the road), we calculate the roll.
The Roll controller works better than expected, and was used to consistently set the fastest lap times. There are plans to modify and experiment with the controller to improve upon the design. One plan is to add a look ahead than adjust speed downward when the heading difference between the vehicle and the desired path is increasing. This indicates the vehicle is approaching a turn. In this way the vehicle can slow in anticipation of turning, and accelerate to maximum allowed roll as the road straightens. This will allow for more aggressive speeds at all points as the vehicle will not enter a turn at full speed and lose control.
Here is a demo of Roll controller agent.
Traditional Stanley Controller
While the Stanley controller works most of the time, it does not work reliably, in that it has uncovered a situation where the next waypoint provided by the system does not always update. Because the Stanley controller uses both heading and crosstrack error as inputs to the steering command, the misinformation provided by the expired waypoint sometimes causes the heading error to cancel the crosstrack error, resulting in a near zero steering command in a turn. This causes the vehicle to proceed straight to the outside edge of the curve. We plan to alleviate this problem by modifying the code to cycle through the way point queue to the next available whenever it is calculated the waypoint is behind the vehicle. The controller seems to work fine other than this glitch.
Here is a demo of Stanley controller agent.
Performance Breakdown
Here is a table showing the performance of our racing agents and their associated controllers
| Agent | Controller | Lap Time (Seconds) | Lap Time (Min:Sec) |
|---|---|---|---|
| 0 | PID | 121.8162 | 2:01 |
| 1 | RL PID | 119.6559 | 1:59 |
| 2 | ROLL | 103.2749 | 1:43 |
| 3 | STANLEY | None | None |
Calibration
Our calibration ends up being a success, being able to drive straight when issued our new command of 1554 on steering. This will be our new “straight” command.
Results shown below.
Discussion and Next Steps
Discussion of Results
By looking at the performance breakdown of our racing agents and their controllers, it is clear that the Roll controller produced the fastest lap time.
Interestingly, our RL PID controller and traditionally tuned PID controller produced very similar results indicating that our RL PID agent was trained incredibly well as it produced a very similar outcome to that of a human setting the k values through traditional control methods.
The Roll controller was the big winner of our racing agents, having an 18.54 second faster lap time than the traditionally tuned PID controller.
Awards
Out of 7 teams, our project ended up placing 3rd in the ROAR race. We also finished with the Reliability Award, being the only team with 0 collisions. This absolutely meets our design criteria since we achieved the fastest possible lap time without running into barriers or other cars.
Difficulties with Project
On the physical car, we are using Vive Tracker for our localization service. However, the calibration is a bit unintuitive. For example, the goggles had to be placed in a specific direction and the Vive Tracker had to be carefully placed on the vehicle or else the pitch of the car would be off. The Vive Tracker was also seen as “lost” a few times which caused unrecoverable runtime crashes from Steam/OpenVR that required a manual restart of the server.
On the physical car, in terms of vehicle safety and throttle limits, we ended up setting the throttle boundary from (0,1) instead of the traditional (-1,1). This is because we ran into issues when the vehicle was going very fast and the PID controller instructed the vehicle to go in reverse to slow down immediately. Going in reverse is not optimal when trying to go as fast as possible. Since our car is only needed to move forward, this was found to be fine, however if our vehicle needed to ever move in reverse, this would not be a good thing to do.
There were various times where our physical car hit a barrier during testing at a very high speed and damaged the car! To stop this from happening, we set a limit on the motor RPM and maximum speed of the vehicle. After testing, we slowly increased these values until the desired limit was reached for the car to go around the track quickly without flying off the track on the curves.
Flaws and Hacks
Although our solution performed very well in the ROAR race, there are some flaws with our solution that could be investigated/improved upon. Although our Roll controller was our fastest racing agent, with a 18.54 second faster lap time than the traditionally tuned PID controller, remarkably enough, the Roll controller is not a tried and tested system. It was an innovation of our group to use the roll control as a way to output the throttle. This means that, on a real track, with sliding and disturbances, the Roll controller could not perform as well as our simulation suggests. Additionally, there are some hacks for speeding up the roll calculation such as using the horizon, sparse sampling, KNN, and execution of calculation at certain time steps. We need a more robust algorithm to speed up the calcultions.
Next Steps
If we had more additional time, we would love to bring our simulation to life. At the Richmond Field Station, we had the opportunity to test our physical car on a flat track with three tight curves. However, if we could test our physical car on a new track with more disturbances (e.g. inclines, rain, etc.), we could better replicate driving in the real-world and continue fine-tuning our PID k values.
The collection of waypoints that are fed into our physical car could also be improved. This is because our waypoints were collected by manually driving the car around our created track, introducing significant human error (e.g. we didn’t drive exactly straight on a straight path).
When calculating the next desired position of the car, the lookahead value is very important. We set the lookahead value to be relatively small to reduce crashes on curves and create a smoother path overall. The lookahead value was also set to be constant throughout the car’s race. However, this can cause the car to do some unnecessary calculations when, for example driving fast on a straight path. So another improvement we could make to reduce lap time is to smartly adjust the lookahead based on different track scenarios and velocities.
Next steps for the Stanley Controller are to implement the proposed solution and see if it fixes the problem with the waypoint not updating and causing the controller to “stall out”. The proposed fix is to skip any waypoint that has “expired” – eg is behind the vehicle. The next waypoint in the queue will be accessed and this cycle will continuously repeat until a waypoint in front of the vehicle is found.
The RollControll has room for improvement. A look ahead can be added to detect when a turn is coming. This could be done by taking the rate of change of the heading error, and when it is increasing and greater than a given rate, the throttle will be lowered proportionally. When the look-ahead heading error is zero or decreasing, acceleration can be resumed according to the RollControll algorithm.
Finally, the various controllers can all be tuned in the gym to maximize performance. They can also be tested in the physical car, to test cross platform viability.