First steps with CARLA

The CARLA simulator is a comprehensive solution for producing synthetic training data for applications in autonomous driving (AD) and also other robotics applications. CARLA simulates a highly realistic environment emulating real world towns, cities and highways and the vehicles and other objects that occupy these driving spaces.

The CARLA simulator is further useful as an evaluation and testing environment. You can deploy the AD agents you have trained within the simulation to test and evaluate their performance and safety, all within the security of a simulated environment, with no risk to hardware or other road users.

In this tutorial, we will cover a standard workflow in CARLA, from launching the server and connecting the client, through to adding vehicles, sensors and generating training data to use for machine learning. This tutorial is meant to be light on details and go as efficiently as possible through the key steps in using CARLA to produce machine learning training data. For further details on each part of the workflow, such as the multitude of vehicles available in the blueprint library or the alternative types of sensors available, please consult the links in the text or browse the left menu.

• Launching CARLA
• Loading a map
• Spectator navigation
• Adding NPCs
• Add sensors
• Animate vehicles

Launching CARLA and connecting the client

CARLA can be launched using the command line using the executable in Windows or the shell script in Linux. Follow the installation instructions for Linux and Windows then launch CARLA from the command line:

cd /carla/root
./CarlaUE4.sh

To manipulate CARLA through the Python API, we need to connect the Python client to the server through an open port. The client controls the simulator through the client and world objects Open a Python notebook or create a new script, then add the following code to the start of the script or the main function:

import carla
import random

# Connect to the client and retrieve the world object
client = carla.Client('localhost', 2000)
world = client.get_world()

The client object serves to maintain the client's connection to the server and has a number of functions for applying commands and loading or exporting data. We can load an alternative map or reload the current one (resetting to initial state) using the client object.

The port can be chosen as any available port and is set to 2000 by default, you can also choose a host different from localhost by using a computer's IP address. This way, the CARLA server can be run on a networked machine, while the python client can be run from a personal computer. This is particularly useful for differentiating the GPU used for running the CARLA simulator and that used for neural network training, both of which can be highly demanding on graphics hardware.

Loading a map

In the CARLA API, the world object provides access to all elements of the simulation, including the map, objects within the map, such as buildings, traffic lights, vehicles and pedestrians. The CARLA server normally loads a default map (normally Town10). If you want to launch CARLA with an alternate map, use the config.py script:

./config.py --map Town05 

We can also use the world object to load a map from the client:

client.load_world('Town05')

Please find more information about CARLA maps here.

Spectator navigation

The spectator is a view into the simulation. By default, the spectator opens in a new window when you run the CARLA server on a computer with a screen attached, unless you specify the -RenderOffScreen command line option.

The spectator is helpful to visualize your simulation. Using the spectator, you can familiarize yourself with the map you've loaded, and see the result of any changes you are making, such as adding vehicles, changing the weather, turning on/off various layers of the map and for debugging purposes.

You can fly the spectator around the world using the mouse to control the pitch and yaw of the spectator view and the QWE-ASD keys to move the spectator:

• Q - move upwards (towards the top edge of the window)

• E - move downwards (towards the lower edge of the window)

• W - move forwards

• S - move backwards
• A - move left
• D - move right

Left click and drag the mouse in the spectator window up and down to control pitch and left and right to control yaw.

The spectator and its properties can be accessed and manipulated through the Python API:

# Retrieve the spectator object
spectator = world.get_spectator()

# Get the location and rotation of the spectator through its transform
transform = spectator.get_transform()

location = transform.location
rotation = transform.rotation

# Set the spectator with an empty transform
spectator.set_transform(carla.Transform())
# This will set the spectator at the origin of the map, with 0 degrees
# pitch, yaw and roll - a good way to orient yourself in the map

Adding NPCs

Now we've loaded the map and the server is up and running we now need to populate our simulation with some vehicles to simulate a real environment with traffic and other road users or non-player characters (NPCs).

To spawn vehicles, first, we need to select the vehicles we want from the blueprint library.

# Get the blueprint library and filter for the vehicle blueprints
vehicle_blueprints = world.get_blueprint_library().filter('*vehicle*')

Now we have the blueprints, we need to find some appropriate spots in the map to spawn our vehicles. Each CARLA map provides pre-defined spawn points spread evenly throughout the map on the roads for this purpose.

# Get the map's spawn points
spawn_points = world.get_map().get_spawn_points()

# Spawn 50 vehicles randomly distributed throughout the map 
# for each spawn point, we choose a random vehicle from the blueprint library
for i in range(0,50):
    world.try_spawn_actor(random.choice(vehicle_blueprints), random.choice(spawn_points)))

Now we should also add a vehicle that will be the centerpoint of our simulation. To train an autonomous agent we need to simulate a the vehicle that it the autonomous agent will control. In CARLA parlance, we often refer to this vehicle as the "Ego vehicle".

ego_vehicle = world.spawn_actor(random.choice(vehicle_blueprints), random.choice(spawn_points))

In addition to vehicles, CARLA also provides pedestrians to add to simulations to simulate realistic driving scenarios. Vehicles and pedestrians are referred to as actors in the CARLA parlance, learn more about them here.

Add sensors

Modern autonomous vehicles understand and interpret their environment through an array of attached sensors. These sensors include things such as optical video cameras, optical flow cameras, LIDARs, RADARs and accelerometers. CARLA has models of numerous types of sensors built in to create training data for machine learning. The sensors can be attached to a vehicle, or they can be attached to a fixed point to model for example a CCTV camera.

Here we will attach a standard camera sensor to the ego vehicle to record some video data:

# Create a transform to place the camera on top of the vehicle
camera_init_trans = carla.Transform(carla.Location(z=1.5))

# We create the camera through a blueprint that defines its properties
camera_bp = world.get_blueprint_library().find('sensor.camera.rgb')

# We spawn the camera and attach it to our ego vehicle
camera = world.spawn_actor(camera_bp, camera_init_trans, attach_to=ego_vehicle)

Once we have spawned the camera, we need to set it recording through the listen() method. The listen method takes as argument a callback that defines what to do with the data. You can either stream it to another program or save it to disk.

We will use a lambda function as a callback to save the data to disk:

# Start camera with PyGame callback
camera.listen(lambda image: image.save_to_disk('out/%06d.png' % image.frame))

This will save the data to the out/ folder as a series of PNG image files named according to the simulation frame number.

There are a multitude of different types of sensors to choose from. Here you can delve deeper into the array of sensors available and how to use them.

Animate vehicles with traffic manager

Now we've added our traffic and ego vehicle to the simulation and started recording camera data, we now need to set the vehicles in motion using the Traffic manager. The Traffic manager is a component of CARLA that controls vehicles to autonomously move around the roads of the map within the simulation, following the road conventions and behaving like real road users.

We can find all the vehicles in the simulation using the world.get_actors() method, filtering for all the vehicles. We can then use the set_autopilot() method to hand over control of the vehicle to the Traffic Manager.

for vehicle in world.get_actors().filter('*vehicle*'):
    vehicle.set_autopilot(True)

Now your simulation is running, with numerous vehicles driving around the map and a camera recording data from one of those vehicles. This data can then be used to feed a machine learning algorithm for training an autonomous driving agent. The Traffic manager has many functions for customising traffic behaviour, learn more here.

This is the most basic possible set up for a simulation, now you can go into further details deeper into documentation about the many extra sensors you can use to generate data, and the many other features of CARLA that can make your simulations more detailed and more realistic.