Documentation of Safebench

Overview

• Safebench Components
• Challenge Tracks
  • Track 1 (Perception Attack)
  • Track 2 (Perception Defense)
  • Track 3 (Planning Attack)
  • Track 4 (Planning Defense)
• Reference

Getting Started

• Installation
  • Step 1. Setup Safebench
  • Step 2. Setup Carla
  • Potential Issue
• Run Example
  • Option 1. Desktop Users
  • Option 2. Remote Server Users

Submission Guidance

• How to Create Agent
  • Create Perception Agent
  • Create Planning Agent
• How to Create Scenario
  • Create Perception Agent
  • Create Planning Agent
• Hosting a Challenge via EvalAI
  • Step 1: Set Up Main Repo
  • Step 2: Set Up Challenge Configuration
  • Step 3: Define Evaluation Code
  • Step 4: Edit Challenge HTML Templates
• How to Submit Results

Functions and API

• Modules and Functions
  • Carla Runner (safebench/carla_runner.py)
  • Gym module (safebench/gym_carla/)
  • Agent Module (safebench/agent/)
  • Scenario Module (safebench/scenario/)