init

Author: Y-vic

README.md

@@ -1 +1,7 @@
-# Nexus
+# Nexus
+
+## Getting Started
+- [Installation](docs/install.md)
+- [Prepare Dataset](docs/prepare_dataset.md)
+- [Train and Eval](docs/train_eval.md)
+

docs/install.md

@@ -0,0 +1,71 @@
+# Installation
+
+## Setup conda environment
+
+    ```python
+    conda create -n nexus python=3.9
+    conda activate nexus
+
+    git clone https://github.com/OpenDriveLab/Nexus.git && cd Nexus
+    pip install -r requirements.txt
+    ```
+
+## Install Nuplan Devkit
+
+1. Navigate to the `third_party` directory:
+   ```bash
+   cd Nexus/third_party
+   ```
+
+2. Clone the Nuplan Devkit repository:
+   ```bash
+   git clone https://github.com/motional/nuplan-devkit.git
+   ```
+
+3. Install the Nuplan Devkit:
+   ```bash
+   cd nuplan-devkit
+   pip install -e .
+   ```
+
+4. Set the `NUPLAN_DEVKIT_PATH` environment variable:
+   ```bash
+   export NUPLAN_DEVKIT_PATH=$YOUR_PATH_TO_Nexus/Nexus/third_party/nuplan-devkit
+   ```
+5. install following packages:
+   ```bash
+   pip install aiofiles aioboto3 flatdict adjustText loralib easydict einops_exts
+   pip install waymo-open-dataset-tf-2-12-0==1.6.4
+   ```
+
+## Install ALF
+
+Follow these steps to install ALF while ignoring the dependencies of torch:
+
+1. Navigate to the `third_party` directory:
+
+    ```bash
+    cd Nexus/third_party
+    ```
+
+2. Clone the ALF repository:
+
+    ```bash
+    git clone https://github.com/HorizonRobotics/alf.git
+    ```
+
+3. Edit the `setup.py` file to ignore the dependencies of torch:
+
+    Comment out lines 52, 53, and 54 in the `setup.py` file:
+
+    ```python
+    # 'torch==2.2.0',
+    # 'torchvision==0.17.0',
+    # 'torchtext==0.17.0',
+    ```
+
+4. Install ALF:
+
+    ```bash
+    pip install -e .
+    ```

docs/prepare_dataset.md

@@ -0,0 +1,30 @@
+# Dataset Preparation
+
+## Reformatting the Original Dataset
+
+### Waymo Dataset
+To reformat Waymo's scenario protocol buffer data, download the dataset and then specify the `--wod_path` argument to the local path of the Waymo Motion dataset. Finally, execute the following command to split the data:
+
+```bash
+python scripts/preprocess/process_wod_data.py --wod_path your/path/to/waymo/motion/dataset
+```
+
+### Nuplan Dataset
+For instructions on setting up the Nuplan dataset, refer to [Nuplan Devkit Documentation](https://nuplan-devkit.readthedocs.io/en/latest/dataset_setup.html).
+
+## Caching the Dataset
+To train efficiently, it is essential to cache the data first. Follow these steps:
+
+### Waymo Dataset
+Run the following script to cache the Waymo dataset:
+
+```bash
+sh ./scripts/preprocess/cache_wod_data.sh
+```
+
+### Nuplan Dataset
+Run the following script to cache the Nuplan dataset:
+
+```bash
+sh ./scripts/preprocess/cache_nuplan_data.sh
+```

docs/train_eval.md

@@ -0,0 +1,84 @@
+# Training and Evaluation Guide
+
+## Training
+To train on the Nuplan Dataset, execute the following command:
+```bash
+sh ./scripts/training/train_nexus_nuplan.sh
+```
+
+To train on the Waymo Open Dataset, execute the following command:
+```bash
+sh ./scripts/training/train_nexus_wod.sh
+```
+
+## Evaluation
+
+### Waymo Sim Agents Benchmark
+
+#### Steps to Fix Bugs in Waymo Open Dataset Package
+
+1. **Install the Waymo Open Dataset Package**
+   Make sure you have installed the Waymo Open Dataset package by running:
+   ```bash
+   pip install waymo-open-dataset-tf-2-12-0==1.6.4
+   ```
+
+2. **Modify the File**
+   Edit the file located at `/usr/local/lib/python3.9/dist-packages/waymo_open_dataset/wdl_limited/sim_agents_metrics/metrics.py`. Change the code at line 47 from:
+   ```python
+   config_path = '{pyglib_resource}waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_2024_config.textproto'.format(pyglib_resource='')
+   ```
+   to:
+   ```python
+   import os
+   # Get the resource path from an environment variable
+   pyglib_resource = os.getenv('PYGLIB_RESOURCE_PATH', '')
+
+   # Construct the full config path
+   config_path = f'{pyglib_resource}/waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_config.textproto'
+   ```
+
+3. **Set Environment Variable**
+   Set the environment variable for the resource path by running:
+   ```bash
+   export PYGLIB_RESOURCE_PATH=/usr/local/lib/python3.9/dist-packages
+   ```
+
+By following these steps, you will ensure that the configuration path is correctly set and the package functions properly.
+
+
+
+#### Start Evaluation
+To evaluate on the Waymo Sim Agents Benchmark, follow these steps:
+
+1. Running the testing script:
+    ```bash
+    sh ./scripts/testing/test_nexus_wod_rollouts.sh
+    ```
+
+2. Running the offline evaluation script with multiprocessing
+    ```bash
+    python ./scripts/testing/offline_sim_agents.py --pkl_dir ${YOUR_PKL_DUMP_PATH} --nprocess 32 --output_dir ${OUTPUT_DIR}
+    ```
+
+### Metrics(e.g. FDE, ADE, ...) Evaluation
+To evaluate specific metrics like FDE, ADE, Collision rate, etc., run the following command:
+   ```bash
+   sh ./scripts/testing/test_nexus_metrics.sh
+   ```
+
+## Checkpoints
+
+We provide several checkpoints for model reproduction. To use a checkpoint, download it and replace the checkpoint path in the bash script:
+
+```bash
++checkpoint.ckpt_path=PATH_TO_CHECKPOINT \
++checkpoint.strict=True \
+```
+
+### Checkpoint List
+We provided the following CKPT:
+| Model         | Dataset | Checkpoint                                                                                  |
+|---------------|---------|---------------------------------------------------------------------------------------------|
+| Nexus-nuplan     | NuPlan  |   [Google Cloud](https://storage.googleapis.com/93935945854-us-central1-blueprint-config/epoch_llama_sm.ckpt)                                                                  
+| Nexus-wod  | Waymo Motion dataset  | [Google Cloud](https://storage.googleapis.com/93935945854-us-central1-blueprint-config/epoch_llama_sm.ckpt) |

nuplan_extent/__init__.py

@@ -0,0 +1,14 @@
+from nuplan.common.actor_state.vehicle_parameters import VehicleParameters
+
+
+def get_ideal_one_parameters() -> VehicleParameters:
+    return VehicleParameters(
+        vehicle_name="Leading Ideal One",
+        vehicle_type="gen1",
+        width=1.96,
+        front_length=3.9,
+        rear_length=1.13,
+        wheel_base=2.935,
+        cog_position_from_rear_axle=1.67,
+        height=1.777,
+    )

nuplan_extent/__pycache__/__init__.cpython-39.pyc

@@ -0,0 +1,159 @@
+import math
+
+import numpy as np
+
+
+class EdgeComputer:
+    """
+    Estimate the left and right road edges given a raster bev map and expert future waypoints.
+    For each future waypoint, find its corresponding left and right road edge.
+    """
+
+    def __init__(
+            self,
+            bev_map: np.ndarray,
+            trajectory: np.ndarray,
+            resolution: float = 0.5,
+            y_flip: bool = True,
+    ):
+        """
+        :param bev_map: one channel bev map [H, W], with 1 stands for drivable area and 0 stands for
+                        non-drivable area (currently only supports H==W).
+        :param trajectory: future waypoints with shape [N, 3], where N is the number of waypoints and
+                           for each waypoint it contains (x, y, yaw).
+        :param resolution: resolution of the bev map, used to convert trajectory coords value into pixels.
+        :param y_flip: whether to flip y coord value while converting, defaults to True for Carla data.
+        """
+        assert len(
+            bev_map.shape) == 2, "The bev map should only have one channel."
+        h, w = bev_map.shape
+        assert h == w, "The bev map should have same H and W."
+        self.map_half_size = h / 2
+        self.resolution = resolution
+        self.y_flip = y_flip
+        self.bev_map = bev_map
+
+        # convert trajectory xy to pixel value in bev map
+        self.traj_in_pixel = self.to_pixel(trajectory[:, :2])
+        self.angles = trajectory[:, 2]
+        # number of waypoints
+        self.num_waypoints = trajectory.shape[0]
+
+    def to_pixel(self, trajectory):
+        """Convert coords into pixels."""
+        assert trajectory.shape[1] == 2
+        k = -1 if self.y_flip else 1
+        traj_in_pixel = np.zeros_like(trajectory)
+        traj_in_pixel[:, 0] = trajectory[:, 0] / \
+            self.resolution + self.map_half_size
+        traj_in_pixel[:, 1] = k * trajectory[:, 1] / \
+            self.resolution + self.map_half_size
+        return traj_in_pixel
+
+    def to_coords(self, traj_in_pixel):
+        """Convert pixels back into coords."""
+        assert traj_in_pixel.shape[1] == 2
+        k = -1 if self.y_flip else 1
+        trajectory = np.zeros_like(traj_in_pixel)
+        trajectory[:, 0] = (
+            traj_in_pixel[:, 0] - self.map_half_size) * self.resolution
+        trajectory[:, 1] = (
+            traj_in_pixel[:, 1] - self.map_half_size) * k * self.resolution
+        return trajectory
+
+    def get_traj_in_pixel(self):
+        return self.traj_in_pixel
+
+    @staticmethod
+    def get_slope(angle):
+        """Get the slope given yaw angle."""
+        tan_angle = math.tan(angle)
+        return tan_angle
+
+    def get_edges(self,
+                  max_steps: int,
+                  delta_l: float = 0.5,
+                  tolerance: int = 0,
+                  ideal_steps: int = -1):
+        """Estimate left and right edges given the expert waypoints.
+
+        The basic idea is to sample along the line that passes through the waypoint
+        and is perpendicular to the expert route continuesly until a non-drivable area
+        is found.
+        :param max_steps: maximum steps to sample along one direction.
+        :param delta_l: the interval between each sample alone the line.
+        :param tolerance: the 'delay' to stop sampling, if it's 0 then stop immediately after
+                          find the edge, otherwise keep sampling for {tolerance} steps after
+                          find the edge.
+        :param ideal_steps: the most possible steps to sample that could find the edge, used
+                            when bev map is incomplete. In other words, this gives the bev map
+                            an 'imaginary' drivable boundary.
+        """
+        left_edges = []
+        right_edges = []
+        # record the search steps for both direction in previous waypoints
+        self.search_steps = np.zeros((self.num_waypoints, 2))
+        self.max_steps = max_steps
+        self.delta_l = delta_l
+        self.tolerance = tolerance
+        self.ideal_steps = ideal_steps if ideal_steps > 0 else max_steps
+
+        for i in range(self.num_waypoints):
+            left_edge = self._find_points_one_direction(i, right=False)
+            right_edge = self._find_points_one_direction(i, right=True)
+            left_edges.append(left_edge)
+            right_edges.append(right_edge)
+        left_edges = np.array(left_edges)
+        right_edges = np.array(right_edges)
+        return left_edges, right_edges
+
+    def _find_points_one_direction(
+            self,
+            waypoint_step: int,
+            right: bool,
+    ):
+        """Sample along one direction.
+        :param waypoint_step: indicate the current searching waypoint.
+        :param right: indicate the searching direction, whether it's left or right.
+        """
+        x = self.traj_in_pixel[waypoint_step, 0]
+        y = self.traj_in_pixel[waypoint_step, 1]
+        slope = self.get_slope(self.angles[waypoint_step])
+        delta_x = self.delta_l / (math.sqrt(1 + slope**2))
+        road_edge = None
+        # if direction is right, x value is incremented
+        k = 1 if right else -1
+        # column index, left is 0 and right is 1
+        col_ind = 1 if right else 0
+
+        # get the sampling steps from the last waypoint for reference.
+        if waypoint_step == 0:
+            last_steps = self.ideal_steps
+        else:
+            last_steps = self.search_steps[waypoint_step - 1, col_ind]
+
+        # sample along x in one direction
+        found = -1
+        step = 1
+        while (found != 0) and (step <= self.max_steps):
+            if found > 0:
+                found -= 1
+            x_next = x + k * step * delta_x
+            y_next = slope * x_next + (y - slope * x)
+            if found == -1 and self.bev_map[round(y_next), round(x_next)] == 0:
+                found = self.tolerance
+            if found == 0:
+                road_edge = (x_next, y_next)
+                steps = step
+            step += 1
+
+        # if edge is not found after reaching max sampling steps, use the sampling
+        # steps from last waypoint as the imaginary edge.
+        if road_edge is None:
+            x_next = x + k * last_steps * delta_x
+            y_next = slope * x_next + (y - slope * x)
+            road_edge = (x_next, y_next)
+            steps = last_steps
+
+        self.search_steps[waypoint_step, col_ind] = steps
+        return road_edge