Skip to content

docs: update README #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ktro2828 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: update README #39

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 76 additions & 93 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,119 +1,102 @@
# accelerated_image_processor

A ROS2 package that provides GPU-accelerated image processing capabilities for efficient image rectification and compression. This package is designed to handle high-throughput image processing tasks using hardware acceleration when available.

## Features

- GPU-accelerated image rectification using:
- NVIDIA Performance Primitives (NPP)
- OpenCV CPU implementation
- OpenCV CUDA implementation
- Hardware-accelerated JPEG compression using:
- NVIDIA JPEG encoder (for Jetson platforms)
- NVIDIA NVJPEG library
- TurboJPEG (CPU fallback)
- Configurable processing pipeline
- Support for RGB8 and BGR8 image formats
- Task queue management for handling high-throughput scenarios
- ROS2 component-based architecture

## Dependencies

### Required
- OpenCV
#### For GPU/HW acceleration
- CUDA Toolkit
- NVIDIA Performance Primitives (NPP)
- NVJPEG (for discrete GPU environment)
- Jetson Multimedia API (for Jetson platforms)
#### For CPU acceleration
- libturbojpeg
`accelerated_image_processor` is a set of C++ and Python libraries for accelerated image processing.
It provides common image data structures, image/video compression, video decompression, rectification pipelines, ROS 2 nodes, and benchmark tools.

## Installation
> [!NOTE]
> `src/accelerated_image_processor` is a legacy implementation and is intentionally not described here.
> The current implementation is split into the packages listed below.

1. Install the required dependencies:
```bash
sudo apt install ros-$ROS_DISTRO-cv-bridge ros-$ROS_DISTRO-image-geometry libturbojpeg0-dev
```
## Packages

2. Clone this repository into your ROS2 workspace:
```bash
cd ~/ros2_ws/src
git clone https://github.com/tier4/accelerated_image_processor.git
```
| Package | Role | ROS dependency |
| ------------------------------------------- | -------------------------------------------------------- | -------------- |
| `accelerated_image_processor_common` | Common datatypes, parameters, and processor base classes | No |
| `accelerated_image_processor_compression` | JPEG/video compression processors | No |
| `accelerated_image_processor_decompression` | CUDA-accelerated FFmpeg video decompression | No |
| `accelerated_image_processor_pipeline` | Rectification processors | No |
| `accelerated_image_processor_python` | Python bindings for common/compression/decompression | No |
| `accelerated_image_processor_ros` | ROS 2 components/nodes and ROS message conversions | Yes |
| `accelerated_image_processor_benchmark` | Benchmark CLI/library | Yes |

3. Build the package:
```bash
cd ~/ros2_ws
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-up-to accelerated_image_processor
```
## Supported processors

## Usage
### Compression

The package provides a ROS2 component that can be loaded either as a standalone node or as part of a component container.
| Processor | Format | Backend | Device/platform |
| ---------------------- | ------ | -------------------------------------- | ---------------- |
| `CpuJPEGCompressor` | `JPEG` | TurboJPEG | CPU |
| `NvJPEGCompressor` | `JPEG` | nvJPEG | CUDA-capable GPU |
| `JetsonJPEGCompressor` | `JPEG` | Jetson Multimedia API | NVIDIA Jetson |
| `JetsonH264Compressor` | `H264` | Jetson Multimedia API / NvVideoEncoder | NVIDIA Jetson |
| `JetsonH265Compressor` | `H265` | Jetson Multimedia API / NvVideoEncoder | NVIDIA Jetson |
| `JetsonAV1Compressor` | `AV1` | Jetson Multimedia API / NvVideoEncoder | NVIDIA Jetson |

### Parameters
JPEG backend selection is automatic in priority order: Jetson, nvJPEG, then TurboJPEG.
Video compression is currently Jetson-only.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `rect_impl` | string | "npp" | Rectification implementation to use ("npp", "opencv_cpu", or "opencv_gpu") |
| `alpha` | double | 0.0 | Rectification alpha parameter |
| `jpeg_quality` | int | 60 | JPEG compression quality (0-100) |
| `do_rectify` | bool | true | Enable/disable image rectification |
| `max_task_queue_length` | int | 5 | Maximum number of images that can be queued for processing. A smaller value may cause dropped frames, while a larger value may lead to increased latency and higher memory usage. |
### Decompression

### Topics
| Processor | Input formats | Backend | Device/platform |
| ------------------------- | --------------------- | ----------------- | ---------------- |
| `FfmpegVideoDecompressor` | `H264`, `H265`, `AV1` | FFmpeg + CUDA/NPP | CUDA-capable GPU |

#### Subscribed Topics
- `image_raw` (sensor_msgs/Image): Raw input image
- `camera_info` (sensor_msgs/CameraInfo): Camera calibration information
### Pipeline

#### Published Topics
- `image_rect` (sensor_msgs/Image): Rectified image
- `image_rect/compressed` (sensor_msgs/CompressedImage): Compressed rectified image
- `image_raw/compressed` (sensor_msgs/CompressedImage): Compressed raw image
- `camera_info_rect` (sensor_msgs/CameraInfo): Camera calibration information for the rectified image
| Processor | Task | Backend | Device/platform |
| --------------------- | ------------- | ----------------------------------- | ---------------- |
| `NppRectifier` | Rectification | NVIDIA Performance Primitives (NPP) | CUDA-capable GPU |
| `OpenCvCudaRectifier` | Rectification | OpenCV CUDA | CUDA-capable GPU |
| `CpuRectifier` | Rectification | OpenCV | CPU |

### Launch Examples
Rectifier backend selection is automatic in priority order: NPP, OpenCV CUDA, then CPU.

1. As a standalone node:
```bash
ros2 run accelerated_image_processor accelerated_image_processor_node
```
## Installation

### ROS 2 workspace build

Clone into a ROS 2 workspace and build only the current packages.

2. With custom parameters:
```bash
ros2 run accelerated_image_processor accelerated_image_processor_node --ros-args -p rect_impl:=npp -p jpeg_quality:=80
```
git clone https://github.com/tier4/accelerated_image_processor.git
cd accelerated_image_processor

## Camera Intrinsics Publication
rosdep update && rosdep install -y --from-paths src --ignore-src --rosdistro ${ROS_DISTRO}

According to the [definition](https://docs.ros2.org/latest/api/sensor_msgs/msg/CameraInfo.html) of `sensor_msgs/msg/CameraInfo`, its parameters of `K` and `P` are described as follows:
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
```

> ```
> Intrinsic camera matrix, for the raw (distorted) images
> [fx 0 cx]
> K = [ 0 fy cy]
> [ 0 0 1]
> ```
### Python package in a non-ROS CUDA environment

> ```
> the intrinsic (camera) matrix of the processed (rectified) image
> [fx' 0 cx' Tx]
> P = [ 0 fy' cy' Ty]
> [ 0 0 1 0]
> ```
This repository can be installed as a Python package without ROS 2.
System packages are still required for native extensions.

The contents of `K` in `camera_info_rect` published by this node will be identical to the upper-left 3x3 portion of `P` from the input `camera_info`, as long as the specified `alpha` value is the same as the original (i.e., the one used during camera calibration). Typically, `alpha==0.0` is used.
Example for Ubuntu 22.04 + CUDA environment:

However, this node allows the input of an arbitrary `alpha` value, which may result in `P` from the input `camera_info` not accurately representing the intrinsic values for undistorted output images.
```bash
sudo apt update && sudo apt install -y \
build-essential \
cmake \
git \
libavcodec-dev \
libavutil-dev \
libboost-python-dev \
libopencv-dev \
libturbojpeg0-dev \
ninja-build \
pkg-config \
python3-dev \
python3-pip
```

Therefore, this node publishes the camera info for the undistorted images. The updated camera info is generated by:
Install with `uv`:

1. Calculating a new camera matrix based on `K` and `D` from the input `camera_info`
2. Copying the contents of input `camera_info` into the output `camera_info_rect`
3. Filling `K` in the output `camera_info_rect` with the values calculated during step 1
4. Setting `D` in the output `camera_info_rect` to zeros
5. Filling the upper-left 3x3 portion of `P` in the output `camera_info_rect` with the values calculated during step 1.
```bash
uv add git+https://github.com/tier4/accelerated_image_processor.git
```

Or install with `pip`:

This updated camera info is useful for applications that require the intrinsic parameters of the rectified image for further processing.
```bash
pip install git+https://github.com/tier4/accelerated_image_processor.git
```