Event-based Background-Oriented Schlieren

This is the official repository for Event-based Background-Oriented Schlieren, IEEE T-PAMI 2023 by
Shintaro Shiba, Friedhelm Hamann, Yoshimitsu Aoki and Guillermo Callego.

Paper | Video | Dataset | News

If you use this work in your research, please cite it (see also here):

@Article{Shiba23pami,
  author        = {Shintaro Shiba and Friedhelm Hamann and Yoshimitsu Aoki and Guillermo Gallego},
  title         = {Event-based Background-Oriented Schlieren},
  journal       = {IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI)},
  year          = {2024},
  volume        = {46},
  number        = {4},
  pages         = {2011-2026},
  doi           = {10.1109/TPAMI.2023.3328188}
}

---

Setup

Requirements

• python: 3.8.x, 3.9.x, 3.10.x

• ffmpeg ... This is necessary for ffmpeg-python.

Tested environments

• Mac OS Monterey (both M1 and non-M1)
• Ubuntu (CUDA 11.1, 11.3, 11.8)
• PyTorch 1.9-1.12.1, or PyTorch 2.0.

Installation

Quick setup

Minimal quick setup is to use venv.

python3 -m venv ~/work/venv
source ~/work/venv/bin/activate  # activate env
python3 -V  # make sure the version, 3.8 or higher
pip3 install scipy==1.9.1 optuna==2.10.1 opencv-python matplotlib plotly \
ffmpeg-python h5py hdf5plugin PyYAML Pillow sklearn scikit-image argparse openpiv \
torch torchvision black isort kaleido

Full setup

You can use poetry.

poetry install

This install dev dependencies (format, test) too.

Download dataset

Please download the dataset and put into your local folder. You can download the files from the official source or Google Drive The structure of the folder is as follows:

(root)/datasets/
    CCS/   # fixed keyword: dataset recorded with the co-capture system (CCS).
        (sequence_name)/
            basler_0/
                frames.mp4          # frame data
                config.yaml
            prophesee_0/
                cd_events.raw       # event data in .raw format (Prophesee)
                events.hdf5         # event data converted in HDF5 format
                trigger_events.txt
            homography.txt          # calibration file

EBOS dataset recorded with Co-Capture System (CCS)

The Prophesee camera provides .raw format file, which requires to install OpenEB. We converted this file into hdf5 file formats for better compatibility among languages and OS. So, there is no need to install OpenEB.

Execution

Run

An example is:

python3 bos_event.py --config_file ./configs/hot_plate1.yaml --eval

Config

The configuration for each experiment is provided to the script through yaml files.

Please see the readme.

Development

Note you use poetry for development.

Format

You can quickly run format of the codes:

make fmt

Code Authors

• @shiba24
• @FriedhelmHamann

LICENSE

Please check License.

Acknowledgement

TBD

---

Additional Resources

• Research page (TU Berlin, RIP lab)
• Research page (Keio University, Aoki Media Lab)
• Course at TU Berlin
• Survey paper
• List of Resources
• CoCapture: GUI for recording with multi camera systems.