Build instructions

Building

Note: PyPI and conda packages are available which can be installed without building from source. See install instructions.

When building from source, this package will download and install the native TileDB library unless the --tiledb argument is provided or TileDB Embedded is installed in global system path.

Note: if the Numpy and Cython dependencies are not installed, pip will try to build them from source. This can take a long time and make the install appear to "hang." Pass the -v flag to pip to monitor the build process.

If you wish to use a custom version of the TileDB library and the install location is not in the compiler search path, use the --tiledb argument to setup.py or create a requirements.txt file that specifies the tiledb install path manually.

$ cat > tiledb_requirements.txt <<EOF
  tiledb==<version> --install-option="--tiledb=<path/to/tiledb/install>"
  EOF
$ pip install -r tiledb_requirements.txt

Do not forget to put the built .so / .dylib / .dll on the dynamic linker path, otherwise TileDB-Py will fail to load the shared library upon import.

Installing From Source

TileDB-Py Build Dependencies:

• NumPy
• pybind11
• Cython
• C++11 compiler
• CMake >= 3.3

TileDB-Py Runtime Dependencies:

• Numpy

Linux / OSX

Execute the following commands:

git clone https://github.com/TileDB-Inc/TileDB-Py.git
cd TileDB-Py
pip install -r requirements_dev.txt
python setup.py build_ext --inplace
python setup.py install

If using conda/mamba for development:

# make sure you are in the base venv first
(base) user@hostname:~$ conda activate base

# create a new venv called new-tiledb-env (or name of your choice) with Python 3.10 (or version of your choice)
(base) user@hostname:~$ conda create -n new-tiledb-env python=3.10

# activate your new venv
(base) user@hostname:~$ conda activate new-tiledb-env

# move into the the base directory of TileDB-Py
(new-tiledb-env) user@hostname:~$ cd TileDB-Py/

# install the necessary dependencies for development
(new-tiledb-env) user@hostname:~/TileDB-Py$ pip install -r misc/requirements_ci.txt

# install TileDB-Py
(new-tiledb-env) user@hostname:~/TileDB-Py$ python setup.py develop

If you wish to use a custom version of the TileDB library and it is installed in a non-standard location, pass the path to setup.py with the --tiledb= flag. If you want to pass extra compiler/linker flags during the C++ extension compilation step use --cxxflags= or --lflags=.

$ python setup.py build_ext --inplace --tiledb=/home/tiledb/dist

If TileDB is installed in a non-standard location, you also need to make the dynamic linker aware of libtiledb's location. Otherwise when importing the tiledb module you will get an error that the built extension module cannot find libtiledb's symbols:

$ env LD_LIBRARY_PATH="/home/tiledb/dist/lib:$LD_LIBRARY_PATH" python -m unittest -v

For macOS the linker environment variable is DYLD_LIBRARY_PATH.

Installing on Windows

If you are building the extension on Windows, first install a Python distribution such as Miniconda. You can then either build TileDB from source, or download the pre-built binaries.

Once you've installed Miniconda and TileDB, open the Miniconda command prompt and execute:

cd TileDB-Py
conda install conda-build
conda install virtualenv
virtualenv venv
venv\Scripts\activate
pip install -r requirements_dev.txt
python setup.py build_ext --inplace --tiledb=C:\path\to\TileDB\
set PATH=%PATH%;C:\path\to\TileDB\bin
python -m unittest -v

Note that if you built TileDB from source, then replace C:\path\to\TileDB with C:\path\to\TileDB\dist.

Developing and testing TileDB-Py

TileDB-Py includes a handy Conda environment definition file for setting up a test environment:

$ conda env create -f environment.yml

This will create a tiledbpy conda environment with all the development library dependencies.

You can run the unittests from the source folder without having the package installed. First build the package in place from the source directory:

$ python setup.py build_ext --inplace

Tests can now be run using Python's unittest framework:

$ python -m unittest -v

Doctests can be run using the doctest module:

$ python -m doctest -o NORMALIZE_WHITESPACE -f tiledb/libtiledb.pyx

You can also install a symlink named site-packages/tiledb.egg-link to the development folder of TileDB-Py with:

$ pip install --editable .

This enables local changes to the current development repo to be reflected globally.

Building TileDB-Py against libtiledb from conda

TileDB core (libtiledb) is distributed as a separate conda package, tiledb, which may be used for local builds of TileDB-Py:

activate the desired conda environment
run conda install tiledb
run python setup.py build_ext --inplace --tiledb=$CONDA_PREFIX