Skip to content

Commit

Permalink
Merge branch 'main' into ft/partition-binder
Browse files Browse the repository at this point in the history
  • Loading branch information
c-thiel authored Aug 12, 2024
2 parents 0c4615c + 182b20d commit 3861719
Show file tree
Hide file tree
Showing 58 changed files with 4,043 additions and 1,054 deletions.
83 changes: 83 additions & 0 deletions .github/workflows/bindings_python_ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: Bindings Python CI

on:
push:
branches:
- main
pull_request:
branches:
- main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}
cancel-in-progress: true

jobs:
check-rust:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check format
run: cargo fmt --all -- --check
- name: Check clippy
run: cargo clippy --all-targets --all-features -- -D warnings

check-python:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
pip install ruff
- name: Check format
working-directory: "bindings/python"
run: |
ruff format . --diff
- name: Check style
working-directory: "bindings/python"
run: |
ruff check .
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-latest
- macos-latest
- windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: 3.8
- uses: PyO3/maturin-action@v1
with:
working-directory: "bindings/python"
command: build
args: --out dist --sdist
- name: Run tests
working-directory: "bindings/python"
shell: bash
run: |
set -e
pip install dist/pyiceberg_core-*.whl --force-reinstall
pip install pytest
pytest -v
2 changes: 1 addition & 1 deletion .github/workflows/ci_typos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,4 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Check typos
uses: crate-ci/[email protected].2
uses: crate-ci/[email protected].6
7 changes: 5 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,12 @@
# specific language governing permissions and limitations
# under the License.

/target
/Cargo.lock
target
Cargo.lock
.idea
.vscode
**/.DS_Store
dist/*
**/venv
*.so
*.pyc
8 changes: 5 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,14 +110,16 @@ $ cargo version
cargo 1.69.0 (6e9a83356 2023-04-12)
```

#### Install docker
#### Install Docker or Podman

Currently, iceberg-rust uses docker to set up environment for integration tests.
Currently, iceberg-rust uses Docker to set up environment for integration tests. Podman is also supported.

You can learn how to install docker from [here](https://docs.docker.com/get-docker/).
You can learn how to install Docker from [here](https://docs.docker.com/get-docker/).

For macos users, you can install [OrbStack](https://orbstack.dev/) as a docker alternative.

For podman users, refer to [Using Podman instead of Docker](docs/contributing/podman.md)

## Build

* To compile the project: `make build`
Expand Down
21 changes: 13 additions & 8 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,14 @@
[workspace]
resolver = "2"
members = [
"crates/catalog/*",
"crates/examples",
"crates/iceberg",
"crates/integrations/*",
"crates/test_utils",
"crates/catalog/*",
"crates/examples",
"crates/iceberg",
"crates/integrations/*",
"crates/test_utils",
]
exclude = [
"bindings/python"
]

[workspace.package]
Expand All @@ -36,7 +39,7 @@ rust-version = "1.77.1"

[workspace.dependencies]
anyhow = "1.0.72"
apache-avro = "0.16"
apache-avro = "0.17"
array-init = "2"
arrow-arith = { version = "52" }
arrow-array = { version = "52" }
Expand Down Expand Up @@ -67,12 +70,13 @@ log = "^0.4"
mockito = "^1"
murmur3 = "0.5.2"
once_cell = "1"
opendal = "0.47"
opendal = "0.48"
ordered-float = "4.0.0"
parquet = "52"
pilota = "0.11.2"
pretty_assertions = "1.4.0"
port_scanner = "0.1.5"
regex = "1.10.5"
reqwest = { version = "^0.12", default-features = false, features = ["json"] }
rust_decimal = "1.31.0"
serde = { version = "^1.0", features = ["rc"] }
Expand All @@ -85,7 +89,8 @@ tempfile = "3.8"
tokio = { version = "1", default-features = false }
typed-builder = "^0.19"
url = "2"
uuid = "1.6.1"
urlencoding = "2"
uuid = { version = "1.6.1", features = ["v7"] }
volo-thrift = "0.10"
hive_metastore = "0.1.0"
tera = "1"
20 changes: 13 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,33 +29,39 @@ Working on [v0.3.0 Release Milestone](https://github.com/apache/iceberg-rust/mil

The Apache Iceberg Rust project is composed of the following components:

| Name | Release | Docs |
|------------------------|------------------------------------------------------------|------------------------------------------------------|
| [iceberg] | [![iceberg image]][iceberg link] | [![docs release]][iceberg release docs] |
| [iceberg-datafusion] | - | - |
| [iceberg-catalog-glue] | - | - |
| [iceberg-catalog-hms] | [![iceberg-catalog-hms image]][iceberg-catalog-hms link] | [![docs release]][iceberg-catalog-hms release docs] |
| [iceberg-catalog-rest] | [![iceberg-catalog-rest image]][iceberg-catalog-rest link] | [![docs release]][iceberg-catalog-rest release docs] |
| Name | Release | Docs |
|------------------------|------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| [iceberg] | [![iceberg image]][iceberg link] | [![docs release]][iceberg release docs] [![docs dev]][iceberg dev docs] |
| [iceberg-datafusion] | - | [![docs dev]][iceberg-datafusion dev docs] |
| [iceberg-catalog-glue] | - | [![docs dev]][iceberg-catalog-glue dev docs] |
| [iceberg-catalog-hms] | [![iceberg-catalog-hms image]][iceberg-catalog-hms link] | [![docs release]][iceberg-catalog-hms release docs] [![docs dev]][iceberg-catalog-hms dev docs] |
| [iceberg-catalog-rest] | [![iceberg-catalog-rest image]][iceberg-catalog-rest link] | [![docs release]][iceberg-catalog-rest release docs] [![docs dev]][iceberg-catalog-rest dev docs] |

[docs release]: https://img.shields.io/badge/docs-release-blue
[docs dev]: https://img.shields.io/badge/docs-dev-blue
[iceberg]: crates/iceberg/README.md
[iceberg image]: https://img.shields.io/crates/v/iceberg.svg
[iceberg link]: https://crates.io/crates/iceberg
[iceberg release docs]: https://docs.rs/iceberg
[iceberg dev docs]: https://rust.iceberg.apache.org/api/iceberg/

[iceberg-datafusion]: crates/integrations/datafusion/README.md
[iceberg-datafusion dev docs]: https://rust.iceberg.apache.org/api/iceberg_datafusion/

[iceberg-catalog-glue]: crates/catalog/glue/README.md
[iceberg-catalog-glue dev docs]: https://rust.iceberg.apache.org/api/iceberg_catalog_glue/

[iceberg-catalog-hms]: crates/catalog/hms/README.md
[iceberg-catalog-hms image]: https://img.shields.io/crates/v/iceberg-catalog-hms.svg
[iceberg-catalog-hms link]: https://crates.io/crates/iceberg-catalog-hms
[iceberg-catalog-hms release docs]: https://docs.rs/iceberg-catalog-hms
[iceberg-catalog-hms dev docs]: https://rust.iceberg.apache.org/api/iceberg_catalog_hms/

[iceberg-catalog-rest]: crates/catalog/rest/README.md
[iceberg-catalog-rest image]: https://img.shields.io/crates/v/iceberg-catalog-rest.svg
[iceberg-catalog-rest link]: https://crates.io/crates/iceberg-catalog-rest
[iceberg-catalog-rest release docs]: https://docs.rs/iceberg-catalog-rest
[iceberg-catalog-rest dev docs]: https://rust.iceberg.apache.org/api/iceberg_catalog_rest/

## Supported Rust Version

Expand Down
35 changes: 35 additions & 0 deletions bindings/python/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[package]
name = "pyiceberg_core_rust"
version = "0.0.1"
edition = "2021"
homepage = "https://rust.iceberg.apache.org"
rust-version = "1.77.1"
# This crate is used to build python bindings, we don't want to publish it
publish = false

license = "Apache-2.0"
keywords = ["iceberg"]

[lib]
crate-type = ["cdylib"]

[dependencies]
iceberg = { path = "../../crates/iceberg" }
pyo3 = { version = "0.22", features = ["extension-module"] }
44 changes: 44 additions & 0 deletions bindings/python/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing,
~ software distributed under the License is distributed on an
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
-->

# Pyiceberg Core

This project is used to build an iceberg-rust powered core for pyiceberg.

## Setup

```shell
python -m venv venv
source ./venv/bin/activate

pip install maturin
```

## Build

```shell
maturin develop
```

## Test

```shell
maturin develop -E test
pytest -v
```
43 changes: 43 additions & 0 deletions bindings/python/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[build-system]
requires = ["maturin>=1.0,<2.0"]
build-backend = "maturin"

[project]
name = "pyiceberg_core"
version = "0.0.1"
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]

[project.optional-dependencies]
test = ["pytest"]

[tool.maturin]
features = ["pyo3/extension-module"]
python-source = "python"
module-name = "pyiceberg_core.pyiceberg_core_rust"

[tool.ruff.lint]
ignore = ["F403", "F405"]
21 changes: 21 additions & 0 deletions bindings/python/python/pyiceberg_core/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

from .pyiceberg_core_rust import *

__doc__ = pyiceberg_core_rust.__doc__
__all__ = pyiceberg_core_rust.__all__
Loading

0 comments on commit 3861719

Please sign in to comment.