Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

garaga_rs WASM bindings #180

Merged
merged 85 commits into from
Sep 10, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
dc18ae4
Adds preliminary support for WASM bindings
raugfer Aug 30, 2024
9396af7
Adds a CI workflow to build the WASM package and publish to NPM
raugfer Aug 30, 2024
1f994db
Adds --release flag to wasm-pack build in setup.sh
raugfer Aug 30, 2024
b0f1724
Adds support for wasm32 target tests
raugfer Aug 30, 2024
91794c6
Merge branch 'main' into rust-wasm-binding
raugfer Aug 30, 2024
58d3e48
Merge branch 'main' into rust-wasm-binding
raugfer Sep 2, 2024
1ff2b0b
Adds a profile.release section to Cargo.toml
raugfer Sep 3, 2024
d012d74
Adds preliminary implementation of patch-post-wasm-pack script
raugfer Sep 4, 2024
49d00a3
Fixes import issue in the wasm-pack patch script
raugfer Sep 4, 2024
0a24f7a
Reviews the wasm-pack patch script to handle both ECMAScript and Comm…
raugfer Sep 4, 2024
0d489b0
Removes default export when patching wasm-pack output
raugfer Sep 4, 2024
35792b1
Adds comments and applies proper indentation
raugfer Sep 4, 2024
ee63e4a
Removes wasm-pack build from setup.sh
raugfer Sep 4, 2024
1e83d5c
Modifies patch-post-wasm-pack to clean up and call wasm-pack
raugfer Sep 4, 2024
36e617f
Renames wasm-pack patch script
raugfer Sep 4, 2024
fff0dff
Adds more comments to wasm-pack patching script
raugfer Sep 4, 2024
c19aa43
Refactors and adds comments to wasm-pack-build-and-patch script
raugfer Sep 5, 2024
2763dea
Final adjustments to garaga_rs cargo/wasm-pack configuration
raugfer Sep 5, 2024
21ea2c8
Adjustments to comments
raugfer Sep 5, 2024
0f31f50
Adds an important comment about untested CI publication to NPM registry
raugfer Sep 5, 2024
7f0bf73
Fixes comments
raugfer Sep 5, 2024
c548762
use --no-default-features
feltroidprime Sep 5, 2024
d310f40
handle errors gracefully in binding
feltroidprime Sep 5, 2024
3c8bd9e
Adds preliminary NPM folder for garaga_rs hybrid package
raugfer Sep 5, 2024
79abbf4
Refactors patch script
raugfer Sep 5, 2024
2f75306
Changes patch file extension to .cjs
raugfer Sep 5, 2024
66becbc
Fixes patch script errors
raugfer Sep 5, 2024
f383847
Updates wasm-pack generated files after patching
raugfer Sep 5, 2024
971ed9d
Updates the docker configuration for reproducible builds
raugfer Sep 5, 2024
22a65ad
Updates comments
raugfer Sep 5, 2024
21cae90
Adds a possibility to change UID/GID in Dockerfile
raugfer Sep 5, 2024
3daf043
Upgrades devDependencies in package.json
raugfer Sep 5, 2024
4050002
Improves TypeScript configuration
raugfer Sep 5, 2024
b461d00
Cleans up garaga_rs npm configuration
raugfer Sep 5, 2024
db4825c
Adds preliminary configuration of the integration-test-suite folder
raugfer Sep 5, 2024
dd70b68
Updates wasm workflow to check for discrepancies in the generated code
raugfer Sep 5, 2024
bcbf709
Changes Dockerfile to build package as root (for now)
raugfer Sep 5, 2024
0bfd943
Reviews Dockerfile for building npm package
raugfer Sep 6, 2024
9e9f149
Adds npm pack to Dockerfile
raugfer Sep 6, 2024
ca46285
Updates npm build and publish CI action
raugfer Sep 6, 2024
1c99653
Attempts to fix WASM CI to upload built npm package file as artifact
raugfer Sep 6, 2024
5f39355
Minor adjustments to WASM CI
raugfer Sep 6, 2024
4cdda8f
Removes now obsolete wasm-pack-build-and-patch script
raugfer Sep 6, 2024
a12e14b
Updates Dockerfile to fix file permissions during build
raugfer Sep 6, 2024
8bef4c9
Improve comments
raugfer Sep 6, 2024
02dfe29
Adds web-js-esm-react to intergration-test-suite folder
raugfer Sep 6, 2024
383b7ac
Adds nodejs-ts-esm-tsc to intergration-test-suite folder
raugfer Sep 6, 2024
b216043
Adds turbo configuration for integration-test-suite
raugfer Sep 6, 2024
8b16a0d
Adds test-integration action to WASM CI
raugfer Sep 6, 2024
a2ded5c
Fixes the test-integration CI action for WASM
raugfer Sep 6, 2024
8aab6ab
Minor adjustments to WASM patch script
raugfer Sep 9, 2024
c519cb4
Adjusts tests towards standardization
raugfer Sep 9, 2024
8c39fa3
Adds "test" script for the web based tests of integration-test-suite
raugfer Sep 9, 2024
7091164
Modifies "test" script for the console based tests of integration-tes…
raugfer Sep 9, 2024
27e4b64
Makes adjustments to web based tests to capture only relevant output
raugfer Sep 9, 2024
d50886c
Improves web based test cases
raugfer Sep 9, 2024
e9e4f31
Updates CI to run NPM integration tests
raugfer Sep 9, 2024
caa93de
Minor fix to wasm.yml
raugfer Sep 9, 2024
25aeb77
Merge branch 'main' into rust-wasm-binding
feltroidprime Sep 9, 2024
1bbe2b5
Removes now unnecessary element tag from web based tests
raugfer Sep 9, 2024
1c8788a
Merge remote-tracking branch 'refs/remotes/origin/rust-wasm-binding' …
raugfer Sep 9, 2024
cdf05b7
Attempts to fix CI by adjusting wasm.yml
raugfer Sep 9, 2024
bb50d55
Adds ci-wasm to Makefile
raugfer Sep 9, 2024
782dda5
Updates code generated by wasm-pack
raugfer Sep 9, 2024
271dfe2
Adds sudo prefix to apt-get commands in wasm.yml
raugfer Sep 9, 2024
d8198d5
Fixes puppeteer dependency
raugfer Sep 9, 2024
3e606da
Adds debug commands to wasm.yml
raugfer Sep 9, 2024
bdbafa4
Attempts to fix wasm.yml CI
raugfer Sep 9, 2024
6a60b51
Attempts yet again to fix wasm.yml CI
raugfer Sep 9, 2024
008328d
Downgrades upload-artifact/download-artifact to v3
raugfer Sep 9, 2024
a1c300d
Adds git status --porcelain to wasm.yml CI output
raugfer Sep 9, 2024
9c8c04a
Updages code generated by wasm-pack
raugfer Sep 9, 2024
1832576
Adds Cargo.lock to git in order to enable reproducible builds
raugfer Sep 9, 2024
e81c89c
Fixes Dockerfile to apply fix-chown script to project root
raugfer Sep 9, 2024
672d8ab
Reverts artifact upload/donwload action scripts to v4
raugfer Sep 9, 2024
45ba21d
Fixes bug when usaging the fix-chown script
raugfer Sep 9, 2024
f2155c0
Adds a workaround for the corrupted artifact when downloading the .tg…
raugfer Sep 9, 2024
10e6a9c
Removes package-lock.json files from integration-test-suite to fix wa…
raugfer Sep 9, 2024
79a0a65
Improves CI action label/description
raugfer Sep 9, 2024
68c3f72
Renames test.cjs to scrape.cjs
raugfer Sep 9, 2024
4ba3692
Merge branch 'main' into rust-wasm-binding
raugfer Sep 9, 2024
6d7269c
Cleans up unnecessary code
raugfer Sep 10, 2024
f3042b4
Adds puppeteer install command to avoid warning
raugfer Sep 10, 2024
e730fbb
Replaces /tmp/output.txt by ./output.txt to prevent side effects
raugfer Sep 10, 2024
82d31d1
Removes explicit version number from integration tests dependencies
raugfer Sep 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .github/workflows/wasm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: WASM

on:
push:
branches:
- main
- master
tags:
- '*'
pull_request:
workflow_dispatch:

jobs:
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 'lts/*'
- uses: dtolnay/rust-toolchain@stable
- run: cargo install wasm-pack
- name: Test build
working-directory: tools/garaga_rs
run: wasm-pack test --node --release --no-default-features

release:
name: Release
runs-on: ubuntu-latest
# Note this will only run when a new tag is pushed
if: "startsWith(github.ref, 'refs/tags/')"
needs: [test]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 'lts/*'
registry-url: 'https://registry.npmjs.org'
- uses: dtolnay/rust-toolchain@stable
- run: cargo install wasm-pack
- name: Publish to npm
working-directory: tools/garaga_rs
# publication has not been tested after applying the patch
# possibly we will have to publish using npm instead
run: |
wasm-pack login
./wasm-pack-build-and-patch
wasm-pack publish --access=public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_API_TOKEN }}
27 changes: 26 additions & 1 deletion tools/garaga_rs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,33 @@ edition = "2021"
name = "garaga_rs"
crate-type = ["cdylib"]

[dependencies]
[profile.release]
lto = true
debug = false
opt-level = 3

[features]
# both features are on by default, but note that
# - python bindings are automatically excluded on wasm32-unknown-unknown
# - wasm bindings are garbage collected when not on wasm32-unknown-unknown
default = ["python", "wasm"]
python = ["dep:pyo3"]
wasm = ["dep:wasm-bindgen"]

[dev-dependencies]
wasm-bindgen-test = "0.3"

# assumes python dependencies when not on wasm32-unknown-unknown
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
pyo3 = { version = "0.22", features = ["extension-module", "num-bigint"] }

# assumes wasm dependencies when on wasm32-unknown-unknown
[target.'cfg(target_arch = "wasm32")'.dependencies]
wasm-bindgen = "0.2"

[dependencies]
pyo3 = { version = "0.22", features = ["extension-module", "num-bigint"], optional = true }
wasm-bindgen = { version = "0.2", optional = true }
num-bigint = "0.4"
num-traits = "0.2"
ark-bn254 = "0.4"
Expand Down
35 changes: 6 additions & 29 deletions tools/garaga_rs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,34 +5,11 @@ pub mod final_exp_witness;
pub mod io;
pub mod msm;
pub mod poseidon_transcript;
pub mod python_bindings;

use pyo3::{prelude::*, wrap_pyfunction};
// automatically excludes python bindings on wasm32-unknown-unknown (pyo3 not supported)
#[cfg(all(feature = "python", not(target_arch = "wasm32")))]
pub mod python_bindings;

#[pymodule]
fn garaga_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(python_bindings::g2::g2_add, m)?)?;
m.add_function(wrap_pyfunction!(python_bindings::g2::g2_scalar_mul, m)?)?;
m.add_function(wrap_pyfunction!(
python_bindings::pairing::multi_pairing,
m
)?)?;
m.add_function(wrap_pyfunction!(
python_bindings::pairing::multi_miller_loop,
m
)?)?;
m.add_function(wrap_pyfunction!(
python_bindings::final_exp_witness::get_final_exp_witness,
m
)?)?;
m.add_function(wrap_pyfunction!(
python_bindings::hades_permutation::hades_permutation,
m
)?)?;
m.add_function(wrap_pyfunction!(
python_bindings::extf_mul::nondeterministic_extension_field_mul_divmod,
m
)?)?;
m.add_function(wrap_pyfunction!(python_bindings::ecip::zk_ecip_hint, m)?)?;
Ok(())
}
// automatically includes wasm bindings on wasm32-unknown-unknown
#[cfg(any(feature = "wasm", target_arch = "wasm32"))]
pub mod wasm_bindings;
19 changes: 19 additions & 0 deletions tools/garaga_rs/src/python_bindings/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,22 @@ use pyo3::{

const CURVE_BN254: usize = 0;
const CURVE_BLS12_381: usize = 1;

#[pymodule]
fn garaga_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(g2::g2_add, m)?)?;
m.add_function(wrap_pyfunction!(g2::g2_scalar_mul, m)?)?;
m.add_function(wrap_pyfunction!(pairing::multi_pairing, m)?)?;
m.add_function(wrap_pyfunction!(pairing::multi_miller_loop, m)?)?;
m.add_function(wrap_pyfunction!(
final_exp_witness::get_final_exp_witness,
m
)?)?;
m.add_function(wrap_pyfunction!(hades_permutation::hades_permutation, m)?)?;
m.add_function(wrap_pyfunction!(
extf_mul::nondeterministic_extension_field_mul_divmod,
m
)?)?;
m.add_function(wrap_pyfunction!(ecip::zk_ecip_hint, m)?)?;
Ok(())
}
42 changes: 42 additions & 0 deletions tools/garaga_rs/src/wasm_bindings.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
use num_bigint::{BigInt, BigUint};
use std::str::FromStr;
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn msm_calldata_builder(
values: Vec<JsValue>,
scalars: Vec<JsValue>,
curve_id: usize,
) -> Vec<JsValue> {
let values: Vec<BigUint> = values.into_iter().map(jsvalue_to_biguint).collect();
let scalars: Vec<BigUint> = scalars.into_iter().map(jsvalue_to_biguint).collect();
let result = crate::msm::msm_calldata_builder(&values, &scalars, curve_id);
result.into_iter().map(bigint_to_jsvalue).collect()
}

fn jsvalue_to_biguint(v: JsValue) -> BigUint {
let s = (JsValue::from_str("") + v).as_string().unwrap();
BigUint::from_str(&s).expect("Failed to convert value to non-negative bigint")
}

fn bigint_to_jsvalue(v: BigInt) -> JsValue {
JsValue::bigint_from_str(&v.to_string())
}

#[cfg(test)]
mod tests {
use super::*;
use num_bigint::{BigInt, BigUint};
use wasm_bindgen_test::wasm_bindgen_test;

// This test runs only in wasm32-unknown-unknown targets
// wasm-pack test --node --release --no-default-features
#[wasm_bindgen_test]
pub fn test_bigint_marshalling() {
let v = 31415usize;
assert_eq!(
jsvalue_to_biguint(bigint_to_jsvalue(BigInt::from(v))),
BigUint::from(v)
);
}
}
193 changes: 193 additions & 0 deletions tools/garaga_rs/wasm-pack-build-and-patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
#!/usr/bin/env node

// This script will patch a 'wasm-pack --target web' output folder
// to make the code platform agnostic (browser vs node.js/commonjs vs esm)

// Sample usage of garaga_rs as an NPM package:
// import * as garaga_rs from 'garaga_rs';
// async function main() {
// await garaga_rs.init();
// garaga_rs.msm_calldata_builder(/* ...args... */);
// }

const fs = require('fs');
const path = require('path');
const child_process = require('child_process');

function transpileExports(input) {
// this is a simple/non-comprehensive implementation
// can be replaced by a real transpiler (e.g. npx mjs-to-cjs in.mjs out.cjs)
return input
// deals with: export function ...
.replace(/\bexport\s+function\s+(\w+)\s*\(/g, 'exports.$1 = $1;\nfunction $1(')
// deals with: export { ... }
.replace(/\bexport\s*\{([\w\s,]*)\}\s*;/g, (match, $1) => {
const ids = $1.split(',').map((id) => id.trim());
return ids.map((id) => 'exports.' + id + ' = ' + id + ';').join('\n');
});
}

function patch() {
// package folder
const pkgFolder = path.join(__dirname, 'pkg');

// package file name/path
const jsonName = 'package.json';
const jsonFile = path.join(pkgFolder, jsonName);

// package name
const pkgName = JSON.parse(fs.readFileSync(jsonFile, 'utf8'))['name'];

// typing file name/path
const tsName = pkgName + '.d.ts';
const tsFile = path.join(pkgFolder, tsName);

// old file names/paths
const jsName = pkgName + '.js';
const jsFile = path.join(pkgFolder, jsName);

const wasmName = pkgName + '_bg.wasm';
const wasmFile = path.join(pkgFolder, wasmName);

const wasmtsName = pkgName + '_bg.wasm.d.ts';
const wasmtsFile = path.join(pkgFolder, wasmtsName);

// new file names/paths
const cjsName = pkgName + '.cjs';
const cjsFile = path.join(pkgFolder, cjsName);

const mjsName = pkgName + '.mjs';
const mjsFile = path.join(pkgFolder, mjsName);

const wasmcjsName = pkgName + '_bg.wasm.cjs';
const wasmcjsFile = path.join(pkgFolder, wasmcjsName);

const wasmmjsName = pkgName + '_bg.wasm.mjs';
const wasmmjsFile = path.join(pkgFolder, wasmmjsName);

// patches .json file:
// - sets .cjs as main
// - sets .mjs as module
// - sets sideEffects to false
// - replaces references to .js file by .cjs and .mjs files
// - replaces references to .wasm file by .wasm.cjs and .wasm.mjs files
// - adds exports section
{
const input = fs.readFileSync(jsonFile, 'utf8');
const json = JSON.parse(input);
json['main'] = cjsName;
json['module'] = mjsName;
json['sideEffects'] = false;
json['files'] = json['files'] || [];
json['files'] = json['files'].filter((name) => ![jsName, wasmName].includes(name));
json['files'].push(cjsName);
json['files'].push(mjsName);
json['files'].push(wasmcjsName);
json['files'].push(wasmmjsName);
json['exports'] = json['exports'] || {};
json['exports']['.'] = json['exports']['.'] || {};
json['exports']['.']['require'] = './' + json['main'];
json['exports']['.']['import'] = './' + json['module'];
json['exports']['.']['types'] = './' + json['types'];
const output = JSON.stringify(json, undefined, 2);
fs.writeFileSync(jsonFile, output, 'utf8');
}

// patches .d.ts file:
// - adds an export entry for the init() function
// - removes the default export
{
const exportFunc = 'export function init(): Promise<InitOutput>;';
const input = fs.readFileSync(tsFile, 'utf8')
.replace(/\bexport\s+default\s+[^;]+;/, '');
const output =
exportFunc + '\n\n' +
input;
fs.writeFileSync(tsFile, output, 'utf8');
}

// creates .mjs file:
// - uses .wasm.mjs file as the default WASM module
// - adds and exports the init() function
// - replaces the default WASM load behavior via URL by direct use
// - removes the default export
{
const importStmt = 'import default_module_or_path from \'./' + wasmmjsName + '\';';
const exportFunc = 'export function init() { return __wbg_init({ module_or_path: default_module_or_path }); }';
const input = fs.readFileSync(jsFile, 'utf8')
.replace('new URL(\'' + wasmName + '\', import.meta.url)', 'default_module_or_path')
.replace(/\bexport\s+default\s+[^;]+;/, '');
const output =
importStmt + '\n\n' +
exportFunc + '\n\n' +
input;
fs.writeFileSync(mjsFile, output, 'utf8');
}

// creates .cjs file:
// - uses .wasm.cjs file as the default WASM module
// - adds and exports the init function
// - replaces the default WASM load behavior via URL by the direct use
// - removes the default export
// - transpiles ESM style exports to CommonJS style exports
{
const importStmt = 'const default_module_or_path = require(\'./' + wasmcjsName + '\');';
const exportFunc = 'exports.init = init;\nfunction init() { return __wbg_init({ module_or_path: default_module_or_path }); }';
const input = fs.readFileSync(jsFile, 'utf8')
.replace('new URL(\'' + wasmName + '\', import.meta.url)', 'default_module_or_path')
.replace(/\bexport\s+default\s+[^;]+;/, '');
const output =
'\'use strict\';' + '\n\n' +
importStmt + '\n\n' +
exportFunc + '\n\n' +
transpileExports(input);
fs.writeFileSync(cjsFile, output, 'utf8');
}

// encodes .wasm file as .wasm.cjs:
// - creates .wasm.cjs exporting the binary contents of the .wasm file
{
const input = fs.readFileSync(wasmFile).toString('base64');
const output =
'\'use strict\';' + '\n\n' +
'module.exports = "data:application/wasm;base64,' + input + '";';
fs.writeFileSync(wasmcjsFile, output, 'utf8');
}

// encodes .wasm file as .wasm.mjs:
// - creates .wasm.mjs exporting the binary contents of the .wasm file
{
const input = fs.readFileSync(wasmFile).toString('base64');
const output =
'export default "data:application/wasm;base64,' + input + '";';
fs.writeFileSync(wasmmjsFile, output, 'utf8');
}

// cleans up now obsolete files:
// - removes .js, .wasm, and .wasm.d.ts files
{
fs.rmSync(jsFile);
fs.rmSync(wasmFile);
fs.rmSync(wasmtsFile);
}
}

function build() {
// executes wasm-pack in the current folder
const command = 'wasm-pack build --target web --release';
feltroidprime marked this conversation as resolved.
Show resolved Hide resolved
child_process.execSync(command, { cwd: __dirname });
}

function clean() {
// removes the pkg folder
const pkgFolder = path.join(__dirname, 'pkg');
fs.rmSync(pkgFolder, { recursive: true, force: true });
}

function main() {
clean();
build();
patch();
}

main();
Loading
Loading