Jm/ext ts guest new (#125)

* Jm/ext ts host (#117) Scale-65 This adds typescript host support for extensions. Scale-65 --------- Signed-off-by: Jimmy Moore <[email protected]> Signed-off-by: Shivansh Vij <[email protected]> Co-authored-by: Shivansh Vij <[email protected]> Signed-off-by: Jimmy Moore <[email protected]> * Initial extension generator code Signed-off-by: Jimmy Moore <[email protected]> First version with guest/host generated in cli working Signed-off-by: Jimmy Moore <[email protected]> Couple changes Signed-off-by: Jimmy Moore <[email protected]> First ver compile golang guest Signed-off-by: Jimmy Moore <[email protected]> Updated to first version working guest+host in go Signed-off-by: Jimmy Moore <[email protected]> Extensions now get reset at start of a run Signed-off-by: Jimmy Moore <[email protected]> Updated to use scale-extension-interfaces Signed-off-by: Jimmy Moore <[email protected]> Updated extension host go.mod to include correct ver of scale-extension-interfaces Signed-off-by: Jimmy Moore <[email protected]> Started on rust extension impl Signed-off-by: Jimmy Moore <[email protected]> Fixup for golang extension Signed-off-by: Jimmy Moore <[email protected]> Initial rust generator for types from signature to extension Signed-off-by: Jimmy Moore <[email protected]> Updated ext generator for rust Signed-off-by: Jimmy Moore <[email protected]> Updated for rust compile Signed-off-by: Jimmy Moore <[email protected]> Latest updates Signed-off-by: Jimmy Moore <[email protected]> Latest rust guest ext working Signed-off-by: Jimmy Moore <[email protected]> Updated to use extension hash rather than name in wasm fns Signed-off-by: Jimmy Moore <[email protected]> Fixing for tests Signed-off-by: Shivansh Vij <[email protected]> Signed-off-by: Jimmy Moore <[email protected]> Pulled in changes from staging, and fixed things up. e2e tests working on scale-cli Signed-off-by: Jimmy Moore <[email protected]> Removed unnecessary templates etc Signed-off-by: Jimmy Moore <[email protected]> Initial extension generator code Signed-off-by: Jimmy Moore <[email protected]> First ver compile golang guest Signed-off-by: Jimmy Moore <[email protected]> Extensions now get reset at start of a run Signed-off-by: Jimmy Moore <[email protected]> Updated to use scale-extension-interfaces Signed-off-by: Jimmy Moore <[email protected]> * Start on typescript generator Signed-off-by: Jimmy Moore <[email protected]> * First parts of ts ext templating Signed-off-by: Jimmy Moore <[email protected]> * Updated config.ts and module.ts to deal with extensions. Added scale-extension-interfaces import. Signed-off-by: Jimmy Moore <[email protected]> Updated ext generator Signed-off-by: Jimmy Moore <[email protected]> First ver of typescript extension generating host types etc Signed-off-by: Jimmy Moore <[email protected]> Module now correctly returns BigInt on ext function Signed-off-by: Jimmy Moore <[email protected]> Added extension impl interface typescript Signed-off-by: Jimmy Moore <[email protected]> Updated ts host templates. Just need function bodies now Signed-off-by: Jimmy Moore <[email protected]> First version of typescript host working e2e Signed-off-by: Jimmy Moore <[email protected]> Rebased to jm/ext and started tidying up Signed-off-by: Jimmy Moore <[email protected]> Reverted scalefile change, removed unused templates Signed-off-by: Jimmy Moore <[email protected]> Fixed up typescript type generation in extension Signed-off-by: Jimmy Moore <[email protected]> Removed dupe type templates in typescript extensions Signed-off-by: Jimmy Moore <[email protected]> Added generator for typescript extension host integration test Signed-off-by: Jimmy Moore <[email protected]> Added ts host generator, and start of extension test typescript Signed-off-by: Jimmy Moore <[email protected]> ts host ext tests passing Signed-off-by: Jimmy Moore <[email protected]> Lint golang fix Signed-off-by: Jimmy Moore <[email protected]> Fixed ts lint any issue Signed-off-by: Jimmy Moore <[email protected]> Fixed rust templ Signed-off-by: Jimmy Moore <[email protected]> Clean up pr Signed-off-by: Jimmy Moore <[email protected]> * Changes to typescript builder for mux passthrough Signed-off-by: Jimmy Moore <[email protected]> * Added ts guest template, generator. Adjusted build to call out to wasm-toolkit to do mux replaces Signed-off-by: Jimmy Moore <[email protected]> ts ext test compiling. Just need to have it call extension Signed-off-by: Jimmy Moore <[email protected]> Typescript guest extension passing integration tests Signed-off-by: Jimmy Moore <[email protected]> Test fix Signed-off-by: Jimmy Moore <[email protected]> Lint fixes Signed-off-by: Jimmy Moore <[email protected]> Lint fix Signed-off-by: Jimmy Moore <[email protected]> Add: js_builder-x86_64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Add: js_builder-aarch64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> More lint Signed-off-by: Jimmy Moore <[email protected]> Add: js_builder-aarch64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Add: js_builder-x86_64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Added missing index.ts for ts ext test Signed-off-by: Jimmy Moore <[email protected]> Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Jimmy Moore <[email protected]> * interfaces removed Signed-off-by: Jimmy Moore <[email protected]> * Add: js_builder-x86_64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-aarch64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-aarch64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * chgs to default packaged js builds * update signature lockfile * Updated js_builder with rustv1.74.1. Updated workflows pinned to v1.74.1 for now * Add: js_builder-x86_64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-aarch64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-aarch64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Lint go fix * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-aarch64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * linter ci throwing non-specifics * CI linter update * Add: js_builder-aarch64-unknown-linux-gnu Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-apple-darwin Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Add initial ext docs Signed-off-by: Daniel Phillips <[email protected]> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * More extension docs Signed-off-by: Daniel Phillips <[email protected]> * Add: js_builder-x86_64-pc-windows-msvc Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> --------- Signed-off-by: Jimmy Moore <[email protected]> Signed-off-by: Shivansh Vij <[email protected]> Signed-off-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel Phillips <[email protected]> Co-authored-by: Shivansh Vij <[email protected]> Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Daniel Phillips <[email protected]> Co-authored-by: Daniel Phillips <[email protected]>
loopholelabs · Mar 8, 2024 · 2e3d8fc · 2e3d8fc
1 parent 64dbccf
commit 2e3d8fc
Show file tree

Hide file tree

Showing 60 changed files with 9,419 additions and 227 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -41,8 +41,8 @@ jobs:
 
       - name: Set up Rust
         run: |
-          rustup toolchain install stable --profile minimal --no-self-update
-          rustup default stable
+          rustup toolchain install 1.74.1 --profile minimal --no-self-update
+          rustup default 1.74.1
           rustup target add wasm32-wasi
         shell: bash
 

diff --git a/.github/workflows/typescript_compiler.yml b/.github/workflows/typescript_compiler.yml
@@ -13,8 +13,8 @@ jobs:
         uses: actions/checkout@v3
       - name: Set up Rust
         run: |
-          rustup toolchain install stable --profile minimal --no-self-update
-          rustup default stable
+          rustup toolchain install 1.74.1 --profile minimal --no-self-update
+          rustup default 1.74.1
           rustup target add wasm32-wasi
         shell: bash
 
@@ -64,8 +64,8 @@ jobs:
 
       - name: Set up Rust
         run: |
-          rustup toolchain install stable --profile minimal --no-self-update
-          rustup default stable
+          rustup toolchain install 1.74.1 --profile minimal --no-self-update
+          rustup default 1.74.1
           rustup target add ${{ matrix.name }}
         shell: bash
 
@@ -95,4 +95,4 @@ jobs:
           commit: "--signoff"
           message: "Add: js_builder-${{ matrix.name }}"
           add: compile/typescript/builder/${{ matrix.output }}
-          pull: "--rebase --autostash"
+          pull: "--rebase --autostash"
diff --git a/.golangci.yaml b/.golangci.yaml
@@ -1,7 +1,6 @@
 linters:
   enable:
     - gocritic
-    - goimports
     - gosimple
     - govet
     - ineffassign
@@ -17,6 +16,7 @@ linters:
     - gocognit
     - gocyclo
     - goconst
+    - goimports
 
 linters-settings:
   gocognit:
@@ -35,4 +35,4 @@ linters-settings:
   govet:
     # enable or disable analyzers by name
     disable:
-      - shadow
+      - shadow
diff --git a/build/typescript.go b/build/typescript.go
@@ -17,6 +17,7 @@
 package build
 
 import (
+	"bytes"
 	"encoding/hex"
 	"errors"
 	"fmt"
@@ -25,15 +26,18 @@ import (
 	"os/exec"
 	"path"
 	"path/filepath"
+	"strconv"
 	"strings"
 
-	"github.com/loopholelabs/scale/extension"
-
 	"github.com/evanw/esbuild/pkg/api"
 
+	"github.com/loopholelabs/wasm-toolkit/pkg/customs"
+	"github.com/loopholelabs/wasm-toolkit/pkg/wasm/wasmfile"
+
 	"github.com/loopholelabs/scale/compile/typescript"
 	"github.com/loopholelabs/scale/compile/typescript/builder"
-
+	"github.com/loopholelabs/scale/extension"
+	extGen "github.com/loopholelabs/scale/extension/generator/typescript"
 	"github.com/loopholelabs/scale/scalefile"
 	"github.com/loopholelabs/scale/scalefunc"
 	"github.com/loopholelabs/scale/signature"
@@ -304,11 +308,77 @@ func LocalTypescript(options *LocalTypescriptOptions) (*scalefunc.V1BetaSchema,
 		return nil, fmt.Errorf("unable to compile scale function using js_builder: %w", err)
 	}
 
-	data, err := os.ReadFile(path.Join(build.Path, "scale.wasm"))
+	// Do the extension transform here...
+	wfile, err := wasmfile.New(path.Join(build.Path, "scale.wasm"))
 	if err != nil {
 		return nil, fmt.Errorf("unable to read compiled wasm file: %w", err)
 	}
 
+	confImp := customs.RemapMuxImport{
+		Source: customs.Import{
+			Module: "env",
+			Name:   "ext_mux",
+		},
+		Mapper: map[uint64]customs.Import{},
+	}
+
+	confExp := customs.RemapMuxExport{
+		Source: "ext_resize",
+		Mapper: map[uint64]string{},
+	}
+
+	// Go through extensions and add the required mappings for wasm adjustment.
+	for _, extSchema := range options.ExtensionSchemas {
+
+		h, err := extSchema.Hash()
+		if err != nil {
+			return nil, fmt.Errorf("unable to get extension hash")
+		}
+		hash := hex.EncodeToString(h)
+		id, err := strconv.ParseUint(hash[0:8], 16, 64)
+		if err != nil {
+			return nil, fmt.Errorf("unable to get extension hash")
+		}
+		confExp.Mapper[id] = fmt.Sprintf("ext_%s_Resize", hash)
+
+		// Now go through all functions in the extension...
+		for _, f := range extSchema.Functions {
+			fname := fmt.Sprintf("ext_%s_%s", hash, f.Name)
+			fid := extGen.GetCallID(hash, "", f.Name)
+			confImp.Mapper[fid] = customs.Import{
+				Module: "env",
+				Name:   fname,
+			}
+		}
+
+		for _, ifc := range extSchema.Interfaces {
+			for _, f := range ifc.Functions {
+				fname := fmt.Sprintf("ext_%s_%s_%s", hash, ifc.Name, f.Name)
+				fid := extGen.GetCallID(hash, ifc.Name, f.Name)
+				confImp.Mapper[fid] = customs.Import{
+					Module: "env",
+					Name:   fname,
+				}
+			}
+		}
+	}
+
+	err = customs.MuxImport(wfile, confImp)
+	if err != nil {
+		return nil, fmt.Errorf("unable to parse extension import remap")
+	}
+
+	err = customs.MuxExport(wfile, confExp)
+	if err != nil {
+		return nil, fmt.Errorf("unable to parse extension export remap")
+	}
+
+	var wasmBin bytes.Buffer
+	err = wfile.EncodeBinary(&wasmBin)
+	if err != nil {
+		return nil, fmt.Errorf("unable to encode wasm binary")
+	}
+
 	signatureHash, err := options.SignatureSchema.Hash()
 	if err != nil {
 		return nil, fmt.Errorf("unable to hash signature: %w", err)
@@ -346,7 +416,7 @@ func LocalTypescript(options *LocalTypescriptOptions) (*scalefunc.V1BetaSchema,
 		Language:   scalefunc.TypeScript,
 		Manifest:   packageJSONData,
 		Stateless:  options.Scalefile.Stateless,
-		Function:   data,
+		Function:   wasmBin.Bytes(),
 	}, nil
 
 }
diff --git a/compile/typescript/builder/interpreter/main.rs b/compile/typescript/builder/interpreter/main.rs
@@ -19,7 +19,7 @@ pub mod time;
 
 use once_cell::sync::OnceCell;
 use quickjs_wasm_sys::{
-    ext_js_undefined, JSContext, JSRuntime, JSValue, JS_BigIntToUint64, JS_Call,
+    ext_js_undefined, JSContext, JSRuntime, JSValue, JS_BigIntToUint64, JS_NewBigUint64, JS_Call,
     JS_DefinePropertyValueStr, JS_Eval, JS_GetArrayBuffer, JS_GetException, JS_GetGlobalObject,
     JS_GetPropertyStr, JS_GetPropertyUint32, JS_IsError, JS_NewContext, JS_NewInt32_Ext,
     JS_NewObject, JS_NewRuntime, JS_NewUint32_Ext, JS_EVAL_TYPE_GLOBAL, JS_PROP_C_W_E,
@@ -46,9 +46,12 @@ static mut ENTRY_MAIN: OnceCell<JSValue> = OnceCell::new();
 static mut ENTRY_INITIALIZE: OnceCell<JSValue> = OnceCell::new();
 static mut ENTRY_RUN: OnceCell<JSValue> = OnceCell::new();
 static mut ENTRY_RESIZE: OnceCell<JSValue> = OnceCell::new();
+static mut ENTRY_EXT_RESIZE: OnceCell<JSValue> = OnceCell::new();
 
 static SCRIPT_NAME: &str = "index.js";
 
+pub const TYPESCRIPT_EXT_MUX: &str = "scale_ext_mux";
+
 // If the wizer_opt feature is enabled, we will export a function called wizer.initialize
 // so wizer knows what entrypoint to use.
 #[cfg(feature = "wizer_opt")]
@@ -111,6 +114,14 @@ fn initialize_runtime() {
             scale_signature_interfaces::TYPESCRIPT_NEXT,
             &next_wrap,
         );
+
+        helpers::set_callback(
+          context,
+          global,
+          TYPESCRIPT_EXT_MUX,
+          &ext_mux_wrap,
+        );
+
         helpers::set_callback(
             context,
             global,
@@ -221,6 +232,10 @@ fn initialize_runtime() {
         let resize_fn = JS_GetPropertyStr(context, exports, resize_key.as_ptr());
         ENTRY_RESIZE.set(resize_fn).unwrap();
 
+        let ext_resize_key = CString::new("ext_resize").unwrap();
+        let ext_resize_fn = JS_GetPropertyStr(context, exports, ext_resize_key.as_ptr());
+        ENTRY_EXT_RESIZE.set(ext_resize_fn).unwrap();
+
         ENTRY_EXPORTS.set(exports).unwrap();
         JS_RUNTIME.set(runtime).unwrap();
         JS_CONTEXT.set(context).unwrap();
@@ -476,6 +491,88 @@ fn next_wrap(
     }
 }
 
+#[link(wasm_import_module = "env")]
+extern "C" {
+    #[link_name = "ext_mux"]
+    fn _ext_mux(id: u64, instance: u64, ptr: u32, size: u32) -> u64;
+}
+
+// Wrap the exported ext_mux function so it can be called from the js runtime
+// from JS, it should be called as scale_ext_mux(id: bigint, instance: bigint, ptr: number, len: number) -> bigint
+fn ext_mux_wrap(
+    context: *mut JSContext,
+    _: JSValue,
+    _: c_int,
+    js_value: *mut JSValue,
+    _: c_int,
+) -> JSValue {
+    unsafe {
+        let js_id = JS_GetPropertyUint32(context, *js_value, 0);
+        let mut id: u64 = 0; 
+        JS_BigIntToUint64(context, &mut id, js_id);
+
+        let js_instance = JS_GetPropertyUint32(context, *js_value, 1);
+        let mut instance: u64 = 0; 
+        JS_BigIntToUint64(context, &mut instance, js_instance);
+
+        let ptr = JS_GetPropertyUint32(context, *js_value, 2) as u32;
+        let len = JS_GetPropertyUint32(context, *js_value, 3) as u32;
+        let v = _ext_mux(id, instance, ptr, len);
+        return JS_NewBigUint64(context, v);
+    }
+}
+
+// from ext_resize(id: bigint, size: number) -> number
+#[export_name = "ext_resize"]
+#[no_mangle]
+pub extern "C" fn ext_resize(id: u64, size: u32) -> *mut u8 {
+    unsafe {
+        let context = JS_CONTEXT.get().unwrap();
+        let exports = ENTRY_EXPORTS.get().unwrap();
+        let extresizefn = ENTRY_EXT_RESIZE.get().unwrap();
+
+        let mut args: Vec<JSValue> = Vec::new();
+        let jval_id = JS_NewBigUint64(*context, id);
+        args.push(jval_id);
+        let jval = JS_NewInt32_Ext(*context, size as i32);
+        args.push(jval);
+
+        let ret = JS_Call(
+            *context,
+            *extresizefn,
+            *exports,
+            args.len() as i32,
+            args.as_slice().as_ptr() as *mut JSValue,
+        );
+
+        if (ret >> 32) as i32 == JS_TAG_EXCEPTION {
+            let e = JS_GetException(*context);
+            let exception =
+                helpers::to_exception(*context, e).expect("getting exception during resize failed");
+
+            let mut stack = None;
+            let is_error = JS_IsError(*context, e) != 0;
+            if is_error {
+                let cstring_key = CString::new("stack")
+                    .expect("getting new CString for JS stack during resize failed");
+                let raw = JS_GetPropertyStr(*context, e, cstring_key.as_ptr());
+                if (raw >> 32) as i32 != JS_TAG_UNDEFINED {
+                    stack.replace(helpers::to_exception(*context, raw));
+                }
+            }
+            let mut err = format!("exception from js runtime during resize: {}", exception);
+            if let Some(Ok(stack)) = stack {
+                if stack.len() > 0 {
+                    err.push_str(&format!("\nstack:\n{stack}"));
+                }
+            }
+            panic!("{}", err);
+        }
+
+        ret as *mut u8
+    }
+}
+
 // Wrap the exported address_of function so it can be called from the js runtime
 fn address_of_wrap(
     context: *mut JSContext,