test compiler output with sim API

[ekeren]

Wanted to start a discussion about the option to test the compiler using the sim api.
Here is a small example of how one test can look like (using jest)

code.w

bring cloud;

let bucket = new cloud.Bucket();
inflight my_inflight(a:str): str{
  bucket.put("file.txt", a);
};

new cloud.Function(my_inflight);

basic.test.ts

import * as sdk from '@winglang/wingsdk';
import * as child from 'child_process';

//TOOD replace this with something that is not quick and dirty 
const SIM_FILE = __dirname + "/../app.wx";
const WING_COMMAND = "/Users/eyalkeren/monada-workspace/wing-test-propsal/node_modules/.bin/wing";
const WING_CODE =  __dirname + "/code.w"

let simulator : sdk.testing.Simulator
describe('basic compiler tests', () => { 
  beforeAll(async () => {
    child.execSync(WING_COMMAND + " compile --target sim " + WING_CODE )
  });
  beforeEach(async ()=> {
    simulator = new sdk.testing.Simulator({simfile : SIM_FILE}) //TOOD
    await simulator.start()
  });
  afterEach(async () => {
    await simulator.stop()
  })
  test('test function writes to bucket', async () => {
    let bucket = simulator.getResourceByPath("root/cloud.Bucket") as sdk.sim.IBucketClient
    let fn = simulator.getResourceByPath("root/cloud.Function") as sdk.sim.IFunctionClient
    await fn.invoke("somevalue")
    let output = await bucket.get('file.txt')
    expect(output).toBe('somevalue')
  });
});

[Chriscbr]

Totally! This is a great idea 👍

[eladb]

child.execSync(WING_COMMAND + " compile --target sim " + WING_CODE)

I think we should expose our compiler as a library API (I even suspect it already is exported), so spawning a child process is not needed.

So basically, I'd want something like:

import { compile } from '@winglang/wing';
import * as testing from '@winglang/sdk';

const wxpath = await compile('./hello.w', { target: 'sim' });
const sim new testing.Simulator({ simfile: wxpath });
// ...

[ekeren]

const wxpath = await compile('./hello.w', { target: 'sim' });

@yoav-steinberg , do we have an API to do this?

[Chriscbr]

We could expose the compile command through the CLI (npm library). Want to create an issue?

[ekeren]

Done #882

[ekeren]

I've been playing around with writing unit tests in Wing that can run on sim and on the clouds.
Eventually something that will can invoked as wing test --target <target>

I've created a not-so-real example, in order to display how this can work.
Off course, reporting of test count and failures will come out of the box eventually.

bring cloud;

let all_tests = new TestSuite("My Example");

let test_counter = new cloud.Counter() as "test_counter";
let TEST_STARTED = "tests started";
let TEST_PASS = "tests pass";
let TEST_FAIL = "tests failed";

// Runs the following inflight function before any inflight test is running in this suite
all_tests.before("collect test statistics", inflight (test: Test) => {
  test_counter.inc(1, key: TEST_STARTED);
});

// Runs the following inflight function after any inflight test is running in this suite
all_tests.after("collect test statistics", inflight (test: Test) => {
  if test.test_result.pass {
    test_counter.inc(1, key: TEST_PASS);
  } 
  if (tests.test_result.fail) {
    test_counter.inc(1, key: TEST_FAIL);
  }
});

// Runs this inflight function after all tests has finished running in this suite
all_tests.after_all("report amount of tests runned", inflight () => {
  print("tests: ${counter.get(key: TEST_STARTED)}\n"
        + "\tpass: ${counter.get(key: TEST_STARTED)}\n"
        + "\tfail:${counter.get(key: TEST_FAIL)}");
});

// (If we want to support running test in preflight)
// Adding a test for checking preflight behavior that has nothing to do with inflight
all_tests.add_test("Bucket public property", (test: Test): void => {
  let bucket = new Cloud.bucket(public: true); 
  test.assert.is_true(bucket.public);  
});

// Adding a test for checking inflight behavior that has nothing to do with preflight
all_tests.add_test("num + arithmetic", inflight (test: Test): void => {
  let x = 23; 
  let y = 19; 
  test.assert.equal(42, x + y);  
});

  
/**
* Adding a suite of inflight tests that have the same preflight setup
* This code will generate multiple instance of Test 
* resources, having a single Test resource for every inflight test (leaf) 
* that is added to the suite,  so all inflight tests are isolated and can 
* run simultaneously 
**/  
all_tests.add_suite("bucket", (bucket_suite: TestSuite): void => { 
  let bucket = new cloud.Bucket();
  // this is a before inflight function under this test 
  bucket_suite.before("clean the bucket", inflight (test: Test): void => {
    for f in bucket.list() {
      bucket.delete(f);
    }
  });
  
  // Adding another suite inside a suite is possible 
  bucket_suite.add_suite("CRUD", (crud_suite: TestSuite): void => { 
    let file_name = "file";
    
    crud_suite.add_test("put and get", inflight (test: Test): void => {
      let expected_content = "some data";
      bucket.put(file_name, expected_content);
      test.assert.equal(expected_content, bucket.get(file_name));
    });

    crud_suite.add_test("list and delete operation", inflight (test: Test): void => {
      test.assert.equal(0, bucket.list().length);
      bucket.put(file_name, "some content");
      test.assert.equal(1, bucket.list().length);
      bucket.delete(file_name);
      test.assert.equal(0, bucket.list().length);
    });
  });

  bucket_suite.add_suite("events", (event_suite: TestSuite): void => { 
    let counter = new cloud.Counter();
    let uploaded_file_name = "uploaded";
    bucket.on_upload(inflight (key: str): void => {
      counter.inc();
      // TestSuite.current_test is an inflight API of TestSuite resource
      event_suite.current_test.assert.equals(uploaded_file_name, key);
    });
    
    events_suite.add_test("upload a file", inflight (test: Test): void => {
      bucket.upload(uploaded_file_name);
    });
  });
});

[Chriscbr]

Awesome example @ekeren !

There's definitely value in having tests for preflight. We saw this a lot with CDK -- for example, taking snapshots of your low-level infrastructure, and adding assertions like "When I create a Foo, I expect the generated infrastructure includes at least one GCP function and one GCP bucket". Anything that was possible to test in AWS CDK (or CDKTF, etc.) should also be doable in Wing.

Open question: for inflight tests, do you think it makes sense to be able to compile this test suite to run both on a simulator and a cloud provider? (Just like I can compile any other Wing program to different clouds?) I could see this being useful (1) for Wing users, but also (2) for us building the SDK (when I implement a new feature X on bucket, I want to assert it behaves the same way no matter where I deploy it...)

[ekeren]

The above example should work on any target (sim and any cloud provider)

I'll add this to the description