Ava shared workers

[sathishsoundharajan]

Hello Team,

I'm trying out shared workers concept in ava. Idea here is to create hapijs server in shared worker and share that server uri across the test files.

Here is the repro https://github.com/sathishsoundharajan/shared-workers

When i'm trying to run 2 test files simultaneously only one of the test files stuck and it always fails

[novemberborn]

You're only registering the server after it's ready, which means there is a race condition if both test files request it. Instead write the promise to the map, and then await that for each request from the test files.

Also, you'll want to use this with AVA 4 where the test files run in worker threads, rather than child processes, which should make it all a bit quicker.

[sathishsoundharajan]

Thank you @novemberborn

1. I cannot upgrade to ava 4 because my Node.js version is stuck at v12.16.1 ( for UT alone i'm planning to run it in v12.17 because ava requires it )

2. Please correct me if i'm wrong, are you saying await bootstrap() function which actually starts the server and stores inside sharedContexts map will have race condition if both test files are running simultaneously ? If that is the case based on your workaround provided by you i have to remove the await which surrounds the bootstrap() function, but if i do that, i will run into these errors

3. Error: #<Promise> could not be cloned

4. Server object cannot be cloned using zlib

5. Also if each test file awaits the server start up it will would create multiple servers right ?

If i have understood wrongly from your feedback, please correct me, if possible can u give me small snippet ?

[sathishsoundharajan]

On further debugging https://github.com/avajs/ava/blob/v3.15.0/lib/worker/ipc.js#L167-L168 this specific event emitter does not getting any message that is reason 1 of the test files are stuck.

[sathishsoundharajan]

Even running the test-files serially also does not solve the problem.

{
      "type": "node",
      "request": "launch",
      "name": "Debug AVA test file",
      "runtimeExecutable": "${workspaceFolder}/node_modules/.bin/ava",
      "args": [
        "--serial"
      ],
      "runtimeArgs": [
        "tests/test-1.js", "tests/test-2.js"
      ],
      "outputCapture": "std",
      "skipFiles": [
        "<node_internals>/**/*.js"
      ]
    }

[sathishsoundharajan]

I'm beginning to think the POC code i have added is completely wrong and ava shared workers should be used very differently..

[novemberborn]

Oh, it's this line! https://github.com/sathishsoundharajan/shared-workers/blob/571ddc35f32028852600137baa8cab7e13371e94/tests/_worker.js#L13

You're in a for await loop which receives each message. That line breaks out of the loop.

I think you're trying to model this after https://github.com/avajs/cooperate but that's too advanced for your needs. Try this:

const bootstrap = require('../server');

module.exports = async ({ negotiateProtocol }) => {
  const protocol = negotiateProtocol(['experimental']).ready();

  let server;
  for await (const message of protocol.subscribe()) {
    if (!server) {
      // Other messages will not be processed until this completes.
      server = await bootstrap();
    }
    message.reply(server.info.uri);
  }
};

const path = require('path');
const { registerSharedWorker } = require('ava/plugin');

const protocol = registerSharedWorker({
  filename: path.resolve(__dirname, './_worker.js'),
  supportedProtocols: ['experimental']
});

module.exports = async () => {
  await protocol.available;

  // The communication is straight-forward so we don't need to wrap it in objects:
  for await (const reply of protocol.publish().replies()) {
    // We only want this one reply so we break out of the loop.
    return reply.data;
  }
};

[sathishsoundharajan]

Wow.. Simple return statement :) .. Had to debug this for 5 days..
Hitting my head why didn't see that ..

Thank you @novemberborn

[sathishsoundharajan]

I also wanted to explain why i'm trying this POC more. My Node.js app which current uses Hapi.js framework has 3500+ test cases.

In each test file we are trying to create hapi server ( without starting, just initializing it along with plugins ) and use the server.inject capability provided by hapi to test the API's.

Our current Unit test suite runs for more than 1h 30 mins for each commit. We believe this is due to following reasons

1. nodejs require is too slow because even though our app is micro service we do have more than 200+ files and we use numerous number of node_modules too. So each time ava create process to run the test file (server initialisation and requires takes 15 to 20 seconds to complete and then only it starts to execute the test cases )

2. We also use nocks for request stubbing & sinon for method stubbing, due to avajs nature of running the test files in parallel & even test cases inside the files in parallel nocks request/response stubbing won't work ( as nocks stubs global http method ) we had to serialise our test cases using test.serial.

For these reasons, to solve the first issue i thought of instead of requiring and initialising the app for test process, let's try to use the shared workers concept to create hapijs server once in main process and share that server to test-files. ( which i believe this will reduce test-files start up times )

For the second issue, i don't have any idea on how to solve it :) if anyone from the community able to provide some feedback will be helpful.

[novemberborn]

Since stubs modify shared resources you need to then test serially.

[sathishsoundharajan]

Then i'm not able fully useava potential :(