feat: Track and allow querying in-flight requests (#186)

* test(pending): add tests for pending requests

 - update middleware to support delayed responses

* chore(types): update shape of `InterceptedRequest`

 - `addRequest` always sets `pending: false`
 - while `typeof null === 'object'`, clarify that body and response.body can be null.

* feat(types): update type signatures for `getRequest`, `getRequests`

 - if not passing options object (all past code), will continue to receive completed requests only
 - if passing options object with includePending = false, will receive completed requests only
 - if passing options object with includePending = true, will receive all requests.

* feat: add `hasPendingRequests` test method

 - refactor tests for pending requests into own describe block, and add explicit waits to avoid affecting following tests.

* refactor(types): switch from property declaration to function declaration

Support function overloads in future work

* refactor(sessionStorage): add helper method for support checking, extract method for getting contents

 - add runtime check for incomplete polyfill scenario, where fetch is polyfilled but Promise.all is not.

* feat(getRequest): attach pending property to incomplete retrieved requests

& parse the response body only if marked as fulfilled

* fix(tests): add missed awaits to `assert.rejects` usage

 - in sync mode, these were awaited implicitly. In async mode, however, they must be explicitly awaited.
 - Increase assertion strictness on the matched errors
   - due to an issue with how assert.rejects forwards a regexp containing an escaped regexp, use a validation function to directly invoke assert.matches with the regexp

* feat(fetch-api): add basic support for pending request queries

 - add requests prior to invoking window._fetch
 - complete requests after the full response body is available

* feat(xhr-api): add basic support for pending request queries

 - note: must stringify `lastURL` in case XHR#open is invoked with a URL object.

* refactor(xhr-api): simplify parsePayload method

 - replace `if-else if` chain with `if + return`
 - log the serialization error, if any

* feat(logs): prefix any emitted errors with `[wdio-intercept-service]`

consumers should be able to easily identify the source of an issue when they are running tests
Ideally these logs would be passed to the node / test context, rather than emitted in the tested browser, but depending on the test runner they may be surfaced without much additional work

* test(getRequest): add test for includePending: false (default case)

This is the existing behavior as of 4.1.10 and all earlier versions

* feat(getRequest): wire up the `includePending` request option

simplify assignment to `request` in node context

* test(request-order): add tests for the order of retrieved requests

The behavior in 4.1.10 and earlier versions is that requests are ordered by time of fulfillment (`orderBy: 'END'`). This should remain the default behavior, but users should be able to order by the time the request started as well (which is possibly more under their control, in terms of testing, too).
When pending requests are requested, this default sort will place them last (as they are not yet "completed").

* feat(request-order): wire up `orderBy` parameter in interceptor lib

* refactor: Avoid fetching full request list from browser to check if any request is pending

* feat(assert-api): allow passing `orderBy` options to `assertRequests` methods

 - deprecate boolean param to assertExpectedRequestsOnly
   - pass `inOrder` as an option instead
 - don't suggest asserting on pending requests will work
   - Since a pending request has no response, it has no response status code. (Passing `includePending: true` would result in a TypeError in the library code).
   - A better approach will be to support an option that requires pending requests are completed before performing assertions.

* test(firefox): bump slow delay to 1000 ms to account for slower FF driver

* docs: update README with latest API changes

* docs: update comment, formatting

* refactor: add package prefix constant for logs, errors

* docs: Update README to better describe configuration options for several methods

* docs: update changelog with release summaries

.gitignore

@@ -1,3 +1,4 @@
 errorShots
 node_modules
-test/logs
+test/**/logs
+test/**/dist

CHANGELOG.md

@@ -1,5 +1,26 @@
 ## wdio-intercept-service changelog
 
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v.next) ] v.next / <DATE>
+- Intercept HTTP requests upon initiation, rather than completion (thanks @tehhowch)
+
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v4.1.10) ] 4.1.10 / 16.11.2021
+* Support fetch requests opened with `URL` objects (thanks @tehhowch)
+* Fix return type for `browser.getExpectations()` (thanks @tehhowch)
+
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v4.1.9) ] 4.1.9 / 03.11.2021
+* Run e2e tests in async mode (thanks @tehhowch)
+* Support 'blob' response types in XHR requests (thanks @tehhowch)
+* Run e2e tests for Firefox, too (thanks @tehhowch)
+
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v4.1.8) ] 4.1.8 / 28.10.2021
+* Maintenance upgrade to help enforce IE compatability (thanks @tehhowch)
+
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v4.1.7) ] 4.1.7 / 04.08.2021
+* Add support for WebdriverIO standalone mode (thanks @juenobueno)
+
+### [ [>](https://github.com/webdriverio-community/wdio-intercept-service/tree/v4.1.6) ] 4.1.6 / 19.05.2021
+* Maintenance upgrades (thanks @christian-bromann)
+
 ### [ [>](https://github.com/chmanie/wdio-intercept-service/tree/v4.1.4) ] 4.1.4 / 19.04.2021
 Improved support for parallelization (thanks @RaulGDMM)
 

README.md

@@ -13,7 +13,7 @@ There's one catch though: you can't intercept HTTP calls that are initiated on p
 
 ## Prerequisites
 
-* webdriver.io **v5.x**.
+* webdriver.io **v5.x** or newer.
 
 **Heads up! If you're still using webdriver.io v4, please use the v2.x branch of this plugin!**
 
@@ -112,6 +112,15 @@ It should work with somewhat newer versions of all browsers. Please report an is
 
 ## API
 
+Consult the TypeScript declaration file for the the full syntax of the custom commands added to the WebdriverIO browser object. In general, any method that takes an "options" object as a parameter can be called without that parameter to obtain the default behavior. These "optional options" objects are followed by `?: = {}` and the default values inferred are described for each method.
+
+### Option Descriptions
+
+This library offers a small amount of configuration when issuing commands. Configuration options that are used by multiple methods are described here (see each method definition to determine specific support).
+
+* `orderBy` (`'START' | 'END'`): This option controls the ordering of requests captured by the interceptor, when returned to your test. For backwards compatibility with existing versions of this library, the default ordering is `'END'`, which corresponds to when the request was completed. If you set the `orderBy` option to `'START'`, then the requests will be ordered according to the time that they were started.
+* `includePending` (`boolean`): This option controls whether not-yet-completed requests will be returned. For backwards compatibility with existing versions of this library, the default value is `false`, and only completed requests will be returned.
+
 ### browser.setupInterceptor()
 
 Captures ajax calls in the browser. You always have to call the setup function in order to assess requests later.
@@ -132,33 +141,39 @@ Helper method. Returns all the expectations you've made up until that point
 
 Helper method. Resets all the expectations you've made up until that point
 
-### browser.assertRequests()
+### browser.assertRequests({ orderBy?: 'START' | 'END' }?: = {})
 
 Call this method when all expected ajax requests are finished. It compares the expectations to the actual requests made and asserts the following:
 
 - Count of the requests that were made
 - The order of the requests
 - The method, the URL and the statusCode should match for every request made
+- The options object defaults to `{ orderBy: 'END' }`, i.e. when the requests were completed, to be consistent with the behavior of v4.1.10 and earlier. When the `orderBy` option is set to `'START'`, the requests will be ordered by when they were initiated by the page.
 
-### browser.assertExpectedRequestsOnly(inOrder?: boolean)
+### browser.assertExpectedRequestsOnly({ inOrder?: boolean, orderBy?: 'START' | 'END' }?: = {})
 
-Similar to `browser.assertRequests`, but validates only the requests you specify in your `expectRequest` directives, without having to map out all the network requests that might happen around that. If `inOrder` equals to `true` (default), the requests are expected to be made in the same order as they were setup with `expectRequest`.
+Similar to `browser.assertRequests`, but validates only the requests you specify in your `expectRequest` directives, without having to map out all the network requests that might happen around that. If `inOrder` option is `true` (default), the requests are expected to be found in the same order as they were setup with `expectRequest`.
 
-### browser.getRequest(index: number)
+### browser.getRequest(index: number, { includePending?: boolean, orderBy?: 'START' | 'END' }?: = {})
 
-To make more sophisticated assertions about a specific request you can get details for a specific request after it is finished. You have to provide the index of the request you want to access in the order the requests were initiated (starting with 0).
+To make more sophisticated assertions about a specific request you can get details for a specific request. You have to provide the 0-based index of the request you want to access, in the order the requests were completed (default), or initiated (by passing the `orderBy: 'START'` option).
 
-* `index` (`Number`): number of the request you want to access
+* `index` (`number`): number of the request you want to access
+* `options` (`object`): Configuration options
+* `options.includePending` (`boolean`): Whether not-yet-completed requests should be returned. By default, this is false, to match the behavior of the library in v4.1.10 and earlier.
+* `options.orderBy` (`'START' | 'END'`): How the requests should be ordered. By default, this is `'END'`, to match the behavior of the library in v4.1.10 and earlier. If `'START'`, the requests will be ordered by the time of initiation, rather than the time of request completion. (Since a pending request has not yet completed, when ordering by `'END'` all pending requests will come after all completed requests.)
 
 **Returns** `request` object:
 
 * `request.url`: requested URL
 * `request.method`: used HTTP method
 * `request.body`: payload/body data used in request
 * `request.headers`: request http headers as JS object
-* `request.response.headers`: response http headers as JS object
-* `request.response.body`: response body (will be parsed as JSON if possible)
-* `request.response.statusCode`: response status code
+* `request.pending`: boolean flag for whether this request is complete (i.e. has a `response` property), or in-flight.
+* `request.response`: a JS object that is only present if the request is completed (i.e. `request.pending === false`), containing data about the response.
+* `request.response?.headers`: response http headers as JS object
+* `request.response?.body`: response body (will be parsed as JSON if possible)
+* `request.response?.statusCode`: response status code
 
 **A note on `request.body`:** wdio-intercept-service will try to parse the request body as follows:
 
@@ -170,12 +185,18 @@ To make more sophisticated assertions about a specific request you can get detai
 
 **For the `fetch` API, we only support string and JSON data!**
 
-### browser.getRequests()
+### browser.getRequests({ includePending?: boolean, orderBy?: 'START' | 'END' }?: = {})
 
-Get all captured requests as an array.
+Get all captured requests as an array, supporting the same optional options as `getRequest`.
 
 **Returns** array of `request` objects.
 
+### browser.hasPendingRequests()
+
+A utility method that checks whether any HTTP requests are still pending. Can be used by tests to ensure all requests have completed within a reasonable amount of time, or to verify that a call to `getRequests()` or `assertRequests()` will include all of the desired HTTP requests.
+
+**Returns** boolean
+
 ## TypeScript support
 
 This plugin provides its own TS types. Just point your tsconfig to the type extensions like mentioned [here](https://webdriver.io/docs/typescript.html#framework-types):
@@ -189,7 +210,7 @@ This plugin provides its own TS types. Just point your tsconfig to the type exte
 
 ## Running the tests
 
-A recent version of Chrome is required to run the tests locally. You may need to update the `chromedriver` dependency to match the version installed on your system.
+Recent versions of Chrome and Firefox are required to run the tests locally. You may need to update the `chromedriver` and `geckodriver` dependencies to match the version installed on your system.
 
 ```shell
 npm test

index.js

@@ -1,10 +1,26 @@
 'use strict';
 
 const interceptor = require('./lib/interceptor');
+const PKG_PREFIX = '[wdio-intercept-service]: ';
+class InterceptServiceError extends Error {
+  constructor(message, ...args) {
+    super(PKG_PREFIX + message, ...args);
+  }
+}
+
+const issueDeprecation = (map, key, what) => {
+  if (!map[key]) {
+    console.warn(
+      `${PKG_PREFIX}${what} is deprecated and will no longer work in v5`
+    );
+    map[key] = true;
+  }
+};
 
 class WebdriverAjax {
   constructor() {
     this._wdajaxExpectations = null;
+    this._deprecations = {};
   }
 
   beforeTest() {
@@ -34,8 +50,9 @@ class WebdriverAjax {
       'assertExpectedRequestsOnly',
       assertExpectedRequestsOnly.bind(this)
     );
+    browser.addCommand('hasPendingRequests', hasPendingRequests);
     browser.addCommand('getRequest', getRequest);
-    browser.addCommand('getRequests', getRequest);
+    browser.addCommand('getRequests', getRequests);
 
     function setup() {
       return browser.executeAsync(interceptor.setup);
@@ -50,15 +67,22 @@ class WebdriverAjax {
       return browser;
     }
 
-    function assertRequests() {
+    function assertRequests(options = {}) {
       const expectations = this._wdajaxExpectations;
 
       if (!expectations.length) {
         return Promise.reject(
           new Error('No expectations found. Call .expectRequest() first')
         );
       }
-      return getRequest().then((requests) => {
+
+      // Don't let users request pending requests:
+      if (options.includePending) {
+        throw new InterceptServiceError(
+          'passing `includePending` option to `assertRequests` is not supported!'
+        );
+      }
+      return getRequests(options).then((requests) => {
         if (expectations.length !== requests.length) {
           return Promise.reject(
             new Error(
@@ -135,13 +159,33 @@ class WebdriverAjax {
       });
     }
 
-    function assertExpectedRequestsOnly(inOrder = true) {
+    function assertExpectedRequestsOnly(orderOrOptions) {
       const expectations = this._wdajaxExpectations;
+      let inOrder = true;
+      let options = {};
+      if (typeof orderOrOptions === 'boolean') {
+        issueDeprecation(
+          this._deprecations,
+          'inOrder',
+          'Calling `assertExpectedRequestsOnly` with a boolean parameter'
+        );
+        inOrder = orderOrOptions;
+      } else if (orderOrOptions && typeof orderOrOptions === 'object') {
+        options = orderOrOptions;
+        inOrder = 'inOrder' in orderOrOptions ? orderOrOptions.inOrder : true;
+        delete options.inOrder;
+      }
 
-      return getRequest().then((requests) => {
-        const clonedRequests = [...requests];
+      // Don't let users request pending requests:
+      if (options.includePending) {
+        throw new InterceptServiceError(
+          'passing `includePending` option to `assertExpectedRequestsOnly` is not supported!'
+        );
+      }
+      return getRequests(options).then((requests) => {
+        const clonedRequests = requests.slice();
 
-        let matchedRequestIndexes = [];
+        const matchedRequestIndexes = [];
         for (let i = 0; i < expectations.length; i++) {
           const ex = expectations[i];
 
@@ -192,7 +236,7 @@ class WebdriverAjax {
         } else if (
           inOrder &&
           JSON.stringify(matchedRequestIndexes) !==
-            JSON.stringify(matchedRequestIndexes.concat().sort())
+            JSON.stringify(matchedRequestIndexes.slice().sort())
         ) {
           return Promise.reject(
             new Error('Requests not received in the expected order')
@@ -214,13 +258,16 @@ class WebdriverAjax {
       return this._wdajaxExpectations;
     }
 
-    async function getRequest(index) {
-      let request;
-      if (index > -1) {
-        request = await browser.execute(interceptor.getRequest, index);
-      } else {
-        request = await browser.execute(interceptor.getRequest);
-      }
+    function getRequests(options = {}) {
+      return getRequest(undefined, options);
+    }
+
+    async function getRequest(index, options = {}) {
+      const request = await browser.execute(
+        interceptor.getRequest,
+        index > -1 ? index : undefined,
+        options
+      );
       if (!request) {
         if (index != null) {
           return Promise.reject(
@@ -239,22 +286,35 @@ class WebdriverAjax {
       return transformRequest(request);
     }
 
+    function hasPendingRequests() {
+      return browser.execute(interceptor.hasPending);
+    }
+
     function transformRequest(req) {
       if (!req) {
         return;
       }
 
-      return {
+      const transformed = {
         url: req.url,
         method: req.method && req.method.toUpperCase(),
-        body: parseBody(req.requestBody),
         headers: normalizeRequestHeaders(req.requestHeaders),
-        response: {
+        body: parseBody(req.requestBody),
+        pending: true,
+      };
+      // Check for a '__fulfilled' property on the retrieved request, which is
+      // set by the interceptor only when the response completes. Before this
+      // flag is set, the request is still being processed (e.g. a large response
+      // body is downloading) and therefore is pending.
+      if (req.__fulfilled) {
+        transformed.pending = false;
+        transformed.response = {
           headers: parseResponseHeaders(req.headers),
           body: parseBody(req.body),
           statusCode: req.statusCode,
-        },
-      };
+        };
+      }
+      return transformed;
     }
 
     function normalizeRequestHeaders(headers) {