Use WordPress Playground

[swissspidy]

Fixes #71
Fixes #81

This replaces wp-env with WordPress Playground, using a blueprint to configure the test environment, making it easy to replicate the same environment locally as well.

Plus, the user can provide their own blueprint, which will be merged with the base one.

In a next step, I am considering using blueprints for the test scenarios themselves as well. This way, we can do more than just simply visiting URLs & measuring Server-Timing. For example, we could declaratively tell the action to fill out forms so things like INP can be measured. See #174

Example

Workflow

    - name: Run performance tests
      uses: ./
      with:
        urls: |
          /
          /sample-page/
        # Simply mounts the plugin to `wp-content/plugins/dummy-plugin`
        plugins: |
          ./path/to/my/dummy-plugin
        blueprint: './path/to/my/blueprint.json'
        iterations: 5
        repetitions: 1
        wp-version: 'trunk'
        php-version: '7.4'

User-provided blueprint

{
  "$schema": "https://playground.wordpress.net/blueprint-schema.json",
  "plugins": [
    "performant-translations",
    "akismet"
  ],
  "steps": [
    {
      "step": "defineWpConfigConsts",
      "consts": {
        "WP_DEBUG": true
      }
    },
    {
      "step": "activatePlugin",
      "pluginName": "Dummy Plugin",
      "pluginPath": "/wordpress/wp-content/plugins/dummy-plugin"
    }
  ]
}

---

Some observations

• TTFB is 5x higher, but starting the server is quicker.
• Playground is not always very stable. For example, I tried installing wordpress-seo in a blueprint and it crashed every time when trying to start the server.
  • Update: caused by the SQLite integration

To-do:

• Update documentation
• Improve how blueprints are merged, see Running multiple blueprints WordPress/wordpress-playground#1631 (comment)
• Add example with comparing with previous results

[swissspidy]

Performance Test Results

Performance test results for d4a36c2 are in 🛎️!

Tests

Run	wpDbQueries	wpMemoryUsage	wpBeforeTemplate	wpTemplate	wpTotal	largestContentfulPaint	timeToFirstByte	lcpMinusTtfb
0	43	27.30 MB	203.00 ms	88.50 ms	291.00 ms	408.50 ms	304.40 ms	103.40 ms
1	43	27.30 MB	207.50 ms	89.00 ms	295.00 ms	408.50 ms	308.90 ms	104.30 ms
2	43	27.30 MB	206.00 ms	89.50 ms	293.00 ms	407.95 ms	305.85 ms	103.80 ms
3	43	27.30 MB	204.50 ms	88.00 ms	291.50 ms	407.30 ms	305.25 ms	103.85 ms
4	43	27.30 MB	207.50 ms	89.00 ms	297.00 ms	419.45 ms	309.95 ms	106.45 ms

[swissspidy]

In swissspidy/compare-wp-performance#17 the results seemed less stable than when using wp-env. Not sure I can say the same here 🤔 Worth keeping an eye on.

[adamziel]

This is lovely! Are there any blockers for this work on the Playground side that we could help with?

Playground is not always very stable. For example, I tried installing wordpress-seo in a blueprint and it crashed every time when trying to start the server.

@bgrgicak's is working on reporting installation errors for each plugin in the WordPress plugins repository to help fix them and potentially communicate known issues within Playground. That plugin came up in the work. The error in the web version of Playground is CREATE UNIQUE INDEX wp_yoast_migrations_versionONwp_yoast_migrations(version), so the SQLite integration is at fault. I hope WordPress/sqlite-database-integration#157 will either solve or make it easy to solve many SQLite issues.

[swissspidy]

This is lovely! Are there any blockers for this work on the Playground side that we could help with?

No blockers per se right now I think.

We just have to test this more to determine whether this is actually a good testing environment or not. For example, over at swissspidy/compare-wp-performance#17, some of the gathered metrics didn't seem very stable with Playground.

@bgrgicak's is working on reporting installation errors for each plugin in the WordPress plugins repository to help fix them and potentially communicate known issues within Playground.

That sounds like something that could easily run on Tide :-)

so the SQLite integration is at fault

Ah, good to know, thanks! 💡

[adamziel]

For example, over at swissspidy/compare-wp-performance#17, some of the gathered metrics didn't seem very stable with Playground.

I hear you! I'd like to investigate but I'm not sure how to connect the tabular data to the conclusion the metrics aren't stable. Would you help me connect these dots?

[swissspidy]

For example, over at swissspidy/compare-wp-performance#17, some of the gathered metrics didn't seem very stable with Playground.

I hear you! I'd like to investigate but I'm not sure how to connect the tabular data to the conclusion the metrics aren't stable. Would you help me connect these dots?

Sure, so what that script in that repo does is measure things like the the TTFB difference between two WordPress versions. It does so over 100 iterations and then returns metrics such as the p50 value and standard deviation.

In swissspidy/compare-wp-performance#17 I did this using both wp-env and Playground.

I just re-ran those tests now, comparing WordPress nightly/trunk vs itself.

See wp-env result and Playground result.

I do have to say I like the results better now, but for values such as wp-total, wp-env sees barely a difference (p50 is 0.57 ms), whereas for Playground it is a bit higher (p50 is 3.50 ms).

[swissspidy]

The merging as per WordPress/wordpress-playground#1631 doesn't seem to work.

I'll probably keep the current approach until I can sort this out.