xAPI 2.0 Compatibility + Test Directory

[Argenrost]

This update adds xAPI 2.0 compatibility to the conformance test suite.

• Version support for 2.0.x has been added.
• 2.0 test directory has been added with new test sets for Context Agents and Context Groups, as well as several improvements and adjustments to account for LRS deltas between 1.0.3 and 2.0.

[milt]

@Argenrost @vbhayden You might want to look at the testing for XAPI-00321 (works in a grep), it looks like it still expects the old 1.0.x version for everything aside from the /statements resource, we get: AssertionError: expected '2.0.0' to match /^1\.0\.\d+$|^0?\.9\d*?$/ on /agents, /activities, etc.

[milt]

Some tests also now seem to expect an X-Experience-Api-Version header on 400 responses (not sure they did before), couldn't find discussion of that in the changelogs.

[vbhayden]

Thanks for the heads up Milt!

[milt]

The changes look good for setting the headers! I'm now running into an issue when I try to make an LRS that is backwards compatible with 1.0.x: It looks like there's a test that tries to ensure the alternate request syntax is not accepted, but does not itself send an X-Experience-Api-Version header for 2.0.0. Since the old alt request syntax requests encode that in the body, the LRS has no way to know it is a 2.0.0 request.

There was no XAPI-XXXX code for the fail but it did give me the loc+line: lrs-conformance-test-suite/test/v2_0/H.Communication1.3-AlternateRequestSyntax.js:108:14

[vbhayden]

Better late than never, hopefully fixed with 46d4ee0

Thanks Milt!

[milt]

Hi Trey!
Getting some weird nonsensical test output after the latest commits and was wondering if you had any idea what was going on. For example, here's an output where it appears to be failing a test for an ETAG requirement on documents but the only listed requests/responses pertain to the statements endpoint:

{
    "name": "console",
    "owner": null,
    "flags": {
        "endpoint": "http://localhost:8080/xapi"
    },
    "options": {},
    "rollupRule": "mustPassAll",
    "uuid": "4f291c65-e984-44d3-8bce-e2a7460bf264",
    "startTime": 1685984406843,
    "endTime": 1685984407328,
    "duration": 485,
    "state": "finished",
    "summary": {
        "total": 1408,
        "passed": 28,
        "failed": 1,
        "version": "2.0.0.0"
    },
    "log": {
        "title": "",
        "name": "",
        "requirement": "",
        "log": "REQUEST SUPERREQUEST\n_______________________________________\nPOST /xapi/statements HTTP/1.1\r\nX-Experience-API-Version: 2.0.0\r\nhost: localhost:8080\r\naccept: application/json\r\ncontent-type: application/json\r\ncontent-length: 324\r\nConnection: close\r\n\r\n{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI mbox\",\"mbox\":\"mailto:[email protected]\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/meetings/occurances/34534\"},\"id\":\"b1c28439-90c8-4f2d-8db2-7455c9225e38\"}\n\nRESPONSE SUPERREQUEST\n_______________________________________\nHTTP/1.1 200 OK\nconnection: close\ndate: Mon, 05 Jun 2023 17:00:06 GMT\nx-experience-api-consistent-through: 2023-06-05T17:00:06.947459000Z\nx-experience-api-version: 2.0.0\ncontent-type: application/json;charset=utf-8\n\nb1c28439-90c8-4f2d-8db2-7455c9225e38\n=======================================\nREQUEST SUPERREQUEST\n_______________________________________\nGET /xapi/statements?statementId=b1c28439-90c8-4f2d-8db2-7455c9225e38 HTTP/1.1\r\nX-Experience-API-Version: 2.0.0\r\nhost: localhost:8080\r\nConnection: close\r\n\r\nRESPONSE SUPERREQUEST\n_______________________________________\nHTTP/1.1 200 OK\nconnection: close\ndate: Mon, 05 Jun 2023 17:00:06 GMT\nx-experience-api-consistent-through: 2023-06-05T17:00:06.970985000Z\nx-experience-api-version: 2.0.0\ncontent-type: application/json;charset=utf-8\n\n{\"actor\":{\"objectType\":\"Agent\",\"name\":\"xAPI mbox\",\"mbox\":\"mailto:[email protected]\"},\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/attended\",\"display\":{\"en-GB\":\"attended\",\"en-US\":\"attended\"}},\"object\":{\"objectType\":\"Activity\",\"id\":\"http://www.example.com/meetings/occurances/34534\"},\"id\":\"b1c28439-90c8-4f2d-8db2-7455c9225e38\",\"stored\":\"2023-06-05T17:00:06.940355000Z\",\"timestamp\":\"2023-06-05T17:00:06.940355000Z\",\"authority\":{\"name\":\"Memory LRS\",\"objectType\":\"Agent\",\"account\":{\"name\":\"root\",\"homePage\":\"http://localhost:8080\"}},\"version\":\"2.0.0\"}\n=======================================\n",
        "status": "failed",
        "tests": [
            {
                "title": "(4.2.7) Concurrency",
                "name": "",
                "requirement": "4.2.7",
                "log": "",
                "status": "failed",
                "tests": [
                    {
                        "title": "xAPI uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data.",
                        "name": "xAPI uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data.",
                        "requirement": "",
                        "log": "",
                        "status": "failed",
                        "tests": [
                            {
                                "title": "Concurrency for the Activity State Resource.",
                                "name": "Concurrency for the Activity State Resource.",
                                "requirement": "",
                                "log": "",
                                "status": "failed",
                                "tests": [
                                    {
                                        "title": "If a PUT request is received without either header for a resource that already exists",
                                        "name": "If a PUT request is received without either header for a resource that already exists",
                                        "requirement": "",
                                        "log": "",
                                        "status": "failed",
                                        "tests": [
                                            {
                                                "title": "Return 409 conflict",
                                                "name": "Return 409 conflict",
                                                "requirement": "",
                                                "log": "",
                                                "status": "failed",
                                                "error": "AssertionError: expected 204 to equal 409",
                                                "tests": []
                                            }
                                        ]
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

I'm adding some extra instrumentation to my test LRS to see if I can figure out more, but let me know if this makes any sense to you.

[vbhayden]

Morning @milt, just now seeing this -- been a long summer.

There were issues with the ETag tests and timing due to the introduction of a few async calls being attributed to the wrong requirement. They should all be fine now, but let me know if they're still misbehaving for you.