[RFC] Form Versioning

[winged]

There is a requirement to have support for versioning forms, and what comes with them. Some ideas are floating around, but to get this thing going, we need to follow a structured approach.

Note: This issue is an RFC, discussions are welcome, and no concrete, actionable specification is done yet.

Requirements / Starting point

Hard requirements

The following are things that are not negotiable. Things that absolutely must work.

• Existing documents and workflows must not be broken in any way by updating forms / questions / etc: new cases can use newest versions, but old documents remain on the old versions
• Allow analysis / queries across multiple cases, using different versions as long as relevant questions remain the same
• Allow multiple edits in a "staging" area before a new version is published: While new version is developed, the existing "production" version must not be affected, even for newly started workflows/cases
• Make it easy to migrate: Some installations already use some sort of versioning scheme that works similarly. A migration path must be available

Soft requirements

These should be optimized for, but if for some reason something needs to be worked around or cannot be done, it can be considered acceptable. Ordered by priority/importance.

• P1: avoid too many copies of the same form structure elements (questions, forms, options, ...)
• P1: make it easy for users to understand (UI/UX):
  • when is a new version created
  • why a new version is required
• P1: preview the form before publishing: Ideally, an unpublished form can be tested with a temporary document structure, allowing to check whether is_hidden, is_required JEXL expressions work as expected, as well as validations etc.
• P2: existing versioning schemes to be migrated into the new official versioning structure. This is optional, as it may not be possible at all, but if possible, we should try to do it
• P3: small corrections (typos) should be possible without forcing a new version.
  • acceptable: changing labels / info texts, help texts, including translations
  • not acceptable: changing options, JEXL(?), sub questions
• P4: allow migrating documents to newer versions - exact conditions to be defined
• P4: Cause only minimal breakage to existing applications

Questions

• do we even expose (numerical) "versions" to the users?
• what about new work items in old cases - should they use old versions of their forms instead?
  possible problem: two work items in a circulation of the same case have different forms (is that even
  a problem tho?)

Existing implementations / Workarounds

One installation implements a versioning scheme where a form's version is determined by a hash of the question's slugs within. The hash is appended to the form and corresponding question (hierarchical form), which again affects the slug hash of it's parent form in turn.

The side effect of this is that it is possible to modify questions in-place without triggering a new version, and new versions can be done by creating a new question with a versioned suffix (question-foo-v2 for example). The form structure in this installation is built up automatically from an Excel sheet and some other data sources. In it's current form, it would not be ergonomical to use the form builder for modifications.

Other installations use a more manual versioning approach: Form structures are managed in JSON dumps that are loaded on deploy; structures such as Tasks are also managed in JSON. When a new form version is released, the old forms remain in place, but the new forms are referenced in the tasks and will then be picked up on the next work item or case.

Both of these implementations work, but have downsides. The second variant will potentially still be around, as these forms are not manually configured on the "live" system, but transported via export/import.

Proposal

The following proposal is a rough outline. Not all the details are fleshed out yet, and the impact of these changes needs to be evaluated as well.

Model changes

• Turn the primary key for forms and questions into an UUID field. The Slug will remain, but will only remain unique within one form structure, not globally.
• Add a version field that is counting up whenever a new version is created

Processing changes

• Tasks / Workflows still refer to form slugs. However, when instantiated (new case with document, or new workitem with document), the newest published version of a form is used
• Documents refer to the UUID of the form, and therefore always to a specific version.

Form builder changes

• Changes to versioned forms and questions is limited to small corrections only. Which of these are acceptable is to be defined

• Bigger changes (like changing options, or adding, or removing questions to a form) are only possible on "draft" versions (where the version tag is NULL)
  

• On "publish", a copy of the form is done, recursively, such that any unreleased version is released with it (The exact procedure is still to be defined)

• A navigation point is added where old versions of a question can be viewed, or potentially small issues fixed with it

[winged]

Let us, as a first step, get an agreement on the requirements: Are there any other "hard" requirements? And do we agree on the "soft" ones as well?

• Discuss requirements and prioritization
• Draft technical specification (data model, workflows, dependencies etc)
• Draft UX concept for form builder changes