Migration Strategy for DATEX values

[benStre]

Migration of structured values

Caveats/Limitations

• Only works for values with a type template definition that can be stored in tables when using SQL storage
• For now: only works with types defined with struct
• The struct must have a unique human readable type name the cannot be changed. This allows the runtime the to match multiple versions (template hashes) of a type
• migrations are only supported when using sql storage (sqlite/mysql)

Base migration process (always performed on app startup)

1. When the runtime starts and a type definition is loaded, the previous hash for the type is compared with the current hash
2. If the hashes are different, the new property columns are added to the table
3. deprecated columns are deleted

Process when a custom migration exists

Custom migrations are located in the migrations directory.
Each migration contains a default export with one or multiple migration definitions.
Additionally, a auto-generated migrations.dx file in the migrations directory keeps a log of all type migrations.

• Step 1 and 2 of the base migration are perfomed
• If a hash mismatch for a type is detected, the migrations.dx file is first checked to see if a migration path to the current hash is already registered. If this is the case, all migrations starting from the previous hash (the state of the pointers that are currently in storage) are performed in order.
• If there is no available migration in the migrations.dx file, the migrations directory is checked for any migration ts files that are not yet logged in the migrations.dx file. If one or multiple migration files exist, they are executed for the given type and the files with the corresponding hashes are added to the migrations.dx file.
• After the custom migrations are finished, Step 3 of the base migration is performed

Migration definitions

// migrations/user-with-names.ts

export default migrate(User)
	.map((prevUser, newUser) => {
		newUser.uuid = generateUUID()
		newUser.firstName = prevUser.name.split(' ')[0]   
		newUser.lastName = prevUser.name.split(' ')[1]
	})

// migrations/migrations.dx

(
  'struct:User': (
        (
           migration: ./user-with-names.ts,
           from:      '123456',
           to:        'abcedf'
        ),
        
        (
           migration: ./user-new-attrs.ts,
           from:     'abcedf',
           to:       'eeefffee'
        )
  ),
  ...
)