Add PVI integration #132

GDYendell · 2023-11-08T14:52:43Z

Closes #119

gilesknap

no issues with the code changes!
now trying it out ...

codecov-commenter · 2023-11-08T16:02:11Z

Codecov Report

Merging #132 (c12b33f) into main (b21cfa9) will increase coverage by 3.15%.
Report is 5 commits behind head on main.
The diff coverage is 98.21%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##             main     #132      +/-   ##
==========================================
+ Coverage   68.15%   71.31%   +3.15%     
==========================================
  Files          16       16              
  Lines         804      868      +64     
==========================================
+ Hits          548      619      +71     
+ Misses        256      249       -7

Files	Coverage Δ
src/ibek/__main__.py	`100.00% <100.00%> (ø)`
src/ibek/gen_scripts.py	`100.00% <100.00%> (ø)`
src/ibek/globals.py	`97.05% <100.00%> (+0.76%)`	⬆️
src/ibek/ioc_cmds/assets.py	`18.18% <100.00%> (ø)`
src/ibek/render_db.py	`100.00% <100.00%> (ø)`
src/ibek/runtime_cmds/commands.py	`100.00% <100.00%> (ø)`
src/ibek/support.py	`98.79% <100.00%> (-0.06%)`	⬇️
src/ibek/support_cmds/commands.py	`53.76% <100.00%> (+7.37%)`	⬆️
src/ibek/support_cmds/files.py	`60.00% <100.00%> (+10.00%)`	⬆️
src/ibek/ioc_cmds/commands.py	`50.72% <71.42%> (+2.33%)`	⬆️

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

gilesknap · 2023-11-08T18:36:49Z

ouch!

If only I had done the pydantic conversion!

Do I need an Asyn definition? The top of my asynNDArrayDriver.pvi.device.yaml looks like this
I also recall you Capitalising th asyn in this file before....

label: asynNDArrayDriver
parent: asynPortDriver
children:
  - type: Group
    name: ADSetup
    layout:

+ defs='/epics/ibek-defs/*.ibek.support.yaml'
+ ibek runtime generate /epics/ioc-adaravis/ioc/config/ioc.yaml /epics/ibek-defs/ADAravis.ibek.support.yaml /epics/ibek-defs/ADCore.ibek.support.yaml /epics/ibek-defs/asyn.ibek.support.yaml /epics/ibek-defs/autosave.ibek.support.yaml /epics/ibek-defs/epics.ibek.support.yaml /epics/ibek-defs/iocStats.ibek.support.yaml --out /tmp/st.cmd --db-out /tmp/ioc.subst
Traceback (most recent call last):

  File "/venv/bin/ibek", line 8, in <module>
    sys.exit(cli())

  File "/repos/ibek/src/ibek/runtime_cmds/commands.py", line 49, in generate
    pvi_index_entries, pvi_databases = generate_pvi(ioc_instance)

  File "/repos/ibek/src/ibek/runtime_cmds/commands.py", line 94, in generate_pvi
    device.deserialize_parents([PVI_DEFS])

  File "/repos/pvi/src/pvi/device.py", line 396, in deserialize_parents
    parent_parameters = find_components(self.parent, yaml_paths)

  File "/repos/pvi/src/pvi/device.py", line 437, in find_components
    list(find_components(device.parent, yaml_paths))

  File "/repos/pvi/src/pvi/device.py", line 437, in find_components
    list(find_components(device.parent, yaml_paths))

  File "/repos/pvi/src/pvi/device.py", line 434, in find_components
    device = Device.deserialize(device_yaml)

  File "/repos/pvi/src/pvi/device.py", line 389, in deserialize
    return deserialize(cls, YAML(typ="safe").load(serialized))

  File "/venv/lib/python3.10/site-packages/apischema/deserialization/__init__.py", line 887, in deserialize
    return deserialization_method(

  File "apischema/deserialization/methods.pyx", line 504, in apischema.deserialization.methods.ObjectMethod.deserialize

  File "apischema/deserialization/methods.pyx", line 505, in apischema.deserialization.methods.ObjectMethod.deserialize

  File "apischema/deserialization/methods.pyx", line 1077, in apischema.deserialization.methods.ObjectMethod_deserialize

apischema.validation.errors.ValidationError: ValidationError: [{'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'children'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'layout'], 'err': 'unexpected property'}, {'loc': ['children', 6, 'macros'], 'err': 'missing property'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}, {'loc': ['children', 6, 'pv'], 'err': 'missing property'}, {'loc': ['children', 6, 'pv'], 'err': 'missing property'}, {'loc': ['children', 6, 'pv'], 'err': 'missing property'}, {'loc': ['children', 6, 'pv'], 'err': 'missing property'}, {'loc': ['children', 6, 'pv'], 'err': 'missing property'}, {'loc': ['children', 6, 'type'], 'err': "not one of ['SignalR'] (oneOf)"}, {'loc': ['children', 6, 'type'], 'err': "not one of ['SignalW'] (oneOf)"}, {'loc': ['children', 6, 'type'], 'err': "not one of ['SignalRW'] (oneOf)"}, {'loc': ['children', 6, 'type'], 'err': "not one of ['SignalX'] (oneOf)"}, {'loc': ['children', 6, 'type'], 'err': "not one of ['DeviceRef'] (oneOf)"}, {'loc': ['children', 6, 'type'], 'err': "not one of ['SignalRef'] (oneOf)"}, {'loc': ['children', 6, 'ui'], 'err': 'missing property'}]

gilesknap · 2023-11-08T18:37:57Z

One more question - do we have a schema for device.yaml. If so I can use vscode to debug the yaml itself.

UPDATE: I found the schema and all device YAMLS are passing.

gilesknap · 2023-11-08T18:38:14Z

ooops

GDYendell · 2023-11-08T19:13:35Z

If there is no schema error, it is probably the name case thing. Check label and name fields and make sure they start with a capital.

This may be the bit out of that mess that is useful:

{'loc': ['children', 6, 'name'], 'err': 'not matching pattern ([A-Z][a-z0-9]*)*$ (pattern)'}

So the name of Group 6 (not sure if 0-indexed) needs a capital?

gilesknap · 2023-11-08T19:40:29Z

OK that did it - no more schema errors - but I needed to change quite a few places. I assume we can fix the generator to not make this mistake (and maybe include the restriction in the schema).

I'm now getting

OSError: Cannot find AsynPortDriver.pvi.device.yaml in [PosixPath('/epics/pvi-defs')]

which is correct - will try and generate a AsynPortDriver

GDYendell · 2023-11-08T20:11:52Z

AsynPortDriver.pvi.device.yaml

This should start with a lower case to match the driver name. I generated the pvi yamls here.

gilesknap · 2023-11-08T20:33:48Z

Thanks - but that is the AsynPortDriver itself from Asyn I assume.

Your link does not have one of these and I can't convince PVI to build one. (not sure which DB I should use but its the .h file it doesn't like I think.

(venv) [hgv27681@pc0116 ~]$ pvi convert device /tmp /dls_sw/prod/R3.14.12.7/support/asyn/4-41/asyn/asynPortDriver/asynPortDriver.h --template /dls_sw/prod/R3.14.12.7/support/asyn/4-41/asyn/asynRecord/asynRecord.db Traceback (most recent call last):

  File "/scratch/hgv27681/work/venv/bin/pvi", line 8, in <module>
    sys.exit(app())
             ^^^^^

  File "/scratch/hgv27681/work/venv/lib/python3.11/site-packages/pvi/__main__.py", line 104, in device
    name, parent = extract_device_and_parent_class(h.read_text())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/scratch/hgv27681/work/venv/lib/python3.11/site-packages/pvi/_convert/utils.py", line 10, in extract_device_and_parent_class
    assert match, "Can't find classes"

AssertionError: Can't find classes

gilesknap · 2023-11-08T20:36:29Z

We have lift-off!!

I removed the ref to AsynPortDriver that I mentioned above #132 (comment)

It now works. and I get

These files were renamed in `5a1ee025 fix naming of subst files`

Make generate_links take a specific support module to link rather than doing glob of all directories.

Add PVI_DEFS to list of assets to be exported for runtime stage Change use of Dict to Mapping to make mypy happy; ``` src/ibek/runtime_cmds/commands.py:158: error: Argument "args" to "Database" has incompatible type "dict[str, str]"; expected "dict[str, str | None]" [arg-type] src/ibek/runtime_cmds/commands.py:158: note: "Dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance src/ibek/runtime_cmds/commands.py:158: note: Consider using "Mapping" instead, which is covariant in the value type ```

This PR has highlighted an issue that CI needs to know where to find the ibek support YAML - but ibek already knows Added --ibek-defs for use in CI to generate schema without breaking each time we move the /epics/ibek-defs folder. This is the default. The regenerate_samples.sh script can use --no-ibek-defs to only include the given support yaml definitions.

GDYendell · 2023-11-10T18:33:28Z

@gilesknap I added the last tweaks, visible in the compare changes here.

I know you said it should avoid using if definitions because it is bad practice if it is None, but given the typer weirdness it really is a list, so I think this is the most correct check. I also added a comment in the code about this for clarity.

I will leave you to press the merge button in case there are any issues with this.

gilesknap · 2023-11-11T17:48:33Z

Yep I'm happy with the if definitions thing.

GDYendell requested a review from gilesknap November 8, 2023 14:52

gilesknap reviewed Nov 8, 2023

View reviewed changes

gilesknap closed this Nov 8, 2023

gilesknap reopened this Nov 8, 2023

gilesknap force-pushed the pvi branch from d7bf253 to 16ce851 Compare November 9, 2023 19:53

GDYendell force-pushed the pvi branch 2 times, most recently from 5abc43e to 417f130 Compare November 10, 2023 15:18

gilesknap added 2 commits November 10, 2023 15:24

Remove redundant output files from tests

44950c8

These files were renamed in `5a1ee025 fix naming of subst files`

fix pre-commit

2766c62

GDYendell force-pushed the pvi branch from 417f130 to 511b71b Compare November 10, 2023 15:24

gilesknap and others added 6 commits November 10, 2023 15:28

move all runtime assets to /epics/runtime

ca58618

Update support generate-links to symlink ibek and pvi yaml

eab3d82

Make generate_links take a specific support module to link rather than doing glob of all directories.

Use absolute imports to enable debugging module

5f27215

Bump minimum python version to 3.10

663d5cf

include _globals in ibek support generate-links

5480797

GDYendell force-pushed the pvi branch from 511b71b to c12b33f Compare November 10, 2023 16:04

GDYendell force-pushed the pvi branch from c12b33f to e307124 Compare November 10, 2023 18:27

gilesknap merged commit 1de3a19 into main Nov 11, 2023
12 checks passed

gilesknap deleted the pvi branch November 11, 2023 17:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PVI integration #132

Add PVI integration #132

GDYendell commented Nov 8, 2023 •

edited

Loading

gilesknap left a comment

codecov-commenter commented Nov 8, 2023 •

edited

Loading

gilesknap commented Nov 8, 2023

gilesknap commented Nov 8, 2023 •

edited

Loading

gilesknap commented Nov 8, 2023

GDYendell commented Nov 8, 2023 •

edited

Loading

gilesknap commented Nov 8, 2023

GDYendell commented Nov 8, 2023

gilesknap commented Nov 8, 2023

gilesknap commented Nov 8, 2023

GDYendell commented Nov 10, 2023

gilesknap commented Nov 11, 2023

Add PVI integration #132

Add PVI integration #132

Conversation

GDYendell commented Nov 8, 2023 • edited Loading

gilesknap left a comment

Choose a reason for hiding this comment

codecov-commenter commented Nov 8, 2023 • edited Loading

Codecov Report

gilesknap commented Nov 8, 2023

gilesknap commented Nov 8, 2023 • edited Loading

gilesknap commented Nov 8, 2023

GDYendell commented Nov 8, 2023 • edited Loading

gilesknap commented Nov 8, 2023

GDYendell commented Nov 8, 2023

gilesknap commented Nov 8, 2023

gilesknap commented Nov 8, 2023

GDYendell commented Nov 10, 2023

gilesknap commented Nov 11, 2023

GDYendell commented Nov 8, 2023 •

edited

Loading

codecov-commenter commented Nov 8, 2023 •

edited

Loading

gilesknap commented Nov 8, 2023 •

edited

Loading

GDYendell commented Nov 8, 2023 •

edited

Loading