Add three new ZMQ publishers for TP results / Retrieve TP results

[MichaelHuth]

Publishers added:
testpulse results live
testpulse results 1s update
testpulse results 5s update
testpulse results 10s update

The TP results from TP analysis are packed in a JSON in TP_TSAnalysis and published to ZMQ.
The time information when the next publishing is due is stored in the TUFXOP.

• PUB_Publish was changed to be threadsafe and added an optional argument to control if the JSON should be released.

close #2157

[MichaelHuth]

@t-b Feedback request for general design improvements.

[t-b]

General approach looks good.

Two comments:

• PUB_AddTPResultEntry: We only want a value object if it has a unit. See the other publishing functions for an example. The publishing docu can be generated with FetchAndParseMessage and OUTPUT_DOCUMENTATION_JSON_DUMP.

• PUB_CheckPublishingTime this should use TSD_ReadVar/TSD_WriteVar.

[t-b]

Git review:

ef3f1d2 (Util: Add two conversion function for DA/AD unit string return, 2024-08-22)

Good idea!

• s/in clampmode/on clampmode/

• Can we tweak them to also accept a channel typ parameter (XopChannelConstants) so that they also return the DA units and use these functions in GetChanAmpAssignUnit? Use it GetChanAmpAssignUnit. In that way we only have a single place where we define the units depending on the clamp mode.

27dac75 (TP: Add more information that is transferred to the TP analysis thread, 2024-08-22)

Nice!

6368bbb (PUB: Add four zeromq publishers for TP data, 2024-08-22)

• s/jst/just/ in the commit message.

• Can you move the ZMQ_FILTER_* constants close to AUTO_TP_FILTER? And their names need to be something like "testpulse:", this is done so that people can easily subscribe to all messages from a certain area. See ZMQ_SUBSCRIBE in https://libzmq.readthedocs.io/en/latest/zmq_setsockopt.html. Maybe that needs a comment though.

• PUB_Publish changes can be their own commit.

• The test TestTPPublishing fails here with:

•runwithOpts(testcase = "TestTPPublishing")
  Start of test "MIES with HardwareBasic"
  Entering test suite "UTF_TestPulseAndTPDuringDAQ.ipf"
  Entering test case "TestPulseAndTPDuringDAQ#TestTPPublishing:ITC"
  Entering reentry "TestPulseAndTPDuringDAQ#TestTPPublishing_REENTRY"
  String mismatch (case sensitive):
str1: 0:0:0> h e a r t b e a t
str2: 0:0:0> t e s t p u l s e   r
: is false. Assertion "CHECK_EQUAL_STR(filter, expectedFilter)" failed in TestPulseAndTPDuringDAQ#TestTPPublishing_REENTRY➔FetchPublishedMessage (UTF_HelperFunctions.ipf, line 1426➔228)
  Encountered "AbortOnValue" Code 10009 in test case "TestPulseAndTPDuringDAQ#TestTPPublishing" (UTF_TestPulseAndTPDuringDAQ.ipf)
  Leaving test case "TestPulseAndTPDuringDAQ#TestTPPublishing:ITC"
  Failed with 2 errors
  Leaving test suite "UTF_TestPulseAndTPDuringDAQ.ipf"
  Test finished with 2 errors
    ▶ Assertion "CHECK_EQUAL_STR(filter, expectedFilter)" failed in TestPulseAndTPDuringDAQ#TestTPPublishing_REENTRY➔FetchPublishedMessage (UTF_HelperFunctions.ipf, line 1426➔228)
    ▶ Encountered "AbortOnValue" Code 10009 in test case "TestPulseAndTPDuringDAQ#TestTPPublishing" (UTF_TestPulseAndTPDuringDAQ.ipf)
  End of test "MIES with HardwareBasic"

The problem is likely that FetchPublishedMessage only works with a single filter. So we might need to teach FetchPublishedMessage to check fo multiple filters and return then multiple messages.

• Can we check the exact numeric values of the published data (with some tolerance) instead of using IsInteger or != NaN? The data is in the labnotebook or TPStorage (and really only these two).

• Test in UTF_ZeroMQPublishing.ipf missing.

• This test is also required to fill in the documentation for PUB_TPResult like for the other PUB_XXX functions.

• Can we name the parameters in TP_PrepareAnalysisDF, so that the code in TP_TSAnalysis is easier to grasp? See NWB_ASYNC_SerializeStruct/NWB_ASYNC_DeserializeStruct.

• GetTPResultAsyncBuffer: numCols should only be calculated if required. The normal case it that the wave exists with the correct size.

• Can we NaN the wave in the upgrade case? Just to be sure that we don't have bogus data lying around?

• The wave upgrade cleanup in GetTPStorage is welcome, but should be its own commit.

• GetTPStorage: The number of layers is different in upgrade compared to creation.

• GetTPStorage: Are the label names in layer 31 to 39 from TP_ANALYSIS_DATA_LABELS? If yes we need a comment to say so.

9c33299 (TP: Added two functions that allow to retrieve info about TPs, 2024-08-22)

commit message:

• s/both function/both functions/

• s/meta information/metadata/

• Should be its own commit:

the former function TP_GetStoredTPs was renamed to TP_GetConsecutiveTPsUptoMarker
to prevent misleading naming.

• The code to recreate the DAC data should be its own function and not duplicated.

• TP_GetStoredTPsFromCycle/TP_GetStoredTP: I would expect the returned metadata (tpresult, or the waves in tpsResult) to be 1D.

• Why do both functions not require a headstage parameter? If I understand the code in SCOPE_UpdateOscilloscopeData correctly a tpmarker is constant for all headstages of a given TP. Maybe make the test use two headstages with different clamp modes to verify that.

[MichaelHuth]

• Change TP return functions to have an headstage argument and slice also HS.
  Then return tpADC, tpDAC, tpStorage slice as 1d wave.

• Add test that DimLabels of layers are correctly transferred to ROWs (if Redimension does that itself)

• Use outData wave directly for PUB_TP* argument, scrap tpzmq struct

• add wave getter for outData, use MoveWave in TP_TSAnalysis with named free wave

[t-b]

@MichaelHuth Please rebase

[t-b]

CI is failing.

Review:
92ad402 (Util: Add two conversion function for DA/AD unit string return, 2024-08-22)
5571774 (TP: Add more information that is transferred to the TP analysis thread, 2024-08-22)

Looks all good.

31d1771 (Getters: Cleanup of GetTPStorage wave upgrade, 2024-08-25)

Now looking at this change in isolation, I realize that this is not doing the right thing.

Consider a wave upgrade for version 13. The old code set layers 24 to 30. But
the new code sets layers 17 and 20 to 30. Overwriting valuable data!

2802321 (PUB: Preparation to add four zeromq publishers for TP data, 2024-08-25)
00941de (PUB: Add four publishers to publish TP data, 2024-08-23)

TestTPPublishing_REENTRY:

Instead of

Make/FREE/T filters = {ZMQ_FILTER_TPRESULT_NOW, ZMQ_FILTER_TPRESULT_1S, ZMQ_FILTER_TPRESULT_5S, ZMQ_FILTER_TPRESULT_10S}

you can just write

WAVE filters = DataGenerators#PUB_TPFilters()

0057895 (TP: Rename TP_GetStoredTPs to TP_GetConsecutiveTPsUptoMarker, 2024-08-25)

We nowadays have IsValidHeadstage.

e86dda8 (TP: Added two functions that allow to retrieve info about TPs, 2024-08-22)

Looks good!

[MichaelHuth]

@t-b I diagnosed an issue with the JSON construction for publishing in this PR that requires an update of the JSON XOP to the current version where we fixed the handling of adding values that are large integers.
The current effect is that for certain calculated values from the TP analysis the JSON XOP returns an error. That can happen randomly (here about 1 in 100), depending on the noise.

[t-b]

@t-b I diagnosed an issue with the JSON construction for publishing in this PR that requires an update of the JSON XOP to the current version where we fixed the handling of adding values that are large integers.

Let's update the JSON XOP then, 7a39926 (Update JSON-XOP, 2023-09-25) shows how this is done. We probably would like to update the MacOSX XOP as well.

[t-b]

We return a wave reference wave from the get tp function and I was curious if we are handling recursive wave references correctly, yes we do.

Function Dostuff()

	make/o/WAVE/N=1 data
	make/o/WAVE/N=1 elem1
	make/o/WAVE/N=1 elem2

	data[0] = elem1
	elem1[0] = elem2
	
	print data, elem1, elem2

	print zeromq_test_serializewave(data)
End

gives

•dostuff()
  data[0]= {1.11244e+09}
elem1[0]= {1.11244e+09}
elem2[0]= {0}
  {
    "data": {
        "raw": [
            {
                "data": {
                    "raw": [
                        {
                            "data": {
                                "raw": [
                                    null
                                ]
                            },
                            "date": {
                                "modification": 1725915394
                            },
                            "dimension": {
                                "size": [
                                    1
                                ]
                            },
                            "type": "WAVE_TYPE"
                        }
                    ]
                },
                "date": {
                    "modification": 1725915394
                },
                "dimension": {
                    "size": [
                        1
                    ]
                },
                "type": "WAVE_TYPE"
            }
        ]
    },
    "date": {
        "modification": 1725915394
    },
    "dimension": {
        "size": [
            1
        ]
    },
    "type": "WAVE_TYPE"
}

[t-b]

Fixed conflicts.

[timjarsky]

@campagnola, can you confirm that this PR has the necessary functionality to pass TP data from MIES to ACQ4?

[campagnola]

I will work with Ben to get this tested. Neither of us is very familiar with Igor, though; is the protocol documented somewhere?

[t-b]

To test this I would do:

• Install the user installer on a rig
• Start Igor Pro
• Lock a device
• Start the TP

In a python console:

• Call the function FFI_GetAvailableMessageFilters via ZeroMQ, the JSON message should look like

{
  "version" : 1,
   "CallFunction" : {
     "name" : "FFI_GetAvailableMessageFilters"
   }
}

python example code is at https://github.com/AllenInstitute/ZeroMQ-XOP/tree/main/examples#python-client.

• Create a zeromq sub socket and subscribe to one or more of the topics returned from FFI_GetAvailableMessageFilters. The new ones are:

StrConstant ZMQ_FILTER_TPRESULT_NOW       = "testpulse:results live"
StrConstant ZMQ_FILTER_TPRESULT_1S        = "testpulse:results 1s update"
StrConstant ZMQ_FILTER_TPRESULT_5S        = "testpulse:results 5s update"
StrConstant ZMQ_FILTER_TPRESULT_10S       = "testpulse:results 10s update"

• Look at the sent messages, their layout is defined at the function comment for PUB_TPResult.

We already do have tests that things work like we think it should, so this is more a request to check that it works like you think it should. The idea of the PR is that programs interfacing with MIES (aka ACQ4) don't need to poll the TP values with FFI_ReturnTPValues but get a message with all the data in the requested interval.

Another thing we added is the possibility to query the stored TPs via TP_GetStoredTP.