Deephaven Extensions

[mofojed]

Related to an investigation in what it would take to get matplotlib working in our system, I wanted to make it generic enough that we can add extensions easily for other packages as well (eg seaborn).
This is focused only on python.

I have a proof of concept in my forked repo, see the README there for info on how to start it up:
https://github.com/mofojed/deephaven-core/tree/matplotlib

Exporting objects

Currently, we have an enum ExportedObjectType for all exported objects. To be extendable, we need some way to check if an object in the python session is an object that should be exported to the client, and then we need some way for the client to get details about that object.
Proposal is to add a getDeephavenObject method to any object that should be exported, and that method returns a String that is opaque to Deephaven, gets transferred to the client, and then a UI plugin there can handle the payload however it wishes (eg. base64 decode an image, JSON parse an object from a string, etc).
In the case of the matplotlib example, getDeephavenObject returns a base64 encoded image of the figure being exported: https://github.com/mofojed/deephaven-core/blob/cf0940e3f1737771e5fa29d1dab11a6fcbcc07f5/README.md?plain=1#L17.
On the UI side that base64 encoded image is then decoded and displayed: https://github.com/mofojed/web-client-ui/blob/bb6892ed1419a74c98953a0cb74673e6dd0243f7/packages/dashboard-core-plugins/src/panels/MatPlotLibPanel.jsx#L33

Extension structure

My proposal for Deephaven Extensions is being able to provide a folder/package that contains all the files to add additional functionality on both the backend and frontend. I imagine the folder structure something like:

/MyExtension
  /init
    /grpc-api
    /py-env
  /ui
    /dashboard-plugins

Not too sure what the install command/process would be yet, that is something to discuss here. But in terms of the structure:

• /init folder for initialization scripts. In the matplotlib case, this would need a script in grpc-api to pip install matplotlib and a script in py-env to extend matplotlib.figure.Figure with the getDeephavenObject method. We could add subfolders for any other initialization scripts we may need (eg. for other containers)
• /ui folder for Web UI dashboard plugins. In the matplotlib case, the MatPlotLibPlugin handles it.

Other Considerations

• Changing values/ticking data. How do we support extensions sending data updates from the backend to the UI after the object is created?
• How do we handle small deltas in the exported objects?

[niloc132]

My general opinion is that you need something like three examples to generalize something like this. Of our other exported types, ignoring changing values in the export itself, a Figure seems to be a good second example, but I would encourage trying to find a third that also maps well to this idea.

Figures would offer a possible answer for the ticking data part of the question - when a figure is exported, it also provides tickets for other objects, such as Tables and TableMaps. If the getDeephavenObject method returned not just base64 data of the object itself, but also an array of other tickets (and presumably the index in that array would be referenced from in the base64 data), the client can decide how to subscribe to those tables, and get updates as necessary through either the TableService.exportedTableUpdates stream, or through individual subscription streams.

If the exported object itself can change over time (rather than re-export the entire thing), this might suggest that TableMap would be another good candidate to include in this model. The risk here is that TableMap might end up with some custom serialization mechanism (to handle SmartKeys), to handle its append-only updates. My hunch though is that unless these objects are gigantic, a delta would be easier to manage by resending the entire object, but I think more discussion around specific examples would make sense. If the changing part of the model can be made to fit into an associated Table (consider a Table in a TableMap that consists only of the keys), that would make for a straightforward way to pass the data.

--

One final note, base64 isn't strictly required by the transport mechanism - protobuf can support a bytes field, and we can optionally expose that to the JS UI as a typedarray of bytes, or a base64 string.

[devinrsmith]

I think it's definitely possible to have input table UI-controlled support via our ideas about extensible types:

x = create_double_input_table("x")
x_slider = slider_extension(x) // can be specified server side, or created client side/layout

my_downstream_table = something_else.naturalJoin(x, "").where("X > x")
...

The x_slider type will reference the input table ticket, and the client can then interact w/ the input table ticket via api.

[niloc132]

The actual Table instance could already exist by the time you are exporting it, but perhaps not, or perhaps some final operation is being applied on its way "out the door" now that the widget knows it is being sent. This is how figures work today - the table was already picked when it was made available as a bound variable, but during the course of exporting, the table is sometimes changed slightly (extra columns removed).

Actually marking them for export, and being sure that the appropriate metadata makes its way with them seems like the "hard" part to get right consistently. My off-the-top-of-my-head is that the "get the data for transport" method should return an object with the various fields populated - the data that the client will want, initiate the "unsolicited export" and put the tickets themselves in a different field. Alternatively, instead of plain tickets (and requiring a followup round trip to get the full schema etc), just embed the entire ExportedTableCreationResponse into the main payload as figures currently do - but that requires that every way of serializing include a way to handle the data in that existing protobuf message, and the embedded flight message within it.

--

Input tables will be pretty much just "a table, except it has a flag set on it" so that you know you can poke the input table APIs and reference that ticket. It shouldn't require a wrapper object, just additional metadata/attributes?

[mofojed]

I think including plain tickets would be fine... for table tickets, we'd probably want a map from a name to the table, if we have multiple tables we'll want to know which is which.

[niloc132]

Agreed, but I'm suggesting that this mapping be in the data portion, rather than in the "ticket" portion, so that a) you can have anonymous tickets if you want, and b) you can have the same ticket referred to by more than one name (consider a plot which has multiple series using the same table, with different names, etc).

[mofojed]

Yup I think we're on the same page

[devinrsmith]

As another example that may fall under this category, I've wanted a widget that could be driven off of a Deephaven table. For example, I have a table that has a single column and single ticking row, representing percentage with a value in the range [0.0, 1.0]. I'd like the ability to view it as a half-circle speedometer and/or a temperature gauge.

x = compute_my_perc_table()

w1 = speedometer(x)
w2 = tempgauge(x)

These types of widgets may be driven server + client side; or completely client side (a layout that defines w1/w2 instead of having it done via server vars).

[mofojed]

That's a good example, as it requires a table and updating data.

[devinrsmith]

I'm more interested in drilling down into how we can extend the abstractions / serialization around ExportedObjectType before trying to imagine what it might look from a folder structure. (I have reservations with how we've set some stuff up in the past...)

Specifically, I think the goals are:

1. Support custom extensions without the need to change server code
2. Support "complex" custom type information, without sending the full data payload
3. Provide an easy method for installing / referencing extensions

[mofojed]

Yes, for ExportedObjectType, @niloc132 had some ideas about specifically how to do it using the CustomInfo message part of a Field.
We'll break this down into tickets for each piece and then those tickets can have the implementation details.

[mofojed]

Created a ticket to track the ExportedObjectType stuff: #1485

[mofojed]

Another thing to consider in terms of directory structure... we may want a startup script, and also a resource file that's available on the path. Is there a way to do that cleanly @devinrsmith ?
Just looking at matplotlib, you can pass your own stylesheet, but we'd need to have it somewhere on the path to pass it.

[mofojed]

This could be done by pip installing a package, whether remote or "local" (assuming we copy the whole init folder over or something)