How to differentiate between which hardware the experiment was run on?

[sgbaird]

If the data gets dumped into a single MongoDB collection, there's no way to distinguish which device the experiment was run on, and there's no curation as to whether fake data is being uploaded. There are a few options:

• I could use the Pico ID, but then it's sort of like making the default password "password" in a way that's accessible by anyone when the hardware is connected to the internet (not a huge deal from a security perspective for this demo - someone starts blinking a light and reading light channel data from your device, but not scalable to solutions that require access restriction What does access restriction look like for a scaled up cloud-accessible lab? #91 EDIT: use of Pico ID wouldn't work, because then the Pico ID would be pubicly visible as soon as one data entry gets used - like putting a publicly viewable data entry with the password in it
  • This also leaves room for someone falsely putting data under a particular device when it wasn't actually generated from there. This leads to an access restriction model where only users with specific credentials can contribute to specific collections

In other words, the ID that defines the hardware has to be unique to a device and private to only the users of that device. It should also be separate from credentials that are used to control the hardware, so that the database manager can see the database ID but not have remote hardware access.

On the MongoDB side:

• A role is a named set of permissions that an end-user can have for a document when for a request to the app. Roles are evaluated in order from top to bottom using the apply_when expression.

• Filters modify an incoming MongoDB query to return only a subset of the results matched by the query. Filters will filter out data that is returned by MongoDB before roles are evaluated, making them useful for request performance improvements.

• Requires having a unique (private) key and a collection associated with each hardware instance, where the key and collection get generated by a database manager, or preferably, automatically via some request form that has a mild form of authentication (e.g. Google account)
• MongoDB also has an authentication tab under data access
• API Keys seem to be at the database level, not at the collection level. However, maybe the API key can be distributed freely, while the ID associated with each user is kept private and is part of a custom role or filter. However, registering users requires some setup. EDIT: multiple API keys can be created, each of which have their own IDs associated with them. In other words, API keys can be assigned on a per-person basis, a collection created for each ID, and then have the ID checked before adding to a particular database. The ID should be kept private in this case, but will be accessible by the database manager. The Pico ID doesn't have to be disclosed to the database manager, and it can be changed from the default.

[rekumar]

1. Agree that using using hardware ID's is a bad idea, if only because development boards (arduino's for sure, maybe rpi's) can reuse ID's. At least this is the case for arduino uno's I have purchased from third party resellers.

2. Maybe a public-private key encryption scheme (like RSA, python example) is appropriate here?
   Example: the private key lives on your hardware device, the public key is the device identifier within your database. Any data generated by the device is terminated with some signature, then encrypted using the private key. Your database API then decrypts the data using the public key and checks for the signature before accepting the new data.

As a bonus, you can use these keys to do device-side validation (using its private key) to block instructions from bad actors. Instructions sent to the device would be encrypted using its public key, then decrypted on the device side to check the signature. In this scenario, maybe you don't want to use the public key as the device's ID in the database... you could encrypt the device's ID using a server-side private key to hide this from external viewers of the database.

3. This is a very forward-thinking problem to solve for most of our self-driving labs! Practically speaking, most of these labs will be operating on a university/national lab LAN. In my SDL's, I've relied on 1) our IT professionals to protect our networks from malicious actors, and 2) good API's with type + value checking to make it exceedingly difficult for our internal users to mess things up!

[rekumar]

Of course, happy to participate.

I think once that startup cost is invested in a local system, it's unlikely that it will get converted to cloud-based system unless the barrier is very low.

True, though I'd argue the technological barrier is usually quite low. Any SDL receives an instruction set, so making this cloud-based just takes a wrapper API and some kind of scheduling/queue system. Could be a useful small project...
The real barrier is in maintaining the system like an external user facility. Consumables, maintenance, etc. Not much use for an API if you have to go into lab and set things up yourself anyways, as is the case with most current SDLs. An SDL that can reconfigure/refill itself is a lot more interesting to put on the cloud. Now I'm wondering if there is a defined hierarchy of SDL's similar to levels of self-driving cars...

2. good API's with type + value checking to make it exceedingly difficult for our internal users to mess things up!

Looks like we're thinking along the same lines!

Have you looked at pydantic for this kind of thing? Its pretty nice, clear to work with, and stitches into API's very cleanly. Example for validating job submission to our API:

Defining our Data Model:
https://github.com/CederGroupHub/alab_management/blob/94d02870623eb198e663e7789021e7f3596768c3/alab_management/experiment_view/experiment.py

Validating incoming jobs against the Data Model
https://github.com/CederGroupHub/alab_management/blob/94d02870623eb198e663e7789021e7f3596768c3/alab_management/dashboard/routes/experiment.py#L14-L29

[sgbaird]

I think once that startup cost is invested in a local system, it's unlikely that it will get converted to cloud-based system unless the barrier is very low.

True, though I'd argue the technological barrier is usually quite low. Any SDL receives an instruction set, so making this cloud-based just takes a wrapper API and some kind of scheduling/queue system. Could be a useful small project...

Agreed about requiring a wrapper API and a scheduling/queue system. As I started digging in, I realized there were a lot of nuances dealing with access restrictions, secure data transfer, etc.:

• What does access restriction look like for a scaled up cloud-accessible lab? #91
• https://github.com/sparks-baird/self-driving-lab-demo/discussions/categories/data-and-access-management

In some sense, the "RGB" inputs + sensor data outputs can be a template. See also note about MicroPython below. In terms of scheduling/queue, it's MQTT + FIFO.

The real barrier is in maintaining the system like an external user facility. Consumables, maintenance, etc. Not much use for an API if you have to go into lab and set things up yourself anyways, as is the case with most current SDLs. An SDL that can reconfigure/refill itself is a lot more interesting to put on the cloud.

For sure! Self-reconfigurability and self-refills seem like a beast to tackle in a general way. Large capacity reserves are how I've been addressing this so far. The light demo has "infinite" iterations available, and the liquid demo has the capacity for ~1000 iterations before needing to change out the waste container. Obviously, this gets more difficult with more realistic scenarios. Self-reconfiguring and self-refilling are things worth giving more thought to!

Now I'm wondering if there is a defined hierarchy of SDL's similar to levels of self-driving cars...

Interesting that you mention this. @stanlo229 and I have discussed this a bit. The closest attempt I'm aware of:

Goldman, B.; Kearnes, S.; Kramer, T.; Riley, P.; Walters, W. P. Defining Levels of Automated Chemical Design. J. Med. Chem. 2022, 65 (10), 7073–7087. https://doi.org/10.1021/acs.jmedchem.2c00334.

3. good API's with type + value checking to make it exceedingly difficult for our internal users to mess things up!

Looks like we're thinking along the same lines!

Have you looked at pydantic for this kind of thing? Its pretty nice, clear to work with, and stitches into API's very cleanly. Example for validating job submission to our API:

Defining our Data Model: CederGroupHub/alab_management@94d0287/alab_management/experiment_view/experiment.py

Validating incoming jobs against the Data Model CederGroupHub/alab_management@94d0287/alab_management/dashboard/routes/experiment.py#L14-L29

I've heard of pydantic and it seems like a nice solution for full Python solutions. Thanks for sharing the examples! I struggled a lot with the idea of switching from a single board computer (e.g. RPi 4B) to a microcontroller (i.e. RPi Pico W) and hence MicroPython instead of full Python. The decision to switch was almost exclusively driven by the market constraints on finding RPi 4B's, Zero W's, Zero 2 W's, etc. and the less user-friendly ecosystems (or high cost) of other systems. When I start the powder and solid handling task, I will probably use Hiwonder's ArmPi Pro which has an RPi 4B aboard. That might be a good chance for me to introduce pydantic on the device side.

[rekumar]

Agreed about requiring a wrapper API and a scheduling/queue system. As I started digging in, I realized there were a lot of nuances dealing with access restrictions, secure data transfer, etc.:

I struggled a lot with the idea of switching from a single board computer (e.g. RPi 4B) to a microcontroller (i.e. RPi Pico W) and hence MicroPython instead of full Python.

I understand your perspective a bit more now. In my mind all access control/input checking/etc should be handled at the application level. As long as your hardware only accepts instructions from your application, this should be fine. Not hard to achieve this isolation using an encrypted handshake or an internal (or even dedicated) LAN for communication. The minor risk is that a developer could change the input checking step, though this seems reckless and unlikely. I'd even argue that moving hardware limits to the application level can improve transparency + avoid duplication (ie repeat code at API and hardware levels). However the riskiest cases are probably easy to catch on hardware without something like pydantic (ie if temp>500 raise TooHotError) Additionally, you can imagine limits that involve multiple steps of the SDL (ie temperature limits change based on your solvent's boiling point), which have to be caught at the application level.

Anecdotally, all beamlines I've used as an external user rely on (often easily bypassed) software limits to avoid instrument collisions etc, and they seem to function well enough.

I would love to see a SDL of sufficient scale where these are important issues :)

[sgbaird]

Great points about de-duplication between hardware and API and how to deal with multiple steps. I like the idea of risky situation avoidance being hardcoded on the device. I also like the idea of the application throwing an error for invalid inputs before it starts communicating with external systems.

For SDLs with multiple steps (i.e. multiple pieces of hardware that require separate communication), I lean toward the idea of the user communicating with a central processor as a "firewall" of sorts (e.g., physical hardware or cloud-based server) rather than having the user directly communicate with the individual pieces of hardware. Something where the scope of users that can change code on the physical hardware, server, etc. is narrower than the scope of users that can send requests to it.

I would love to see a SDL of sufficient scale where these are important issues :)

Agreed! Hoping to help make that a reality :)

[sgbaird]

@rekumar, apologies for resurfacing such an old thread, but I wanted to follow up on something.

Agree that using using hardware ID's is a bad idea, if only because development boards (arduino's for sure, maybe rpi's) can reuse ID's. At least this is the case for arduino uno's I have purchased from third party resellers.

Rather than use the hardware ID as some kind of password for authentication, I'm now thinking about using them to make it easier to distinguish between devices when using MQTT communication. Specifically, I'm considering using them within the MQTT "topics" to help organize the messages being sent to various devices. The messages would also include task IDs that get passed back and forth to help prevent crossing wires when messages are being sent around. I guess there's still the risk of two devices happening to have identical APIs and identical hardware IDs..

For context, I've been collecting my thoughts about a "naming schema" in #237.

[sgbaird]

Summary of access restrictions (minus the ability to assign private collections):

privilege	database manager	device owner	device user	general public	solution
read public data	✔️	✔️	✔️	✔️	public API key, MongoDB filters to only public collections
write to database	✔️	✔️	❌	❌	one MongoDB database API key per device owner
assign new database API keys	✔️	❌	❌	❌	via MongoDB online GUI
access map from database API keys to users	✔️	❌	❌	❌	via MongoDB online GUI
send commands to hardware	❌	✔️	✔️	❌	via user-specific authentication
assign new device users	❌	✔️	❌	❌	manually generate new authentication info

The database manager and the device owner each require exclusive, unique authentication. The database manager gets this through MongoDB login credentials. The device owner gets this by having physical access to the hardware and an optional, external server (e.g., pythonanywhere). The user needs to communicate privately with the device which can be handled through manual encryption #133. The device needs to communicate privately with the database, which MongoDB handles.

Related:

• https://crypto.stackexchange.com/questions/9723/vulnerabilities-if-encrypting-the-same-data-with-2-different-keys
• https://www.hackster.io/sandeep-mistry/connect-your-raspberry-pi-pico-w-to-aws-iot-core-8868b7
• https://forum.micropython.org/viewtopic.php?t=11906&p=64760
• https://www.venafi.com/blog/what-difference-between-public-key-and-private-key
• How to differenciate devices/users on mosquitto? eclipse-mosquitto/mosquitto#1645
• https://stackoverflow.com/questions/73000035/trying-to-use-mosquitto-broker-with-tls-using-paho-python

This is closely related to setups for materials Informatics data platforms like Materials Project and NOMAD. Citrination distinguishes between public vs. private collections.

[sgbaird]

MongoDB Docs: Define Permissions for Specific API Keys. Perhaps one collection per device via collection-specific permissions.

User ID of each API key comes from:

Permissions get set as follows:

This seems to be functioning as intended.

I'm still stumped on how to use https://test.mosquitto.org as the host while ensuring that only "authorized" clients get to run experiments on the device. It seems like I need to use SSL with certificates and an encrypted port (which interestingly seems to be commented out on test.mosquitto.org)

• Pico W: Issues with ussl / usocket with encrypted MQTT micropython/micropython#9259
• ussl module for Raspberry Pico W micropython/micropython#9222

EDIT: I think I figured out the secure handshake by using a free instance of HiveMQ and the directions at https://community.hivemq.com/t/getting-started-raspberry-pi-pico-w/1316/21?page=2. PR to follow soon.

[sgbaird]

For provenance, here is a thread from the Eclipse IoT Slack Group:

I want to send instructions to control LEDs from a Python client (we'll call this "Tim") to a MicroPython device (we'll call this "Jill") via the test.mosquitto.org host. Anyone can send instructions to Jill as long as they know the topic, but I want it so that Jill only carries out instructions that were given by Tim.
If I use port 1883, then any messages that are sent are unencrypted and unauthenticated, meaning that a set of instructions can be easily be copied and resent, such that Jill won't be able to tell if the new message is actually from Tim vs. someone pretending to be Tim.
I think one solution is to use encryption (e.g. port 8883) and include a private key that's known only to Tim and Jill. Does this seem right?

Reply from @NorbertHeusser::

Using port 8883 will not really in your case. The TLS layer will only help encrypt the traffic between each of the clients and the broker. It will not be encrypted inside the server. So using an encrypted socket between your client and the broker will only help to avoid anyone will steal you identity. Speaking in you terms anyone between Tim's computer and the server may steal Tims identity (username/password) and kick out the real Tim any time, if you don't use TLS connection.
But still everybody with wildcard subscription can easily see the messages send from Tim to Jill.
To avoid this you would either need to setup you own server where you have full control over the server. And this way nobody may connect to your broker and pickup the data. Or you book a hosted mosquitto somewhere. If you want to test how all of this works you may use one of our (I work for cedalo.com) free trial instances for a limited time.
If you want to stay with one of the open servers like test.mosquitto.org you would need to implement you own end-to-end encryption. Easiest way would be to use a private share secret. So some random secret know by Tim and Jill. Now Tim may use a symmetric encryption algorithm to encode the message and send it encrypted (e.g. as a bas64 encoded string). Now Jill knowing the secret and the algorithm used will be able to decode it. Or you "sign" you message by based on the secret so Jill is able to verify the message is really from Tim by checking the signature.

If you don't have a save way to transfer a secret from Jim to Jill upfront it will get more complicated ;-)

Thank you for the thorough answer! A private server with TLS seems like the way to go. Symmetric encryption alone would (mostly) prevent fake Tim from knowing what the message actually means, but it wouldn't stop fake Tim from copying that same message verbatim and spamming it to Jill, correct?

Reply from @NorbertHeusser:

Yes, you are right. Fake Tim would be able to replay. You could add an incrementing Id with each request and drop requests with older ids. But yes, it get more and more complicated.

EDIT: However, see note from post above about HiveMQ, which solved the issue for me.