Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automating file transfer off the FlowCam #20

Open
metazool opened this issue Aug 8, 2024 · 3 comments
Open

Automating file transfer off the FlowCam #20

metazool opened this issue Aug 8, 2024 · 3 comments
Labels
future Future work that is not foreseeably going to considered, but can be revisited

Comments

@metazool
Copy link
Collaborator

metazool commented Aug 8, 2024

Short summary of discussion with Science Computing folks, also relevant to #16

  1. There's an approved route for attaching lab computers to a restricted part of the internal network which has access to a "transfer proxy" (e.g. a machine that will collect data from "quarantined" machines like the FlowCam and forward it on); it does anti-virus scanning and has options to route data direct to cloud storage
  2. On the instrument end this could be as basic as a PowerShell script running in Task Scheduler, which checks a directory for updates and then sends a JSON document to a webhook on the transfer proxy
  3. We don't know whether permission levels on the FlowCam are elevated enough to make this work, but that would be a helpful way of finding out what else we can't do on it

I've contacted Kelly in the lab to suggest a least-effort test of stage 2...

@metazool
Copy link
Collaborator Author

metazool commented Aug 8, 2024

@albags if this approach works in practise then it could leave us with different choices about data transfer

We've discussed the different cases of reuse value for your work https://github.com/AMI-system/ami-api ("API to push data (images and audio files) into an S3 server.") - potentially seen as a backend for a GUI experience of small to medium sized image datasets

In the internal Teams discussion the suggestion is that the transfer proxy could send the source images (in this case the collages before they are segmented) directly to cloud storage (JASMIN/s3)

It still leaves the decollaging needing to run somewhere but at least detaches that from a named individual's on-premise virtual machine...

@metazool
Copy link
Collaborator Author

metazool commented Sep 9, 2024

From Microscope to Metadata talk from last year's RSECon - caveat, haven't watched and don't know if it has an image ML angle

@metazool metazool added the future Future work that is not foreseeably going to considered, but can be revisited label Sep 11, 2024
@metazool
Copy link
Collaborator Author

metazool commented Oct 8, 2024

I've moved this into the TODO part of the backlog in the "Project" view after getting some feedback from Kelly in the lab - they're able to run Powershell commands in the CLI with administrator privilege - which seems very promising for setting up a scheduled task to do file transfer.

It will need followup with @DamienUKCEH's team about possibility of using a transfer proxy with webhook from the FlowCam built-in PC. It's not an action til November, I've added it TODO so we remember to look at it then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
future Future work that is not foreseeably going to considered, but can be revisited
Projects
None yet
Development

No branches or pull requests

1 participant