Make packing results available online for shareability

[mogres]

Use Case

It would be useful to have simularium packing results available online (on S3 etc.) in order to share URLs for visualization.

Solution

Create a new public sub-bucket on AWS S3 to store simularium results and use the URL to load visualizations.

Alternatives

Dropbox, Drive, Firebase, etc. are alternative hosting services that could be used.

[meganrm]

I updated the CORs rules on our bucket so the links work now. http://localhost:9001/viewer?trajUrl=https://cellpack-results.s3.us-west-2.amazonaws.com/jitter/seed_0_test_spheres_test_config_1.0.0_results.simularium

[mogres]

I updated the CORs rules on our bucket so the links work now. http://localhost:9001/viewer?trajUrl=https://cellpack-results.s3.us-west-2.amazonaws.com/jitter/seed_0_test_spheres_test_config_1.0.0_results.simularium

Confirmed this is working on simularium viewer. @meganrm are we good to close this issue or do we want to wait on reorganizing simularium files separately?

[meganrm]

I think it would be good to have a workflow that looked like this:

1. run a packing
2. the results file (just the simularium file, not all the analysis too) is pushed to an AWS folder, probably within this cellpack-results bucket
3. the viewer opens with the new result loaded

but what we need to work out is how to keep the folder from ballooning. do we clear out older results? do we move things we want to keep to the simularium aws bucket?

ideally we'd also keep track of the meta data of these files in firebase. Ie, a json that has the recipe run, the person who ran it, creation date and link to the aws file. with the idea that in the long run we'd then be able to delete files through the database + a gui

[mogres]

I like the idea of adding the metadata, perhaps we could embed it within the Simularium file description as a first pass?

One idea we could implement to save space is clearing out old packing files periodically (every 30 days?) via github actions.

[meganrm]

In Simularium helper:

• add upload_to_s3 which uploads the results to cellpack-results using the s3 python sdk and then saves metadata to the firebase under results collection. the metadata should have:

  • key: results_name
  • user_name (not sure how to get this right now)
  • datestamp (time and date that it was run, I think firebase automates this)
  • the aws url where the results are stored

• add open_in_simularium which takes in the was url and makes a Simularium url and opens it. ex: https://simularium.allencell.org/viewer?trajUrl=https://cellpack-results.s3.us-west-2.amazonaws.com/jitter/seed_0_test_spheres_test_config_1.0.0_results.simularium

[rugeli]

• how do we want to handle aws credentials? in github secrets or save in .env?
• where do we call functions to store results and open_in_simularium?

concerns:

• it is possible to have the same system username, but unlikely. Do we need to address it?
• following the changes, the pack command will be performing multiple tasks
• .simularium is added after the output path more than one place

[mogres]

• We should consider deleting old result files (>6 months) to conserve space on s3. This could be a github action.