-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add JSON generation #5
Comments
@ryjombari, just FYI, in case you want to follow along. |
@carueda - where would you like me to put the modules? It should be different than the .nc metadata in metadata to avoid confusion. |
Thanks for the jumpstart @danellecline! |
Relaying a thought from communications with Samara @ NOAA: Tooling this demonstration to gather metadata and read data directly from the Google Cloud Platform would pave the way for PBP to be the engine for a lot of NOAA passive acoustic data. If I understand, MinIO can be used to move data from GCP to the processing instance, and PBP can move to using MinIO? How about the gathering of file metadata? |
Thanks @ryjombari. @carueda I'm' wondering if 1) does Minio replace boto3, or do they stand side-by-side, and 2) does boto3 support GCP. I know that minio is S3 compliant and can be used for both local and cloud storage operations - I am using it now for video processing pipelines successfully this way. |
I haven't looked into minio with enough detail to tell for sure, but I think it supports AWS, GCP, and probably others. If so, we could replace the boto3 dependency with minio and adjust the appropriate places dealing with file downloads/uploads (s3 and gs URIs) |
As part of #6, I note that Minio seems to only be for s3 compliant endpoints, and I started using the |
Add JSON generation to simplify the steps for processing with pypam.
Reference repo:
https://github.com/mbari-org/soundscape-wavdata
https://github.com/ioos/soundcoop/
Tasks
The text was updated successfully, but these errors were encountered: