Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JSON generation #5

Closed
2 of 5 tasks
danellecline opened this issue Nov 13, 2023 · 7 comments · Fixed by #7
Closed
2 of 5 tasks

Add JSON generation #5

danellecline opened this issue Nov 13, 2023 · 7 comments · Fixed by #7
Assignees
Labels
enhancement New feature or request p1 Priority 1

Comments

@danellecline
Copy link
Collaborator

danellecline commented Nov 13, 2023

Add JSON generation to simplify the steps for processing with pypam.

Reference repo:
https://github.com/mbari-org/soundscape-wavdata
https://github.com/ioos/soundcoop/

Tasks

@danellecline danellecline added the enhancement New feature or request label Nov 13, 2023
@danellecline danellecline self-assigned this Nov 13, 2023
@danellecline danellecline changed the title JSON generation Add JSON generation Nov 13, 2023
@danellecline
Copy link
Collaborator Author

@ryjombari, just FYI, in case you want to follow along.

@danellecline
Copy link
Collaborator Author

danellecline commented Nov 14, 2023

@carueda - where would you like me to put the modules? It should be different than the .nc metadata in metadata to avoid confusion.
Carlos and D.Cline spoke on Slack about this. This will live in src/metadata.

@ryjombari
Copy link
Collaborator

Thanks for the jumpstart @danellecline!

@ryjombari
Copy link
Collaborator

Relaying a thought from communications with Samara @ NOAA: Tooling this demonstration to gather metadata and read data directly from the Google Cloud Platform would pave the way for PBP to be the engine for a lot of NOAA passive acoustic data. If I understand, MinIO can be used to move data from GCP to the processing instance, and PBP can move to using MinIO? How about the gathering of file metadata?

@danellecline danellecline added the p1 Priority 1 label Nov 17, 2023
@danellecline
Copy link
Collaborator Author

danellecline commented Nov 17, 2023

Thanks @ryjombari. @carueda I'm' wondering if 1) does Minio replace boto3, or do they stand side-by-side, and 2) does boto3 support GCP. I know that minio is S3 compliant and can be used for both local and cloud storage operations - I am using it now for video processing pipelines successfully this way.

@carueda
Copy link
Member

carueda commented Nov 17, 2023

I haven't looked into minio with enough detail to tell for sure, but I think it supports AWS, GCP, and probably others. If so, we could replace the boto3 dependency with minio and adjust the appropriate places dealing with file downloads/uploads (s3 and gs URIs)

@carueda
Copy link
Member

carueda commented Jan 23, 2024

As part of #6, I note that Minio seems to only be for s3 compliant endpoints, and I started using the google-cloud-storage package for handling gs://.

@danellecline danellecline linked a pull request Feb 29, 2024 that will close this issue
@carueda carueda closed this as completed in #7 Mar 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request p1 Priority 1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants