[ENG-4624] [S3 Improvements] Project PR - OSF Backend Part

[cslzchen]

Purpose

Enable OSF users to choose subfolders as add-on storage root in project for S3.

Credit @Johnetordoff for all the work 👍

Project notion: https://www.notion.so/cos/S3-improvements-476a5b07cb7e4f458e9cd4c77cfa03ec
WB part: CenterForOpenScience/waterbutler#406

Changes

• Enables users to configure subfolder as add-on root for S3
• Uses boto3 to get S3 buckets
• Uses :/ as the new delimiter for S3 bucket
• Added a management command to migrate existing configuration to use the :/ delimiter
• Added/Updated unit tests

DevOps Notes

• Deployment requires downtime for two reasons
  • Must be deployed at the same time as WB
  • Must run the management command before any user configures an S3 bucket
• For details, see our release playbook (TBD)

Dev Notes

Here are all child PRs that have been merged into this feature branch.

• Use boto3: [ENG-4624] [S3 Improvements] Use boto3 to access bucket regions using V4 s3 auth #10427
• Clean-up:
  • [S3 Improvements] Clean up and revert old changes - part 2 #10424
  • [S3 Improvements] Clean up and revert old changes - part 1 #10423
  • [S3 Improvements] Clean up and revert old changes - part 0 #10422
• Calling format: [S3 Improvements] Make get bucket names use the correct calling format #10419
  • Note: this has been replaced by using boto3
• Delimiter:
  • [S3 Improvements] Delimiter Migration #10418 (Migration)
  • [S3 Improvements] Use colon delimiter for s3 buckets #10415 (Use :/)
• Non-default region: [S3 Improvements] Fix non-default regions for s3 #10417
• Prototype Subfolder: [ENG-4566] [S3 Improvements] S3 Subfolder Selection Feature #10414

QA Notes

See QA docs in the project notion page

Documentation

Update our user guide for using S3 (TBD)

Side Effects

• Our S3 accounts used for testing is and U.S. account, which has limitation on the enabled buckets.
• With boto3, there are new bucket regions that cannot be handled by WB. With our U.S. account, there are 3 of them (Asia Pacific (Osaka) ap-northeast-3, EU (Paris) eu-west-3 and EU (Stockholm) eu-north-1.)

Ticket

https://openscience.atlassian.net/browse/ENG-4624

[Johnetordoff]

@mfraezz I was wondering what you think about this, obviously I want to avoid migrating the logs, I figure if this is purely additive (the bucket name will stay the same string.) we can add path a new thing without causing discontinuous historical data. I feel just adding something to the logs like this is appropriate if it's in extra.

[mfraezz]

It's a Product decision whether we want to display folder info in these logs. Presently, it will maintain existing behavior and just say linked the Amazon S3 bucket ${bucket} sans any folder path info, which is fine if Product signs off on it.

[mfraezz]

One small request, one Minor open question. Otherwise LGTM

[mfraezz]

Minor, and I don't really know the best place for this one, but related to the comment:

Using folder_id for the bucket and folder_path for the... folder path would be more consistent other addon behavior. What was the rationale for not doing so -- avoiding a migration?

[Johnetordoff]

Yes avoiding the migration was the rational.