You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
We would like to use daft.read_parquet method due to it's efficiency. However, we our files are access via STS assume role credentials from S3. There is an io_config=S3Config we can specify to set access key to use for S3 access, however I see session_token missing.
Describe the solution you'd like
The session_token can be added to S3 Config:
Setting environment variables is not ideal because each file is accessed with different credentials in most cases which would break in a distributed environment.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
* Adds `session_token` field to `S3Config` allowing users to pass temp
session tokens (like from sts)to daft.
* Adds IO integration tests to ensure that we can use the IOConfig to
access s3 with a signed request.
* Closes: #1216
Is your feature request related to a problem? Please describe.
We would like to use
daft.read_parquet
method due to it's efficiency. However, we our files are access via STS assume role credentials from S3. There is anio_config=S3Config
we can specify to set access key to use for S3 access, however I seesession_token
missing.Describe the solution you'd like
The
session_token
can be added to S3 Config:Describe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: