Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Range requests #762

Merged
merged 6 commits into from
Jun 27, 2024
Merged

Range requests #762

merged 6 commits into from
Jun 27, 2024

Conversation

danielballan
Copy link
Member

@danielballan danielballan commented Jun 19, 2024

As noted in #734 the motivating use cases here was DuckDB's httpfs, which tries to do partial access of files over HTTP. Given a server with one DataFrame in it:

tiled serve catalog --temp
from tiled.cilent import from_uri
c = from_uri('http://localhost:8000', api_key='secret')
df = pandas.DataFrame({"A": [1,2,3], "B": [4,5,6], "C": [7,8,9]})
c.write_dataframe(df, key="x")

DuckDB's httpfs plugin issues requests that use the Range header:

import duckdb
db = duckdb.connect(':memory:')
db.execute('install httpfs; load httpfs;')
db.execute('SELECT * FROM read_parquet("http://localhost:8000/api/v1/asset/bytes/x?id=1&api_key=secret")').fetchall()

generating these server logs confirming that Partial Access was invoked:

[531d1958246ae636] 127.0.0.1:59854 (public) - "HEAD /api/v1/asset/bytes/x?id=1&api_key=secret HTTP/1.1" 405 Method Not Allowed
[1e2951b2eaa982dd] 127.0.0.1:59854 (admin) - "GET /api/v1/asset/bytes/x?id=1&api_key=secret HTTP/1.1" 206 Partial Content
[6a5b9f06acf0a7e5] 127.0.0.1:59854 (admin) - "GET /api/v1/asset/bytes/x?id=1&api_key=secret HTTP/1.1" 206 Partial Content
[79d13fc32a84235e] 127.0.0.1:59854 (admin) - "GET /api/v1/asset/bytes/x?id=1&api_key=secret HTTP/1.1" 206 Partial Content
[90ebff815c556fa1] 127.0.0.1:59866 (admin) - "GET /api/v1/asset/bytes/x?id=1&api_key=secret HTTP/1.1" 206 Partial Content

Checklist

  • Add a Changelog entry
  • Add the ticket number which this PR closes to the comment section

Closes #734

Copy link
Contributor

@hyperrealist hyperrealist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, with the caveat that I could only do a shallow review of file_response_with_range.py given my inexperience with starlette. But the logic checks out.

I edited file_response_with_range.py to replace literal http status codes with HTTP_200_OK et. al. from starlette.status.

Also there are few minor comments in the files.

tiled/server/file_response_with_range.py Outdated Show resolved Hide resolved
tiled/server/file_response_with_range.py Show resolved Hide resolved
@danielballan danielballan merged commit 1e94f4d into bluesky:main Jun 27, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Range requests on /assets/bytes/{path}
2 participants