pyatls: urllib3 and requests support with cleanup #10

HernanGatta · 2023-08-27T17:48:24Z

Overview

This PR contains various improvements and features. Namely, this PR:

restructures the validators into a deeper hierarchy for classes to have shorter, cleaner names, and so as to create space for additional validators in the future. For example, atls.validators.AzAasAciValidtor becomes atls.validators.azure.aas.aci.AciValidator;
adds support for monkey-patching urllib3 such that, with the patch in place, all HTTPS requests automatically become HTTP/aTLS requests, enabling users of urllib3 to leverage aTLS (this is not required for our usage, I left this here because it was the first idea I had to make requests work with aTLS, but I had a better one afterward and figured I could leave the direct urllib3 support if someone wants to use it);
adds support for aTLS for the requests library via an adapter that captures URLs with the httpa:// scheme and routes them into HTTP/aTLS connections, enabling users of requests to transparently adopt aTLS.

Note: Commits are structured to be reviewed independently.

Requests Adapter

The primary addition of this PR is item no. 3.

Since the requests adapter ties cleanly into the library, consumers of it can seamlessly leverage the library's support for connection pooling. With connection pooling, it is no longer necessary to create a new aTLS connection for every request. Instead, a readily established connection can be recycled for subsequent requests. This in turn dramatically reduces request latency.

The HTTPAAdapter captures requests made against URLs with the httpa:// scheme and uses this package's HTTPAConnection class. This automatically upgrades any requests made with that scheme to HTTP/aTLS.

Sample Usage:

import requests
from atls.utils.requests import HTTPAAdapter
from atls.validators.azure.aas import AciValidator

validator = AciValidator()

session = requests.Session()
session.mount("httpa://", HTTPAAdapter([validator]))

response = session.request("GET", "httpa://my.aci.confidentialservice.net/index")

print(f"Status: {response.status_code}")
print(f"Response: {response.text}")

github-actions · 2023-08-27T17:53:38Z

🦙 MegaLinter status: ✅ SUCCESS

Descriptor	Linter	Files	Elapsed time
✅ ACTION	actionlint	1	0.02s
✅ PYTHON	black	17	0.94s
✅ PYTHON	flake8	17	0.53s
✅ PYTHON	isort	17	0.27s
✅ YAML	yamllint	5	0.27s

See detailed report in MegaLinter reports

You could have the same capabilities but better runtime performances if you use a MegaLinter flavor:

oxsecurity/megalinter/flavors/[email protected] (59 linters)
oxsecurity/megalinter/flavors/[email protected] (82 linters)

MegaLinter is graciously provided by

zizhong · 2023-08-28T17:19:40Z

python-package/README.md

@@ -72,12 +72,12 @@ running on a confidential ACI instance with the corresponding attestation
 document issuer, and submit an HTTP request:

 ```python
-from atls import AttestedHTTPSConnection, AttestedTLSContext
+from atls import HTTPAConnection, ATLSContext


Would be great to add the httpa example here.

zizhong · 2023-08-28T17:29:30Z

python-package/sample/connect_aci.py

+
+
+def use_requests() -> None:
+    session = requests.Session()


nit: possibly low priority but better to add this as pytest.

I'd like to add unit tests that run automatically but we need an ACI-capable server instance. I do want to add tests for things that can be tested without a server-side component, though; that's on my to-do list.

zizhong · 2023-08-28T17:32:54Z

python-package/sample/connect_aci.py

+    # Mount the HTTP/aTLS adapter such that any URL whose scheme is httpa://
+    # results in an HTTPAConnection object that in turn establishes an aTLS
+    # connection with the server.
+    session.mount("httpa://", HTTPAAdapter([validator]))


How to set timeout for the connection in this case?

session.request(..., timeout=)

octaviansima · 2023-08-29T16:55:34Z

python-package/src/atls/utils/urllib3/patch.py

+    if _orig_urllib3_connection_cls is None:
+        return
+
+    import urllib3


Nit: can we just do this once at the top of the file?

octaviansima · 2023-08-29T16:56:47Z

python-package/src/atls/utils/requests/adapter.py

+from urllib3.util.retry import Retry as Retry
+
+
+class _HTTPAConnectionShim(HTTPAConnection):


Does this class need to be defined twice?

If you look closely, they're slightly different.

Why can't we use this definition for the inner class in the other spot? The only difference I can see is where validators is gotten from

octaviansima · 2023-08-29T21:01:01Z

python-package/src/atls/utils/requests/adapter.py

+from urllib3.util.retry import Retry as Retry
+
+
+class _HTTPAConnectionShim(HTTPAConnection):


Why can't we use this definition for the inner class in the other spot? The only difference I can see is where validators is gotten from

octaviansima · 2023-08-29T21:01:17Z

python-package/src/atls/utils/requests/adapter.py

+    HTTPAConnection class.
+    """
+
+    Validators: List[Validator]


Nit

Suggested change

Validators: List[Validator]

validators: List[Validator]

pyatls: shorten class names

d92f7b9

HernanGatta added feat New feature for the user refactor Refactoring production code, e.g. renaming a variable. labels Aug 27, 2023

HernanGatta self-assigned this Aug 27, 2023

HernanGatta marked this pull request as draft August 27, 2023 17:52

HernanGatta force-pushed the hegatta/improvements branch from 17218a4 to 30d8b70 Compare August 27, 2023 17:55

pyatls: support atls in urllib3

dcb0e25

HernanGatta force-pushed the hegatta/improvements branch 3 times, most recently from d65f47e to 2c9d6ad Compare August 27, 2023 23:46

HernanGatta requested review from chester-leung, zizhong, octaviansima and ggroode August 27, 2023 23:57

HernanGatta marked this pull request as ready for review August 27, 2023 23:58

HernanGatta mentioned this pull request Aug 27, 2023

use requests with attested connection pooling opaque-systems/opaquegateway-python#22

Merged

HernanGatta force-pushed the hegatta/improvements branch 2 times, most recently from d576517 to d701564 Compare August 28, 2023 01:26

HernanGatta added 5 commits August 28, 2023 01:27

pyatls: support atls in requests

ffe9fd7

pyatls: restructure validators into hierarchy

dc71720

samples: update ACI sample

0ce8bdd

pyatls: update pyproject

58c7d96

pyatls: update README

4130e15

HernanGatta force-pushed the hegatta/improvements branch from d701564 to 4130e15 Compare August 28, 2023 01:27

zizhong reviewed Aug 28, 2023

View reviewed changes

octaviansima reviewed Aug 29, 2023

View reviewed changes

address comments

c354134

HernanGatta requested review from octaviansima and zizhong August 29, 2023 20:41

octaviansima reviewed Aug 29, 2023

View reviewed changes

octaviansima approved these changes Aug 29, 2023

View reviewed changes

HernanGatta merged commit cb13afe into dev Aug 29, 2023
1 check passed

HernanGatta deleted the hegatta/improvements branch August 29, 2023 21:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyatls: urllib3 and requests support with cleanup #10

pyatls: urllib3 and requests support with cleanup #10

HernanGatta commented Aug 27, 2023 •

edited

Loading

github-actions bot commented Aug 27, 2023 •

edited

Loading

zizhong Aug 28, 2023

HernanGatta Aug 29, 2023

zizhong Aug 28, 2023

HernanGatta Aug 29, 2023

zizhong Aug 28, 2023

HernanGatta Aug 29, 2023

octaviansima Aug 29, 2023

HernanGatta Aug 29, 2023

octaviansima Aug 29, 2023

HernanGatta Aug 29, 2023

octaviansima Aug 29, 2023

octaviansima Aug 29, 2023

octaviansima Aug 29, 2023

		from urllib3.util.retry import Retry as Retry


		class _HTTPAConnectionShim(HTTPAConnection):

pyatls: urllib3 and requests support with cleanup #10

pyatls: urllib3 and requests support with cleanup #10

Conversation

HernanGatta commented Aug 27, 2023 • edited Loading

Overview

Requests Adapter

Sample Usage:

github-actions bot commented Aug 27, 2023 • edited Loading

🦙 MegaLinter status: ✅ SUCCESS

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HernanGatta commented Aug 27, 2023 •

edited

Loading

github-actions bot commented Aug 27, 2023 •

edited

Loading