Implement proxy v2 architecture, in Rust #1718

… proxy APIs This is the architecture we've dubbed "proxy v2".

We identified multiple features that the current implementation of the proxy is unable to support (e.g. progress reporting, resuming downloads), necessitating a new protocol dubbed "v2". We are taking the opportunity to rewrite this component in Rust. Normal operation is mostly the same, input is received as a JSON blob (parsed and validated by serde) and output in our custom JSON response format. The `url` crate (from servo) is used to assemble and validate the target URL, and `reqwest` fires off the HTTP request. Downloads are now streamed back to the client over stdout, and metadata passed over stderr. Tests and further integration will happen in follow-up commits. Refs #1678. Co-authored-by: Kunal Mehta <[email protected]>

Rewrite the existing tests to be integration tests against a compiled Rust binary. We use the httpbin library to start up a Python webserver and instruct the proxy to connect to it. This allows to test connection properties that aren't recordable in the VCR format, like timeouts or streamed responses. The tests are reorganized to be split into proxy handling and error handling.

The SDK will now unconditionally use proxy v2: in development mode, it'll shell out directly, while production mode will invoke it over qrexec. The dev `./run.sh` script will compile the proxy so it's ready for use before the client starts up. New typed dataclasses represent the two types of responses that can be returned. No user-facting changes are happening at this stage, but this will enable future client features. The union return of send_json_request() means that we need instanceof assertions to make mypy happy. One minor logic bug in API.get_submission() was fixed in the case that no request is made (an undefined exception would've been raised previously).

@vickyliin

We can't use VCR.py's "custom_patches" parameter because our API._send_json_request() is RPC- rather than connection-oriented. But we can just instrument _send_json_request() directly, which is what we do here. We subclass vcr.cassette.Cassette to handle identical requests with different responses, which was suggestd by @vickyliin as a workaround for kevin1024/vcrpy#753. Now that the SDK‒proxy connection is itself instrumented, there's only one path to test, with no special error-handling logic required, so merge TestAPIProxy into TestAPI. I considered merging TestAPI and TestShared as well, now that (without TestAPIProxy) TestAPI is the only subclass of TestShared. But reorganizing the alphabetized helpers in TestShared versus the strictly-sequenced TestAPI methods can wait. The tests are also now more patient with slow deletion operations. I'd want to DRY up this logic if this pattern shows up in more places, but it would require adding another level of indirection. A @Retry decorator isn't appropriate at the level of the test method, and a context manager can't loop over its closure. And re-apply the hack from 880635d by renaming test_logout to start with a "z" so it runs last.

Help the developer through regenerating SDK cassettes against a development server, as well as in CI.

* Switch to `Arch: any` because the package contains compiled code and this enables debhelper's automatic shlibs dependency system. * Rename the binary to `securedrop-proxy` because that's the Rust name.

Audits were done by Kunal over the course of a few months.

Same as the SDK, use our custom VCRAPI wrapper instead of pytest-vcr. Because each of the functional tests log in and out, document the server hack needed to run them one after another in the README.

Now replaced by the Rust version.

Fixes #1761.

We are setting the proxy's origin in QubesDB (via dom0 salt), and as discussed/agreed upon in <freedomofpress/securedrop-workstation#1013>, we're reading it directly in the proxy. We link directly to libqubesdb, generating the Rust wrapper with bindgen so it stays in sync (at build time at least). As a result, some unsafe is needed, but it is clearly annotated with relevant SAFETY notes. If we're not building with QubesDB (for development purposes), we'll continue to read from the environment and skip the bindgen and linking steps entirely. Fixes <freedomofpress/securedrop-workstation#853>. Co-authored-by: Kunal Mehta <[email protected]>

libclang is needed by bindgen. The Qubes signing key was fetched from <https://github.com/QubesOS/qubes-builder-debian/blob/64f530a712539829f8f2c1fbb0907d99b562ccdd/keys/qubes-debian-r4.2.asc>.

… environments

This does nothing because Rust code is linted through the root Makefile's `rust-lint` target, and Python code linted from the root's ruff targets as well.

Historically production systems used ~/QubesIncoming/sd-proxy as a temporary directory because files were sent over qvm-move. This switches to ~/Downloads as a stop-gap. A better interface would be for the caller to provision a temporary directory, and pass it as the path.

Avoid IndexError in the case that `response.stdout` is empty.

Co-authored-by: Kunal Mehta <[email protected]>

The relevant httpbin issue has been resolved and is compatible with newer werkzeug.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement proxy v2 architecture, in Rust #1718

Implement proxy v2 architecture, in Rust #1718

Commits on May 17, 2024