Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxytest: proxy HTTPS request using MITM #5802

Merged
merged 6 commits into from
Oct 28, 2024

Conversation

AndersonQ
Copy link
Member

@AndersonQ AndersonQ commented Oct 16, 2024

What does this PR do?

It extends proxytest to support proxying HTTPS requests.

  • The proxy can now act as a MITM (man-in-the-middle) proxy for HTTPS, generating certificates on the fly for the target host.
  • The proxy assumes it should proxy an HTTPS request if it receives a CONNECT requests.

Why is it important?

It'll allow us to create integration tests for mTLS scenarios

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in ./changelog/fragments using the changelog tool
  • [ ] I have added an integration test or an E2E test

Disruptive User Impact

  • None

How to test this PR locally

Run TestRunHTTPSProxy passing the example build tag and follow the instructions to perform a cRUL request using the proxytest:

  • run the test: go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
  • run the provided curl command

Example:

$ go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
=== RUN   TestRunHTTPSProxy
    proxytest_test.go:451: target server running on https://127.0.0.1:34355
    proxytest.go:100: [proxy-https] time=2024-10-16T11:14:33.982+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:236 msg="running on https://127.0.0.1:40241 -> https://localhost:40241"
    proxytest_test.go:473: proxy running on https://localhost:40241
    proxytest_test.go:479: make request to "https://not-a-server.co" using proxy "https://localhost:40241"
    proxytest_test.go:481: curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
    proxytest_test.go:496: CTRL+C to stop
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:183 msg="STARTING - CONNECT '//not-a-server.co:443' HTTP/1.1 127.0.0.1:40496"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:22 msg="handling CONNECT" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:34 msg="hijacked request" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.092+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:282 msg="original URL: https://not-a-server.co:443/, new URL: https://127.0.0.1:34355/"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.097+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:129 msg="EOF reached, finishing HTTPS handler" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] [8a3b5c0f-50c0-40d1-84af-59412ed8bdc3] DONE 0 - CONNECT //not-a-server.co:443 HTTP/1.1 127.0.0.1:40496
^Csignal: interrupt
$ curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
It works!

Related issues

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

Copy link
Contributor

mergify bot commented Oct 16, 2024

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Oct 16, 2024

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 16, 2024
go.mod Outdated
@@ -14,7 +14,7 @@ require (
github.com/dolmen-go/contextio v0.0.0-20200217195037-68fc5150bcd5
github.com/elastic/elastic-agent-autodiscover v0.9.0
github.com/elastic/elastic-agent-client/v7 v7.16.0
github.com/elastic/elastic-agent-libs v0.12.1
github.com/elastic/elastic-agent-libs v0.12.2-0.20241016085313-16569381887f
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

once elastic/elastic-agent-libs#240 is merged it will be updated to v0.12.2

@AndersonQ AndersonQ force-pushed the 4903-https-proxytest branch 4 times, most recently from b80b705 to 89f4a5d Compare October 16, 2024 09:54
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@AndersonQ AndersonQ force-pushed the 4903-https-proxytest branch 5 times, most recently from fd3a299 to eddcaab Compare October 17, 2024 09:41
Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct overall, had some questions about writing HTTP requests by hand.

testing/proxytest/https.go Outdated Show resolved Hide resolved
Comment on lines 209 to 207
resp := bytes.Buffer{}
resp.WriteString(fmt.Sprintf("HTTP/1.1 %d %s\r\n",
statusCode, http.StatusText(statusCode)))
resp.WriteString("Content-Type: text/plain\r\n")
resp.WriteString(fmt.Sprintf("Content-Length: %d\r\n", len(body)))
resp.WriteString("\r\n")
if len(body) > 0 {
resp.Write(body)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would look better if we instead created a http.Response and written it to the buffer.

@AndersonQ AndersonQ force-pushed the 4903-https-proxytest branch 4 times, most recently from b295b2a to b2614b4 Compare October 25, 2024 09:50
@AndersonQ
Copy link
Member Author

@swiatekm I've finally made the changes you requested. Sorry I ended up force pushing them

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server.
@AndersonQ AndersonQ enabled auto-merge (squash) October 28, 2024 09:43
Copy link

@AndersonQ AndersonQ merged commit ad041d4 into elastic:main Oct 28, 2024
14 checks passed
mergify bot pushed a commit that referenced this pull request Oct 28, 2024
The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server.

(cherry picked from commit ad041d4)
@AndersonQ AndersonQ deleted the 4903-https-proxytest branch October 29, 2024 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve test proxy/mock fleet server to proxy HTTPS connections
4 participants