Docker Image Identification in Tar Files #1643

joeleonjr · 2023-08-18T20:53:39Z

Description:

The scanner will currently untar a file and look through the archive for secrets; however, this approach misses secrets hidden in Docker image layers. (The docker save command turns docker images into tar files.) This PR adds docker image scanning inside all archived files identified as the application/x-tar mimetype. A use-case for this feature would be scanning a filesystem, identifying tar files, and then checking if any of them are docker images.

Checklist:

Tests passing (make test-community)?
Lint passing (make lint this requires golangci-lint)?

ahrav

Overall looks great. Some small comments and just some Go style tweaks. Awesome job!!
🎉

pkg/handlers/archive.go

pkg/handlers/handlers.go

pkg/sources/sources.go

pkg/sources/docker/docker.go

pkg/handlers/archive_test.go

joeleonjr · 2023-08-24T13:11:05Z

@ahrav I moved the integration test into a separate file (and made the updates you suggested), but it's throwing an error. Locally it worked great. This is what I'm seeing:

panic: test timed out after 5m0s
running tests:
	TestFoundKeyInDockerTar (5m0s)

goroutine 62 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2[259](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:260) +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.0/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 5 minutes]:
testing.(*T).Run(0xc00050d860, {0xeaba6e?, 0x52bdfc?}, 0xef8168)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x1798fe0?)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc00050d860, 0xc0005bfc48)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000555180?, {0x176ed40, 0x6, 0x6}, {0xc?, 0xc0005bfd08?, 0x17983e0?})
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000555180)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0x636
main.main()
	_testmain.go:57 +0x19c

goroutine 20 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000506100)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 8 [running]:
	goroutine running on other thread; stack unavailable
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x3ad

goroutine 40 [select]:
github.com/trufflesecurity/trufflehog/v3/pkg/sources/docker.ScanDockerImg({0xfef4b0, 0xc000879d70}, {0xff3c78, 0xc000634040}, 0xc000133380, 0xc00007e370)
	/home/runner/work/trufflehog/trufflehog/pkg/sources/docker/docker.go:217 +0xdc5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile({0xfef4b0?, 0xc000879d70}, {0xfe06c0, 0xc000879da0}, 0x0?, 0x0?)
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/handlers.go:65 +0x6e5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar.func1()
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:88 +0x65
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar in goroutine 8
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:86 +0x8d8
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	[300](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:301).061s

ahrav · 2023-08-24T14:17:45Z

@ahrav I moved the integration test into a separate file (and made the updates you suggested), but it's throwing an error. Locally it worked great. This is what I'm seeing:

panic: test timed out after 5m0s
running tests:
	TestFoundKeyInDockerTar (5m0s)

goroutine 62 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2[259](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:260) +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.0/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 5 minutes]:
testing.(*T).Run(0xc00050d860, {0xeaba6e?, 0x52bdfc?}, 0xef8168)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x1798fe0?)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc00050d860, 0xc0005bfc48)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000555180?, {0x176ed40, 0x6, 0x6}, {0xc?, 0xc0005bfd08?, 0x17983e0?})
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000555180)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0x636
main.main()
	_testmain.go:57 +0x19c

goroutine 20 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000506100)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 8 [running]:
	goroutine running on other thread; stack unavailable
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x3ad

goroutine 40 [select]:
github.com/trufflesecurity/trufflehog/v3/pkg/sources/docker.ScanDockerImg({0xfef4b0, 0xc000879d70}, {0xff3c78, 0xc000634040}, 0xc000133380, 0xc00007e370)
	/home/runner/work/trufflehog/trufflehog/pkg/sources/docker/docker.go:217 +0xdc5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile({0xfef4b0?, 0xc000879d70}, {0xfe06c0, 0xc000879da0}, 0x0?, 0x0?)
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/handlers.go:65 +0x6e5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar.func1()
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:88 +0x65
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar in goroutine 8
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:86 +0x8d8
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	[300](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:301).061s

Oh i wonder if the runner timed out after the 5 min interval. We can try to increase the timeout from 5m to test.

zricethezav · 2023-08-28T16:12:26Z

@joeleonjr would this solve #674 ?

joeleonjr · 2023-08-28T17:41:33Z

@joeleonjr would this solve #674 ?

Good catch. Yes, this closes #674

CLAassistant · 2024-08-17T04:58:08Z

All committers have signed the CLA.

joeleonjr and others added 12 commits August 18, 2023 13:07

reworked tarred docker img scanning

af83970

switched back log context naming

e01f6d0

adjusted contexts in archive tests

5b9a862

Merge branch 'main' into docker-tar-identification-from-filesystem-2

a552373

attempting to fix archive test

aa461ae

temp increase file size in maxreader, handler improvements

2d46cf1

refactored to call docker chunk scanner directly

5f4f513

moved docker base/img hint to sources.go

95f6528

saving test case progress

6b8ce2f

cleaned up code

f411979

reverted size

e523b69

Merge branch 'main' into docker-tar-identification-from-filesystem-2

f7cf80e

ahrav reviewed Aug 23, 2023

View reviewed changes

joeleonjr and others added 2 commits August 24, 2023 08:46

added ahrav's comments

51aadf9

Merge branch 'main' into docker-tar-identification-from-filesystem-2

b38bb0d

joeleonjr added 3 commits August 24, 2023 13:50

Merge branch 'main' into docker-tar-identification-from-filesystem-2

7077f69

Update archive_integration_test.go

b1a1fa3

updating secret string for docker detection

b8468af

ahrav approved these changes Aug 24, 2023

View reviewed changes

joeleonjr marked this pull request as ready for review August 24, 2023 18:19

joeleonjr requested review from a team as code owners August 24, 2023 18:19

joeleonjr requested a review from bill-rich August 24, 2023 18:21

Merge branch 'main' into docker-tar-identification-from-filesystem-2

f623c06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Image Identification in Tar Files #1643

Docker Image Identification in Tar Files #1643

joeleonjr commented Aug 18, 2023 •

edited

Loading

ahrav left a comment

joeleonjr commented Aug 24, 2023

ahrav commented Aug 24, 2023

zricethezav commented Aug 28, 2023

joeleonjr commented Aug 28, 2023

CLAassistant commented Aug 17, 2024 •

edited

Loading

Docker Image Identification in Tar Files #1643

Are you sure you want to change the base?

Docker Image Identification in Tar Files #1643

Conversation

joeleonjr commented Aug 18, 2023 • edited Loading

Description:

Checklist:

ahrav left a comment

Choose a reason for hiding this comment

joeleonjr commented Aug 24, 2023

ahrav commented Aug 24, 2023

zricethezav commented Aug 28, 2023

joeleonjr commented Aug 28, 2023

CLAassistant commented Aug 17, 2024 • edited Loading

joeleonjr commented Aug 18, 2023 •

edited

Loading

CLAassistant commented Aug 17, 2024 •

edited

Loading