Walk directories in filesystem source enumeration

[mcastorina]

Description:

Previously the unit could be a directory, but now it is individual files. This leverages the SourceManager's concurrent units to speed up scanning of large directories.

» find /tmp/files -type f | wc -l
  100000

# before
» ./trufflehog filesystem /tmp/files
2024-01-18T12:48:05-08:00       info-0  trufflehog      finished scanning       {"chunks": 100000, "bytes": 588895, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "27.942921208s"}
# 27.94s

# after
» ./trufflehog filesystem /tmp/files
2024-01-18T12:47:04-08:00       info-0  trufflehog      finished scanning       {"chunks": 100000, "bytes": 588895, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "14.342492916s"}
# 14.34s

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[rosecodym]

LGTM, other than my one question.

It is occurring to me that we might need to all come together sometime soonish to talk about our philosophy of concurrency - I think that different parts of the codebase are treating the topic differently and we might need to do some reconciliation.

[rosecodym]

Does this risk throwing away any errors that we were previously capturing?

[mcastorina]

For reference, the WalkDirFunc describes the err argument:

The err argument reports an error related to path, signaling that WalkDir will not walk into that directory. The function can decide how to handle that error; as described earlier, returning the error will cause WalkDir to stop walking the entire tree.

Perhaps there's an argument that we should record the error in the job report, but as it stands we currently ignore any errors given by WalkDir:

trufflehog/pkg/sources/filesystem/filesystem.go

Lines 112 to 116 in 8380e17

	func (s *Source) scanDir(ctx context.Context, path string, chunksChan chan *sources.Chunk) error {
	return fs.WalkDir(os.DirFS(path), ".", func(relativePath string, d fs.DirEntry, err error) error {
	if err != nil {
	return nil
	}

[dustin-decker]

Looks like this caused a regression
#2350