Draft: Feature/config verification

[zricethezav]

Description:

Gives users the ability to perform custom detector verification without having to spin up a verification server. With this feature TruffleHog makes the requests straight to the provider server much like a regular built-in detector.

ex config:

# config.yaml
detectors:
  - name: airtable detector
    keywords:
      - airtable
    regex:
      appRes: (app[a-zA-Z0-9_-]{14})
      keyPat: \b(key[a-zA-Z0-9_-]{14})\b
    verify:
      - endpoint: "https://api.airtable.com/v0/{$appRes}/Projects"
        directVerify: true
        unsafe: true
        httpMethod: "GET"
        headers:
          - "Authorization: Bearer {$keyPat}"
          - "x-api-key: {$appRes}"
        successRanges:
          - "2xx"
          - "300"
          - "301"
        successBodyContains:
          - "success"

  - name: artsy detector
    keywords:
      - artsy
    regex:
      keyPat: (?i)(?:artsy)(?:.|[\n\r]){0,40}\b([0-9a-zA-Z]{32})\b
      idPat: (?i)(?:artsy)(?:.|[\n\r]){0,40}\b([0-9a-zA-Z]{20})\b
    verify:
      - endpoint: "https://api.artsy.net/api/tokens/xapp_token?client_id={$idPat}&client_secret={$keyPat}"
        directVerify: true
        unsafe: true
        httpMethod: "POST"
        successRanges:
          - "2xx"
        successBodyContains:
          - "expires_at"

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[dustin-decker]

I think we'll need a way to reference test secrets for this and to be able to run tests easily.

Also, if we port any existing detectors we will need a way to specify raw and rawv2 field mapping to the regexes. That part would only be needed for porting detectors, not for new ones.

[zricethezav]

Also, if we port any existing detectors we will need a way to specify raw and rawv2 field mapping to the regexes. That part would only be needed for porting detectors, not for new ones.

We could add something like an internal entry that dictates how we want to handle raw and rawv2. We could include options to use specific capture groups if the regexes have more than one capture group (not sure if any of are detectors do, but just an idea)

  - name: artsy detector
    keywords:
      - artsy
    regex:
      keyPat: (?i)(?:artsy)(?:.|[\n\r]){0,40}\b([0-9a-zA-Z]{32})\b
      idPat: (?i)(?:artsy)(?:.|[\n\r]){0,40}\b([0-9a-zA-Z]{20})\b
    verify:
      - endpoint: "https://api.artsy.net/api/tokens/xapp_token?client_id={$idPat}&client_secret={$keyPat}"
        directVerify: true
        unsafe: true
        httpMethod: "POST"
        successRanges:
          - "2xx"
        successBodyContains:
          - "expires_at"
    internal:
         raw: {$idPat}:{$keyPat}
         rawv2: {$idPat}

[mcastorina]

I really like this idea! What do you think about using an existing templating engine instead of string replacement? I ask because there are weird edge-cases other libraries have already solved (like.. {$double{$nested}shenanigans}). mustache could be an option.

[rosecodym]

I also strongly support at least investigating the use of an existing template engine.

Also, does your draft handle verification indeterminacy yet?

[zricethezav]

What do you think about using an existing templating engine instead of string replacement?

I have a preference for keeping things simple until the need arises for an external library. Do you have an example of an edge case?

Also, does your draft handle verification indeterminacy yet?

No. As stated in the description this draft is for enhancing the existing custom config. It is not meant to be a replacement for the existing detectors.