Broken Domains Used by Detectors

[rgmz]

Description

Since @lc is doing a great job validating existing detectors, I figured it would be useful to do a high-level check of any domains that no longer resolve or are expired.

I will continue to update this list with findings.

Broken

This could indicate that the specific endpoint is broken and needs attention (e.g., Gitter) or that the domain itself is no longer registered/resolves.

• https://api.apiscience.com

  trufflehog/pkg/detectors/apiscience/apiscience.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.apiscience.com/v1/monitors", nil)

• https://api.base-api.io (fixed in Detector-Competition-Fix: Fix/Remove baseapi detector (no longer exists) #1992)
• https://api.datafire.io (fixed in Detector-Competition-Fix: Fix/Remove DataFire, API retired #1995)

  trufflehog/pkg/detectors/datafire/datafire.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.datafire.io/projects/", nil)

• https://api.gitter.im/

  trufflehog/pkg/detectors/gitter/gitter.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.gitter.im/v1/user/me", nil)

• https://api.happi.dev/ (Detector-Competition-Fix: Fix/Remove Happi Detection & Verification #2003)

  trufflehog/pkg/detectors/happi/happi.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.happi.dev/v1/exchange", nil)

• https://api.idbus.com/ (fixed in Detector-Competition-Fix: Fix/Remove BlaBlaBus, API retired #1996)
• https://api.lexigram.io

  trufflehog/pkg/detectors/lexigram/lexigram.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.lexigram.io/v1/lexigraph/search?q=diabetes", nil)

• https://api.flowdock.com (Detector-Competition-Fix: Fix/Remove Flowdock detector #2004)

  trufflehog/pkg/detectors/flowdock/flowdock.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.flowdock.com/flows?users=false", nil)

• https://api.macaddress.io

  trufflehog/pkg/detectors/macaddress/macaddress.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf("https://api.macaddress.io/v1?apiKey=%s&output=json&search=44:38:39:ff:ef:51", resMatch), nil)

• https://api.meta-api.io/

  trufflehog/pkg/detectors/metaapi/metaapi.go

  Line 60 in 6c35dcf

  req, err := http.NewRequest("GET", fmt.Sprintf("https://api.meta-api.io/api/spells/%s/runSync", resSpellMatch), nil)

• https://api.onwater.io/ (domain redirects to https://isitwater.com/; API url is apparently now https://isitwater-com.p.rapidapi.com)

  trufflehog/pkg/detectors/onwaterio/onwaterio.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf("https://api.onwater.io/api/v1/results/23.92323,-66.3?access_token=%s", resMatch), nil)

• https://api.opengraphr.com/ (domain resolves to a blank Forge/laravel site)

  trufflehog/pkg/detectors/opengraphr/opengraphr.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.opengraphr.com/v1/og?api_token="+resMatch+"&url=https://www.marlonpamisa.com/", nil)

• https://api.passbase.com/ (domain redirects to https://parallelmarkets.com/;

  trufflehog/pkg/detectors/passbase/passbase.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.passbase.com/verification/v1/settings", nil)

• https://api.sentimentinvestor.com/ (subdomain no longer resolves but the domain does; it may be spam now, hard to tell)
• https://api.sherpadesk.com (redirects to https://localhost/metadata despite https://www.sherpadesk.com/ being alive??)

  trufflehog/pkg/detectors/sherpadesk/sherpadesk.go

  Line 54 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://api.sherpadesk.com/organizations/", nil)

• https://api.unify.id/ (the base domain redirects to https://www.prove.com)

  trufflehog/pkg/detectors/unifyid/unifyid.go

  Line 52 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "POST", "https://api.unify.id/v1/humandetect/verify", payload)

• https://app.fakejson.com/ (domain still exists/resolves? They may have retired the site)

  trufflehog/pkg/detectors/fakejson/fakejson.go

  Line 53 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "POST", "https://app.fakejson.com/q", payload)

• https://app.lendflow.io/ (domain still exists. API url may have changed)

  trufflehog/pkg/detectors/lendflow/lendflow.go

  Line 54 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://app.lendflow.io/api/v1/deals", nil)

• https://qckm.io/ (no longer resolves because QuickMetrics has shut down; fixed in Detector-Competition-Fix: Fix/Remove QuickMetrics (shutdown) #1997)

  trufflehog/pkg/detectors/quickmetrics/quickmetrics.go

  Line 51 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "POST", "https://qckm.io/list", payload)

• https://sandbox.impala.travel (subdomain no longer resolves, though domain still exists)

  trufflehog/pkg/detectors/impala/impala.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://sandbox.impala.travel/v1/bookings", nil)

• https://scrapersite.com/ (no longer resolves)

  trufflehog/pkg/detectors/scrapersite/scrapersite.go

  Line 53 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf("https://scrapersite.com/api-v1?api_key=%s&url=https://google.com", resMatch), nil)

• https://secretscanner.ladesk.com/ (subdomain no longer resolves; fixed in Detector-Competition-Fix: Fix LiveAgent Detector & Verifier #2001)

  trufflehog/pkg/detectors/liveagent/liveagent.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://secretscanner.ladesk.com/api/v3/agents", nil)

• https://www.glitterlyapi.com/ (no longer resolves)

  trufflehog/pkg/detectors/glitterlyapi/glitterlyapi.go

  Line 50 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", "https://www.glitterlyapi.com/auth", nil)

• https://scrapersite.com/ (doesn't resolve)

  trufflehog/pkg/detectors/scrapersite/scrapersite.go

  Line 53 in 6c35dcf

  req, err := http.NewRequestWithContext(ctx, "GET", fmt.Sprintf("https://scrapersite.com/api-v1?api_key=%s&url=https://google.com", resMatch), nil)

Parked

• TBD

Other

• https://api.scraperbox.com/ (ssl cert is expired)
• https://getsandbox.com/ (SSL_ERROR_RX_RECORD_TOO_LONG - could just be me
• https://mrticktock.com/ (invalid certificate domain)
• https://api.currencybeacon.com/ (redirects to https://api.currencybeacon.com/)
• https://api.prospect.io (redirects to https://overloop.com)
• https://api.statuspage.io (redirects to https://www.atlassian.com/software/statuspage)
• https://crossbrowsertesting.com/ redirects to https://smartbear.com/product/bitbar/

[fumblehool]

@rgmz What if we add a function in Scanner{} to verify if host exists?
I mean, something like s.isValidDetector() bool.
Inside this function, we'll ping the api url to make sure domain exists.

This way, we can add a Github Action check to verify if any detector needs to be removed. WDYT?

[rgmz]

This could be automated to an extent: subdomains that no longer resolve or domains that have expired are easy to check, behavioral changes are a bit harder.

I am curious whether the team has an existing process to run all the TestX_FromChunk tests with live secrets and review problematic results.

[ahrav]

We have established a daily routine where an automated test suite is executed to assess the performance of our detectors, identifying any failures. However, a segment of these failures is attributed to the expiration of test tokens, which were configured during trial phases. Our current focus is on devising strategies to segregate genuine test failures from those arising due to expired credentials. Furthermore, we are in the preliminary stages of broadening our metrics around detection to garner more insights into the issue at hand.

As our detector arsenal expands, acquiring a deeper understanding of each detector's performance and validity becomes paramount. The experiences from this month alone have highlighted a noticeable count of detectors falling into obsolescence, reinforcing the necessity of this endeavor. 😅

This approach to automation, especially concerning the verification of subdomains and domain expirations, should hopefully aid in filtering out trivial issues, allowing us to prioritize addressing more complex behavioral changes.