identity: Update metrics to follow OpenMetrics best practices

[olix0r]

In preparation to add SPIRE integration, this change extracts the identity certificate metrics from the control plane client, moving them into the linkerd-identity crate, as an implementation of Credentials trait (which is updated to include an expiry value). The app::identity module is updated to instrument the credentials store with metrics.

A identity_cert_refresh_timestamp_seconds metric is added to report the time when the identity certificate was last updated.

Breaking changes: identity_cert_refresh_count is renamed to dentity_cert_refreshes_total to follow OpenMetrics best practices, where counters must end with _total.

I've searched GitHub, and the only external references to this metric are in our documentation (which will have to be updated).

---

# HELP identity_cert_expiration_timestamp_seconds Time when the this proxy's current mTLS identity certificate will expire (in seconds since the UNIX epoch).
# TYPE identity_cert_expiration_timestamp_seconds gauge
# UNIT identity_cert_expiration_timestamp_seconds seconds
identity_cert_expiration_timestamp_seconds 1704500155.0
# HELP identity_cert_refresh_timestamp_seconds Time when the this proxy's current mTLS identity certificate were last updated.
# TYPE identity_cert_refresh_timestamp_seconds gauge
# UNIT identity_cert_refresh_timestamp_seconds seconds
identity_cert_refresh_timestamp_seconds 1704413735.7489427
# HELP identity_cert_refreshes The total number of times this proxy's mTLS identity certificate has been refreshed by the Identity provider..
# TYPE identity_cert_refreshes counter
identity_cert_refreshes_total{result="ok"} 1
identity_cert_refreshes_total{result="error"} 0

[codecov]

Codecov Report

Merging #2617 (3f4b7ae) into main (81cbb58) will decrease coverage by 0.04%.
Report is 10 commits behind head on main.
The diff coverage is 91.48%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2617      +/-   ##
==========================================
- Coverage   67.85%   67.82%   -0.04%     
==========================================
  Files         328      328              
  Lines       14980    14994      +14     
==========================================
+ Hits        10164    10169       +5     
- Misses       4816     4825       +9     

Files	Coverage Δ
linkerd/app/src/identity.rs	90.47% <100.00%> (ø)
linkerd/app/src/lib.rs	88.57% <100.00%> (-0.11%)	⬇️
linkerd/identity/src/credentials.rs	0.00% <ø> (ø)
linkerd/identity/src/lib.rs	100.00% <ø> (ø)
linkerd/meshtls/boring/src/creds/store.rs	95.00% <ø> (ø)
linkerd/meshtls/rustls/src/creds/store.rs	92.42% <ø> (ø)
linkerd/meshtls/src/creds.rs	100.00% <100.00%> (ø)
linkerd/meshtls/tests/util.rs	96.83% <100.00%> (+0.06%)	⬆️
linkerd/proxy/identity-client/src/certify.rs	89.18% <100.00%> (-0.43%)	⬇️
linkerd/identity/src/metrics.rs	86.66% <86.66%> (ø)

... and 3 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81cbb58...3f4b7ae. Read the comment docs.

[zaharidichev]

LGTM, just a small comment. Making that a wrapper that implemented Credentials is quite neat