Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

genai: add system_fingerprint attribute #1355

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

xrmx
Copy link

@xrmx xrmx commented Aug 20, 2024

Changes

Add system_fingerprint attribute to genai to track the value from the OpenAI api response as requested in https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2759/files/6383978266d85eb058fd8030e8da2b83e6d90043#r1710313834

Merge requirement checklist

@xrmx xrmx marked this pull request as ready for review August 20, 2024 08:27
@xrmx xrmx requested review from a team August 20, 2024 08:27
- id: gen_ai.system_fingerprint
stability: experimental
type: string
brief: A fingerprint to track any eventual change in the Generative AI environment.
Copy link
Author

@xrmx xrmx Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this description, for reference the instrumentation that requires this attribute used System fingerprint of the system that generated the response

Copy link
Member

@drewby drewby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have two questions:

  1. Can we confirm that this is returned from more vendors/models than OpenAI?
  2. How often does system_fingerprint change? If its low cardinality enough, I'd suggest adding this to the metric as well as a good way to correlate between fingerprint changes and degradations/changes in app performance.

@xrmx
Copy link
Author

xrmx commented Aug 22, 2024

I have two questions:

1. Can we confirm that this is returned from more vendors/models than OpenAI?

I think it's mostly OpenAI specific, not found something similar in Mistral or Anthropic APIs.

2. How often does system_fingerprint change? If its low cardinality enough, I'd suggest adding this to the metric as well as a good way to correlate between fingerprint changes and degradations/changes in app performance.

I don't know

@drewby
Copy link
Member

drewby commented Aug 23, 2024

I have two questions:

1. Can we confirm that this is returned from more vendors/models than OpenAI?

I think it's mostly OpenAI specific, not found something similar in Mistral or Anthropic APIs.

2. How often does system_fingerprint change? If its low cardinality enough, I'd suggest adding this to the metric as well as a good way to correlate between fingerprint changes and degradations/changes in app performance.

I don't know

I have two questions:

1. Can we confirm that this is returned from more vendors/models than OpenAI?

I think it's mostly OpenAI specific, not found something similar in Mistral or Anthropic APIs.

2. How often does system_fingerprint change? If its low cardinality enough, I'd suggest adding this to the metric as well as a good way to correlate between fingerprint changes and degradations/changes in app performance.

I don't know

Can you add this to the metric

I have two questions:

1. Can we confirm that this is returned from more vendors/models than OpenAI?

I think it's mostly OpenAI specific, not found something similar in Mistral or Anthropic APIs.

2. How often does system_fingerprint change? If its low cardinality enough, I'd suggest adding this to the metric as well as a good way to correlate between fingerprint changes and degradations/changes in app performance.

I don't know

I think its worth adding this to the base set for metrics as Conditionally Required, If available : https://github.com/open-telemetry/semantic-conventions/blob/main/model/metrics/gen-ai.yaml

If it exists in Mistral and Anthropic, then it makes sense to keep in in the generic namespace vs a vendor namespace.

@xrmx xrmx marked this pull request as draft August 26, 2024 08:21
Copy link
Contributor

@lmolkova lmolkova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for following up and adding this @xrmx!

Given that it seems to be OpenAI specific, it'd need to be gen_ai.openai.response.system_fingerprint (or similar) attribute.

But before we can add it, we'd need to add the whole openai specific conventions doc - I created #1370 to track it and it's up for grabs for anyone interested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants