Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synthesise Events from Condition transitions #2

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

bboreham
Copy link
Collaborator

@bboreham bboreham commented Mar 11, 2021

Many objects have Conditions, like these on a Deployment:

  conditions:
  - lastTransitionTime: "2021-03-01T17:21:28Z"
    lastUpdateTime: "2021-03-01T17:21:28Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-11-24T13:37:06Z"
    lastUpdateTime: "2021-03-10T13:48:40Z"
    message: ReplicaSet "querier-6c88c56bbf" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing

We can create additional spans from the time these conditions are updated. Current code in this PR just creates zero-length spans same as we get with Events, but even better if we could put the start and end time of the span at the points where the transition goes to False then to True.

It works by starting a Watch() on any object we received an Event from, or anything in its owner chain, to see Conditions change over time.
Still to do: drop the watch on objects that haven't done anything in a while.

We adjust timestamps from condition transitions that have arrived promptly, the same as we do for Events, to give a more fine-grained display in tracing.

Note the "Events" that are synthesised are just used as a convenient internal object to turn into spans; they aren't meant to look like real Events. For instance the ObjectMeta is populated with info useful for debugging.

The unit test for this feature builds on the capture/playback mechanism added in #32.

Example, from Cluster-API:
image

This gives a much richer trace output, showing how objects get
initialized.

We have to do a 'watch' on every object to see Conditions change over
time. We watch everything in the owner chain, to pick up changes at
every level.

We adjust timestamps from condition transitions that have arrived
promptly, the same as we do for Events, to give a more fine-grained
display in tracing.

Note the "Events" that are synthesised are just used as a convenient
internal object to turn into spans; they aren't meant to look like real
Events. For instance the ObjectMeta is populated with info useful for
debugging.
Move capturing the initial state to the watcher, since it knows which
objects are new.

(Capture/playback is used for unit tests)
Note it is using captured data from a different date.
@bboreham bboreham changed the title WIP: synthesise Events from Condition transitions Synthesise Events from Condition transitions May 19, 2021
@bboreham bboreham marked this pull request as ready for review May 19, 2021 11:50
@pavolloffay
Copy link
Contributor

Is my understanding correct that this just adds a new "data-source" to create spans from?

@bboreham
Copy link
Collaborator Author

Conceptually, yes.

In some ways it should be better than Events, since we could get a better start/end timestamp. But in practice it seems we don’t always get each update, so we see a condition reset without seeing it set.

It’s unmerged since it never quite worked right.

rajatvig added a commit to rajatvig/kspan that referenced this pull request Dec 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants