Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert panic on auth failure during state storage #178

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Revert panic on auth failure during state storage #178

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 3 additions & 11 deletions statemachine/statemachine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"runtime"
"sync"
"time"
"strings"

"github.com/lytics/metafora"
)

Expand Down Expand Up @@ -371,16 +371,8 @@ func (s *stateMachine) Run() (done bool) {
metafora.Infof("task=%q transitioning %s --> %s --> %s", tid, state, msg, newstate)

// Save state
if err := s.ss.Store(s.task, newstate); err != nil {
// After upgrading to 1.25.5-gke.2000 we started experiencing the metadata server throwing POD_FINDER_IP_MISMATCH
// errors resulting in failures authenticating to spanner. This panic will cause the pod to cyle
// See https://github.com/lytics/lio/issues/30414
if strings.Contains(err.Error(), "spanner: code = \"Unauthenticated\"") {
metafora.Errorf("task=%q Unable to persist state=%q due to failure to authenticate to spanner.", tid, newstate.Code)
panic(err)
}

metafora.Errorf("task=%q Unable to persist state=%q. Continuing.", tid, newstate.Code)
if err := s.ss.Store(s.task, newstate); err != nil {
metafora.Errorf("task=%q Unable to persist state=%q. Unscheduling.", tid, newstate.Code)
return true
}

Expand Down