fix(retry): Avoid panicking if responses come early

fix(retry): Avoid panicking if responses come early #644