Skip to content

Commit

Permalink
lib/scylla_cloud.py: change retry logic of curl() to exponential backoff
Browse files Browse the repository at this point in the history
Since IaaS services recommended to use exponential backoff logic when
retries to call metadata services, we should do it in our scripts.

Referenced implementation in https://developers.google.com/analytics/devguides/reporting/core/v3/errors?hl=en

see https://docs.aws.amazon.com/general/latest/gr/api-retries.html

Related with scylladb/scylladb#13442
  • Loading branch information
syuu1228 authored and yaronkaikov committed Sep 19, 2023
1 parent ef39bd0 commit 16b454e
Showing 1 changed file with 5 additions and 2 deletions.
7 changes: 5 additions & 2 deletions lib/scylla_cloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import distro
import base64
import datetime
import random
from subprocess import run, CalledProcessError
from abc import ABCMeta, abstractmethod

Expand Down Expand Up @@ -69,8 +70,10 @@ def curl(url, headers=None, method=None, byte=False, timeout=3, max_retries=5, r
return res.read()
else:
return res.read().decode('utf-8')
except (urllib.error.URLError, socket.timeout):
time.sleep(retry_interval)
except (urllib.error.URLError, socket.timeout) as e:
exp_retry_interval = (retry_interval ** retries) + random.random()
logging.error(f'Failed to access {url}: {e.reason}, retrying in {exp_retry_interval}')
time.sleep(exp_retry_interval)
retries += 1
if retries >= max_retries:
raise
Expand Down

0 comments on commit 16b454e

Please sign in to comment.