Vendir manager hitting github rate limit but VENDIR_GITHUB_API_TOKEN missing from allowedEnv in mend hosted app

[gberche-orange]

What would you like help with?

Would Mend consider white listing this environment variable documented at https://carvel.dev/vendir/docs/v0.41.x/github-release/#github-api-rate-limiting

If your public IP address is shared by multiple machines (e.g. workstations in an office), you may run into Github rate limiting errors. vendir as of v0.8.0 supports providing “Personal access token” to increase Github API rate limits. You can specify it via an environment variable:

$ export VENDIR_GITHUB_API_TOKEN=ghp_8c0a3...
$ vendir sync

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

Using vendir manager to update github artefacts requires heavy github api calls which triggers API rate limit exhaustion with message:

Command failed: vendir sync
vendir: Error: Syncing directory 'k8s-manifests':
  Syncing directory 'github.com/carvel-dev/kapp-controller' with github release contents:
    Downloading release info:
      Downloading release details from https://api.github.com/repos/carvel-dev/kapp-controller/releases/tags/v0.52.0 :
        Expected response status 200, but was '403' (hint: consider setting VENDIR_GITHUB_API_TOKEN env variable to increase API rate limits) (body: '{"message":"API rate limit exceeded for 34.239.12.110. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
')

However, after the following env configuration is rejected with the message Env variable name VENDIR_GITHUB_API_TOKEN is not allowed by this bot's allowedEnv

  env: {
    //See https://developer.mend.io/github/.../-/settings
    // > for free accounts, only available for secrets whose name starts with MEND_
    VENDIR_GITHUB_API_TOKEN: "{{ secrets.MEND_VENDIR_GITHUB_API_TOKEN }}"
  },

Logs (if relevant)

Logs

INFO: Repository has invalid config
{
  "error": {
    "validationSource": "renovate.json5",
    "validationError": "The renovate configuration file contains some invalid settings",
    "validationMessage": "Env variable name `VENDIR_GITHUB_API_TOKEN` is not allowed by this bot's `allowedEnv`.",
    "message": "config-validation",
    "stack": "Error: config-validation\n    at mergeRenovateConfig (/usr/local/renovate/lib/workers/repository/init/merge.ts:198:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at getRepoConfig (/usr/local/renovate/lib/workers/repository/init/config.ts:14:12)\n    at initRepo (/usr/local/renovate/lib/workers/repository/init/index.ts:56:12)\n    at Object.renovateRepository (/usr/local/renovate/lib/workers/repository/index.ts:64:14)\n    at attributes.repository (/usr/local/renovate/lib/workers/global/index.ts:202:11)\n    at start (/usr/local/renovate/lib/workers/global/index.ts:187:7)\n    at /usr/local/renovate/lib/renovate.ts:18:22"
  }
}

[rarkins]

Resolution: #30015

Maybe we can also reuse the same installation token which the app uses for all other github.com communication so that manual configuration is not necessary? e.g. look for a github.com token and export it as VENDIR_GITHUB_API_TOKEN prior to running vendir sync.

[gberche-orange]

Thanks @rarkins for the prompt follow up

Resolution: #30015

Would you have an ETA for this feature ? I'm currently working around the issue with a github workflow triggering the vendor sync with the proper VENDIR_GITHUB_API_TOKEN missing from renovate.

But due to renovate https://docs.renovatebot.com/updating-rebasing/#no-rebasing-if-you-have-made-edits No rebasing if you have made edits, the vendor sync is outcome is commited by the gh workflow into a the target branch, i.e. once the renovate branch is merged. This prevents a proper review and impact analysis of the renovate branch if it had properly included VENDIR_GITHUB_API_TOKEN and hence updated artifacts.

Maybe we can also reuse the same installation token which the app uses for all other github.com communication so that manual configuration is not necessary? e.g. look for a github.com token and export it as VENDIR_GITHUB_API_TOKEN prior to running vendir sync.

The default setting for VENDIR_GITHUB_API_TOKEN sounds great.

[rarkins]

You're welcome to submit a PR contributing either of these suggested solutions