Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to Backup Private repos - Github Actions #228

Open
gthomson31 opened this issue Apr 15, 2024 · 10 comments
Open

Failing to Backup Private repos - Github Actions #228

gthomson31 opened this issue Apr 15, 2024 · 10 comments

Comments

@gthomson31
Copy link

Hi there

Wondering if anyone has seen a similar issue we are currently deploying the gickup backups to run via a Github actions workflow replicating the same process as running locally.

When running locally all repos both public and private backup without issue but when running within the actions workflow it is unable to clone private repos using the same token.

Local Run

2024-04-15T10:54:37+01:00 INF starting backup for https://github.com/<REDACTED>/TESTING.git stage=backup
2024-04-15T10:54:37+01:00 INF cloning TESTING path=REDACTED stage=locally
2024-04-15T10:54:39+01:00 INF zipping TESTING path=REDACTED stage=locally
2024-04-15T10:54:39+01:00 INF Backup run complete duration=30.568598333s

Github Actions Log

2024-04-15T09:53:43Z INF starting backup for https://github.com/<ORGNAME>>/TESTING.git stage=backup
2024-04-15T09:53:43Z INF cloning TESTING path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:43Z WRN retry 1 from 5 path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:48Z INF cloning TESTING path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:48Z WRN retry 2 from 5 path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:53Z INF cloning TESTING path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:53Z WRN retry 3 from 5 path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:[58](https://github.com/<WORKFLOW_ORG_NAME>/it-services-testing/actions/runs/8687070876/job/23819854528#step:10:59)Z INF cloning TESTING path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:53:[59](https://github.com/<WORKFLOW_ORG_NAME>/it-services-testing/actions/runs/8687070876/job/23819854528#step:10:60)Z WRN retry 4 from 5 path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:54:04Z INF cloning TESTING path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 stage=locally
2024-04-15T09:54:04Z WRN exit status 128 path=/home/runner/work/<REDACTED>/<ORGNAME>-2024-04-15-09-53-21 repo=TESTING stage=locally
@cooperspencer
Copy link
Owner

Hi,
I am not that well versed in Github Actions but did you set the permissions so that the container can create files? Something like:

permissions:
  contents: write

That's the only thing that comes to mind right now.

@gthomson31
Copy link
Author

gthomson31 commented Apr 15, 2024

Just tested out but unfortunately the same issue

The setting below refers to the default GITHUB_TOKEN that is set when deploying the workflow - I am pulling a repo specific token out from Github secrets and patching this into the config. ( Tried to also unset this token during workflow as a precaution)

permissions:
  contents: write 

I believe the issue stems from a conflict between the GitHub runner's default image and the gickup configuration file. because the token is correctly configured, evident from its ability to fetch public/private repositories without any problems.

Will do some investigation

@gthomson31
Copy link
Author

Dropping our config

source:
  github:
    - token: "TOKEN_IS_PATCHED"
      wiki: true # includes wiki too
      issues: true # back up issues, works only locally
      filter:
        excludeforks: true # exclude forked repositories
destination:
  local:
    # Export this path from Docker with a volume to make it accessible and more permanent.
    - path: "PATH_IS_PATCHED"
      structured: false # checks repos out like hostersite/user|organization/repo
      zip: true # zips the repository after cloned and removes the repository afterwards
      lfs: true # clone lfs repos, !! ATTENTION !! git and git-lfs must be installed on the system!

@cooperspencer
Copy link
Owner

Would you mind sharing the relevant code for the action?

@gthomson31
Copy link
Author

Added the workflow and breakdown below

Workflow Summary: Deploy and Run GitHub Backups

Workflow Name

  • Deploy and Run GitHub Backups

Trigger Conditions

  • Manual Trigger: workflow_dispatch
  • Automatic Trigger: On push events to the main branch

Permissions

  • id-token: write: Necessary for JWT requests
  • contents: read: Needed for the actions/checkout to access repository contents

Jobs in Workflow

1. Setup Tools

  • Environment: Runs on ubuntu-latest
  • Outputs: Current date from a specific step
  • Environment Variables: GICKUP_VERSION set to 0.10.28
  • Steps:
    • Checkout Repository Code: Uses actions/checkout@v4
    • Cache Gickup Tool: Caches the Gickup tool using its version
    • Download and Extract Gickup (if not cached): Downloads and extracts Gickup, setting permissions
    • Get Current Date: Retrieves and sets the current date

2. Create Backup and Upload

  • Dependencies: Requires completion of setup-tools job
  • Environment: Runs on ubuntu-latest
  • Strategy: Uses a matrix for different organizations, e.g., example-org
  • Environment Variables: Specifies path to backup configuration file for each organization
  • Steps:
    • Restore Gickup from Cache: Uses cached Gickup
    • Checkout Repository Code: Uses actions/checkout@v4
    • Add Gickup to PATH: Adds Gickup directory to the system path
    • Configure Secrets and Paths: (Details redacted) for the backup
    • Run Gickup Backup: Executes backup using the specified configuration file
name: Deploy and Run Github Backups

on:
  workflow_dispatch:
  push:
    branches:
      - main  

permissions:
  id-token: write   # This is required for requesting the JWT
  contents: read    # This is required for actions/checkout

jobs:
  setup-tools:
    runs-on: ubuntu-latest
    outputs:
      date: ${{ steps.date.outputs.RUN_DATE }}
    env:
      GICKUP_VERSION: 0.10.28

    steps:

      - name: Checkout code
        uses: actions/checkout@v4

      # Install Gickup and cache it
      - name: Cache Gickup
        id: cache-gickup
        uses: actions/cache@v4
        with:
          path: ~/gickup
          key: gickup-${{ env.GICKUP_VERSION }}

      - name: Download and Extract Gickup if not cached
        if: steps.cache-gickup.outputs.cache-hit != 'true'
        run: |
          mkdir -p ~/gickup
          curl -L https://github.com/cooperspencer/gickup/releases/download/v${{ env.GICKUP_VERSION }}/gickup_${{ env.GICKUP_VERSION }}_linux_amd64.tar.gz -o ~/gickup/gickup.tar.gz
          tar -xzvf ~/gickup/gickup.tar.gz -C ~/gickup
          chmod +x ~/gickup/gickup

      - name: Get Current date
        id: date
        run: echo "RUN_DATE=$(date +'%Y-%m-%d-%H-%M')" >> $GITHUB_OUTPUT

  create-backup-and-upload:
    needs: setup-tools
    name: Backup and Upload for ${{ matrix.org }}
    runs-on: ubuntu-latest
    strategy:
    # This will create a job for each org in the matrix
      matrix:
        org:
          - example-org
    env:
      # This is the path to the backup config file for the org
      BACKUP_CONFIG_FILE: .github/config/github_backups/configs/${{ matrix.org }}.yml

    steps:
      - name: Restore Gickup Cache
        id: cache-gickup
        uses: actions/cache@v4
        with:
          path: ~/gickup
          key: gickup-${{ env.GICKUP_VERSION }}

      - name: Checkout code
        uses: actions/checkout@v4

      # Add Gickup to PATH to make it available
      - name: Add Gickup to PATH
        run: echo "$HOME/gickup" >> $GITHUB_PATH

      # REDACTED Setting the secret path for the org name and Patching the config file with token and path

      # Run the backup
      - name: Run Gickup Backup
        working-directory:
        run: |
          gickup ${{env.BACKUP_CONFIG_FILE}}

@gthomson31
Copy link
Author

gthomson31 commented Apr 15, 2024

Testing running Github API call directly in the workflow against a private repo, passing in the same token used in Gickup Config

Workflow

      - name: "Download using API"
        run: |
          echo "Downloading the repository"
          curl -L \
          -H "Accept: application/vnd.github+json" \
          -H "Authorization: Bearer ${{ secrets[env.GH_BACKUP_TOKEN] }} " \
          -H "X-GitHub-Api-Version: 2022-11-28" \
          https://api.github.com/repos/${{matrix.org}}/testing-private/zipball/main > testing-private.zip
          
          
       - name: "Unzip the repository and list the files"
        run: |
          echo "Unzipping the repository"
          unzip testing-private.zip -d testing-private
          folder=$(ls testing-private | grep <ORG_NAME>)
          ls testing-private/$folder

Workflow Log

Run echo "Unzipping the repository"
Unzipping the repository
Archive:  testing-private.zip
<REDACTED LOGS>
README.md
TEST_FILE_GICKUP

@gthomson31
Copy link
Author

gthomson31 commented Apr 15, 2024

Connecting into the github runner ubuntu-latest Image to run the gickup commands manually and see the log and have found the error below:

Token not being picked up by default

When running through it prompts for Github Username instead of passing in the token.

Using v0.10.28 - linux_amd64

2024-04-15T14:04:28Z INF starting backup for https://github.com/<org_name>/testing-private.git stage=backup
2024-04-15T14:04:28Z INF cloning testing-private path=<org_name>-2024-04-15-14-00-17 stage=locally
Username for 'https://github.com':

@cooperspencer
Copy link
Owner

cooperspencer commented Apr 15, 2024

hm... If you use git lfs it defaults to the git command and doesn't use the go-git module.
It seems like something goes wrong in this part:
https://github.com/cooperspencer/gickup/blob/main/local/local.go#L402

It seems like it doesn't add the token to the URL.

@gthomson31
Copy link
Author

How strange !

I don't actually think we needed to use LFS so have disabled from our config and works fine now

@cooperspencer
Copy link
Owner

I am glad to hear that it worked out. I'll try to investigate what's going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants