Initial commit

Adding docker-compose files
Adding logging
Adding get-repo-stats-svn to collect statistics on SVN repos to estimate cloning time
Adding bridge-repo-converter Docker image
Updating README to reflect experimental status

.gitignore

@@ -0,0 +1,166 @@
+# Sourcegraph
+config/service-account-key.json
+config/config.yaml
+docker-template/
+repo-stats-svn/tmp-repo-metadata/
+repo-stats-svn/*.csv
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

README.md

@@ -0,0 +1,40 @@
+# Run the Sourcegraph Cloud Private Connect Agent and src serve-git together in a Docker Compose file
+
+## Experimental - This is not a supported Sourcegraph product
+This repo was created for Sourcegraph Implementation Engineering deployments, and is not intended, designed, built, or supported for use in any other scenario. Feel free to open issues or PRs, but responses are best effort.
+
+## Why
+Running src serve-git and the agent together on the same Docker network allows the agent to use Docker's DNS to reach src serve-git, and prevents src serve-git's unauthenticated HTTP endpoint from needing to be opened outside of the Docker host.
+
+Docker compose also allows for easier upgrades, troubleshooting, monitoring, logging, flexibility of hosting, etc.
+
+## Setup - Sourcegraph Staff Only
+1. Add the entry to the sourcegraphConnect targetGroups list in the Cloud instance's config.yaml, get your PR approved and merged
+```yaml
+        - dnsName: src-serve-git-ubuntu.local
+          listeningAddress: 100.100.100.0
+          name: src-serve-git-ubuntu-local
+          ports:
+          - 80
+        - dnsName: src-serve-git-wsl.local
+          listeningAddress: 100.100.100.1
+          name: src-serve-git-wsl-local
+          ports:
+          - 80
+```
+2. Clone this repo to a customer's bridge VM, install Docker and Docker's Compose plugin
+3. Copy the config.yaml and service-account-key.json files from the Cloud Ops dashboard, and paste them in the files under the config directory
+4. Modify the config.yaml copied from the Cloud Ops dashboard
+ - `  serviceAccountKeyFile: /service-account-key.json` so that the Go binary inside the running Docker container finds this file in the path that's mapped via the docker-compose.yaml file
+ - Only include the `- dialAddress` entries that this cloud agent can reach, remove the others, so the Cloud instance doesn't try connecting to this instance for code hosts it can't reach
+ - Correct open ports from default 443 to 80
+ - Careful when pasting the config.yaml into Windows, because it may add weird line endings or extra spaces, which breaks YAML, as a whitespace-dependent format
+5. Clone the customer's repos into the `repos-to-serve` directory at the root of this repo on the bridge VM, or update the volume mount path for the src-serve-git service in the docker-compose.yaml file
+6. docker compose up -d
+7. Add a Code Host config to the customer's instance
+ - Type: src serve-git
+ - URL: "http://src-serve-git-ubuntu.local:80"
+ - or
+ - URL: "http://src-serve-git-wsl.local:80"
+ - Note that the :80 port is required, as this seems to default to port 443, even when used with http://
+8. Use the bridge-repo-converter to convert SVN, TFVC, or Git repos, to Git format, which will store them in the `repos-to-serve` directory, or use any other means at your disposal to get the repos into the /`repos-to-serve` directory

bridge-host-ubuntu/config/example-config.yaml

@@ -0,0 +1 @@
+# Paste the config.yaml contents from the Cloud Ops dashboard here

bridge-host-ubuntu/config/example-service-account-key.json

@@ -0,0 +1 @@
+// Paste the config.yaml contents from the Cloud Ops dashboard here

bridge-host-ubuntu/docker-compose.yaml

@@ -0,0 +1,26 @@
+version: '2.4'
+
+services:
+
+  cloud-agent:
+    container_name: cloud-agent
+    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
+    volumes:
+      - ./config/service-account-key.json:/service-account-key.json
+      - ./config/config.yaml:/config.yaml
+    command: ["-config=/config.yaml"]
+    restart: always
+    networks:
+      - sourcegraph
+
+  src-serve-git:
+    container_name: src-serve-git-ubuntu.local
+    image:  index.docker.io/sourcegraph/src-cli:latest
+    volumes:
+      - ../repos-to-serve/:/repos-to-serve:ro
+    command: "serve-git -addr :80 /repos-to-serve"
+    networks:
+      - sourcegraph
+
+networks:
+  sourcegraph:

bridge-host-windows/config/example-config.yaml

@@ -0,0 +1 @@
+# Paste the config.yaml contents from the Cloud Ops dashboard here

bridge-host-windows/config/example-service-account-key.json

@@ -0,0 +1 @@
+// Paste the config.yaml contents from the Cloud Ops dashboard here

bridge-host-windows/docker-compose.yaml

@@ -0,0 +1,26 @@
+version: '2.4'
+
+services:
+
+  cloud-agent:
+    container_name: cloud-agent
+    image: index.docker.io/sourcegraph/src-tunnel-agent:2024-02-05-22-28-333320ee
+    volumes:
+      - ./config/service-account-key.json:/service-account-key.json
+      - ./config/config.yaml:/config.yaml
+    command: ["-config=/config.yaml"]
+    restart: always
+    networks:
+      - sourcegraph
+
+  src-serve-git:
+    container_name: src-serve-git-wsl.local
+    image:  index.docker.io/sourcegraph/src-cli:latest
+    volumes:
+      - ../repos-to-serve/:/repos-to-serve:ro
+    command: "serve-git -addr :80 /repos-to-serve"
+    networks:
+      - sourcegraph
+
+networks:
+  sourcegraph:

bridge-host-windows/wsl2-startup.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+# Use a setup like this: https://iknowthatnow.com/2022/04/04/start-wsl2-services-on-windows-startup/
+
+log_file="./log"
+git_exit_status=""
+docker_compose_exit_status=""
+
+echo "Starting $0 $@" >> $log_file
+
+# Git pull latest commits to main
+if ! git pull
+then
+    git_exit_status=$?
+    echo "git pull failed, exit code $git_exit_status" >> $log_file
+    exit $git_exit_status
+fi
+
+# Start docker compose services
+if ! docker compose up -d
+then
+    docker_compose_exit_status=$?
+    echo "docker compose up failed, exit code $docker_compose_exit_status" >> $log_file
+    exit $docker_compose_exit_status
+fi
+
+
+echo "Finishing $0 $@" >> $log_file

bridge-repo-converter/README.md

@@ -0,0 +1,15 @@
+### Building and running your application
+
+When you're ready, start your application by running:
+`docker compose up --build`.
+
+### Deploying your application to the cloud
+
+First, build your image, e.g.: `docker build -t myapp .`.
+
+If your cloud uses a different CPU architecture than your development
+machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
+you'll want to build the image for that platform, e.g.:
+`docker build --platform=linux/amd64 -t myapp .`.
+
+Then, push it to your registry, e.g. `docker push myregistry.com/myapp`.

bridge-repo-converter/build/.dockerignore

@@ -0,0 +1,35 @@
+# Include any files or directories that you don't want to be copied to your
+# container here (e.g., local build artifacts, temporary files, etc.).
+#
+# For more help, visit the .dockerignore file reference guide at
+# https://docs.docker.com/go/build-context-dockerignore/
+
+**/.DS_Store
+**/__pycache__
+**/.venv
+**/.classpath
+**/.dockerignore
+**/.env
+**/.git
+**/.gitignore
+**/.project
+**/.settings
+**/.toolstarget
+**/.vs
+**/.vscode
+**/*.*proj.user
+**/*.dbmdl
+**/*.jfm
+**/bin
+**/charts
+**/docker-compose*
+**/compose*
+**/Dockerfile*
+**/node_modules
+**/npm-debug.log
+**/obj
+**/secrets.dev.yaml
+**/values.dev.yaml
+LICENSE
+README.md
+Dockerfile