Add preliminary authenticated pip repo functionality

.devcontainer/devcontainer.json

@@ -0,0 +1,24 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/python
+{
+	"name": "Python 3",
+	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
+	"image": "mcr.microsoft.com/devcontainers/python:1-3.11-bullseye"
+
+	// Features to add to the dev container. More info: https://containers.dev/features.
+	"features": {
+		"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}
+	},
+
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+
+	// Use 'postCreateCommand' to run commands after the container is created.
+	// "postCreateCommand": "pip3 install --user -r requirements.txt",
+
+	// Configure tool-specific properties.
+	// "customizations": {},
+
+	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+	// "remoteUser": "root"
+}

.gitattributes

@@ -1 +1,8 @@
 repo2docker/_version.py export-subst
+
+* text=auto eol=lf
+
+*.sh    text eol=lf
+*.bash    text eol=lf
+*.py    text eol=lf
+*.yaml  text eol=lf

.readthedocs.yaml

@@ -1,19 +1,19 @@
-# Configuration on how ReadTheDocs (RTD) builds our documentation
-# ref: https://readthedocs.org/projects/repo2docker-service/
-# ref: https://docs.readthedocs.io/en/stable/config-file/v2.html
-#
-version: 2
-
-sphinx:
-  configuration: docs/source/conf.py
-
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.10"
-
-python:
-  install:
-    - requirements: docs/requirements.txt
-    - method: pip
-      path: .
+# Configuration on how ReadTheDocs (RTD) builds our documentation
+# ref: https://readthedocs.org/projects/repo2docker-service/
+# ref: https://docs.readthedocs.io/en/stable/config-file/v2.html
+#
+version: 2
+
+sphinx:
+  configuration: docs/source/conf.py
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+python:
+  install:
+    - requirements: docs/requirements.txt
+    - method: pip
+      path: .

Dockerfile

@@ -15,8 +15,8 @@ RUN mkdir /tmp/wheelhouse \
 
 FROM alpine:${ALPINE_VERSION}
 
-# install python, git, bash, mercurial
-RUN apk add --no-cache git git-lfs python3 py3-pip py3-setuptools bash docker mercurial
+# install python, git, bash, mercurial, ssh
+RUN apk add --no-cache git git-lfs python3 py3-pip py3-setuptools bash docker mercurial openssh
 
 # install hg-evolve (Mercurial extensions)
 RUN pip3 install hg-evolve --user --no-cache-dir

README.md

@@ -1,42 +1,22 @@
-# <a href="https://github.com/jupyterhub/repo2docker"><img src="https://raw.githubusercontent.com/jupyterhub/repo2docker/8731ecf0967cc5fde028c456f2b92be651ebbc18/docs/source/_static/images/repo2docker.png" height="48px" /> repo2docker</a>
-
-[![Build Status](https://github.com/jupyterhub/repo2docker/workflows/Test/badge.svg)](https://github.com/jupyterhub/repo2docker/actions)
-[![Documentation Status](https://readthedocs.org/projects/repo2docker/badge/?version=latest)](http://repo2docker.readthedocs.io/en/latest/?badge=latest)
-[![Contribute](https://img.shields.io/badge/I_want_to_contribute!-grey?logo=jupyter)](https://repo2docker.readthedocs.io/en/latest/contributing/contributing.html)
-[![Docker Repository on Quay](https://img.shields.io/badge/quay.io-container-green "Docker Repository on Quay")](https://quay.io/repository/jupyterhub/repo2docker?tab=tags)
+# <a href="https://github.com/jupyterhub/repo2docker"><img src="https://raw.githubusercontent.com/jupyterhub/repo2docker/8731ecf0967cc5fde028c456f2b92be651ebbc18/docs/source/_static/images/repo2docker.png" height="48px" /> repo2docker</a> - with "enterprise" scenario bells-and-whistles
 
 `repo2docker` fetches a git repository and builds a container image based on
 the configuration files found in the repository.
 
 See the [repo2docker documentation](http://repo2docker.readthedocs.io)
 for more information on using repo2docker.
 
-For support questions please search or post to https://discourse.jupyter.org/c/binder.
-
-See the [contributing guide](CONTRIBUTING.md) for information on contributing to
-repo2docker.
-
----
-
-Please note that this repository is participating in a study into sustainability
-of open source projects. Data will be gathered about this repository for
-approximately the next 12 months, starting from 2021-06-11.
-
-Data collected will include number of contributors, number of PRs, time taken to
-close/merge these PRs, and issues closed.
-
-For more information, please visit
-[our informational page](https://sustainable-open-science-and-software.github.io/) or download our [participant information sheet](https://sustainable-open-science-and-software.github.io/assets/PIS_sustainable_software.pdf).
+This fork adds various bells-and-whistles to make `repo2docker` more usable in an enterprise context.  Presently this includes:
 
----
+- Support for authenticating against Azure DevOps hosted `pip` repositories using an Azure Service Principal
 
 ## Using repo2docker
 
 ### Prerequisites
 
 1. Docker to build & run the repositories. The [community edition](https://store.docker.com/search?type=edition&offering=community)
    is recommended.
-2. Python 3.6+.
+2. Python 3.10+ (although 3.6+ may work).
 
 Supported on Linux and macOS. [See documentation note about Windows support.](http://repo2docker.readthedocs.io/en/latest/install.html#note-about-windows-support)
 
@@ -99,4 +79,4 @@ The philosophy of repo2docker is inspired by
 
 ## Docker Image
 
-Repo2Docker can be run inside a Docker container if access to the Docker Daemon is provided, for example see [BinderHub](https://github.com/jupyterhub/binderhub). Docker images are [published to quay.io](https://quay.io/repository/jupyterhub/repo2docker?tab=tags). The old [Docker Hub image](https://hub.docker.com/r/jupyter/repo2docker) is no longer supported.
+Repo2Docker can be run inside a Docker container if access to the Docker Daemon is provided, for example see [BinderHub](https://github.com/jupyterhub/binderhub).

repo2docker/__main__.py

@@ -68,7 +68,8 @@ def __call__(self, parser, namespace, values, option_string=None):
 def get_argparser():
     """Get arguments that may be used by repo2docker"""
     argparser = argparse.ArgumentParser(
-        description="Fetch a repository and build a container image"
+        description="Fetch a repository and build a container image",
+        formatter_class=argparse.RawTextHelpFormatter
     )
 
     argparser.add_argument(
@@ -282,6 +283,34 @@ def get_argparser():
         help=Repo2Docker.engine.help,
     )
 
+    argparser.add_argument(
+        "--pip-index-url",
+        type=str,
+        default="https://pipy.python.org/simple",
+        help=Repo2Docker.pip_index_url.help,
+    )
+
+    argparser.add_argument(
+        "--pip-auth",
+        type=str,
+        default="none",
+        help=Repo2Docker.pip_auth.help,
+    )
+
+    argparser.add_argument(
+        "--pip-identity",
+        type=str,
+        default="",
+        help=Repo2Docker.pip_identity.help,
+    )
+
+    argparser.add_argument(
+        "--pip-secret",
+        type=str,
+        default="",
+        help=Repo2Docker.pip_secret.help,
+    )
+
     return argparser
 
 
@@ -464,6 +493,18 @@ def make_r2d(argv=None):
     if args.target_repo_dir:
         r2d.target_repo_dir = args.target_repo_dir
 
+    if args.pip_index_url:
+        r2d.pip_index_url = args.pip_index_url
+
+    if args.pip_auth:
+        r2d.pip_auth = args.pip_auth
+
+    if args.pip_identity:
+        r2d.pip_identity = args.pip_identity
+
+    if args.pip_secret:
+        r2d.pip_secret = args.pip_secret
+
     return r2d
 
 

repo2docker/app.py

@@ -16,12 +16,12 @@
 import tempfile
 import time
 import warnings
-from urllib.parse import urlparse
+from urllib.parse import urlparse, urlunparse, quote
 
 import entrypoints
 import escapism
 from pythonjsonlogger import jsonlogger
-from traitlets import Any, Bool, Dict, Int, List, Unicode, default, observe
+from traitlets import Any, Bool, Dict, Int, List, Unicode, CaselessStrEnum, default, observe
 from traitlets.config import Application
 
 from . import __version__, contentproviders
@@ -462,6 +462,60 @@ def _dry_run_changed(self, change):
         """,
     )
 
+    pip_index_url = Unicode(
+        "https://pypi.python.org/simple",
+        config=True,
+        help="""
+        Sets the pip index URL.
+        """
+    )
+
+    pip_auth = CaselessStrEnum(
+        [
+            "none",
+            "basic",
+            "azure-sp-key",
+            "azure-sp-certificate"
+        ],
+        "none",
+        config=True,
+        help="""
+        pip authentication mechanism.
+
+        - Generic options
+
+          - "none" (default), authentication is disabled.
+          - "basic" uses simple HTTP username+password authentication.
+            Provide pip_identity for username, pip_secret for password.
+
+        - For Azure DevOps, a service principal must be defined
+          by --pip-identity in the format "<tenant_id>/<client_id>"
+          (delimited by a forward-slash)
+
+          - "azure-sp-key" requests a temporary authentication token
+            from Azure using a Service Principal key (password).
+            Provide pip_secret for the Service Principal key.
+          - "azure-sp-certificate" requests a temporary authentication
+            token from Azure using a Service Principal certificate. 
+        """
+    )
+
+    pip_identity = Unicode(
+        "",
+        config=True,
+        help="""
+        See pip_auth mechanisms.
+        """
+    )
+
+    pip_secret = Unicode(
+        "",
+        config=True,
+        help="""
+        See pip_auth mechanisms.
+        """
+    )
+
     def get_engine(self):
         """Return an instance of the container engine.
 
@@ -758,6 +812,58 @@ def find_image(self):
                     return True
         return False
 
+    def _get_authenticated_pip_index_url(self):
+        url = urlparse(self.pip_index_url)
+
+        auth = str(self.pip_auth).lower()
+
+        # we perform authentication exchange if required even in dry-mode,
+        # so the user can validate that it worked.
+        if auth == "basic":
+            username = quote(self.pip_identity)
+            password = quote(self.pip_secret)
+            url = url._replace(netloc="{}:{}@{}".format(username, password, url.hostname))
+
+        elif auth == "azure-sp-key" or auth == "azure-sp-certificate":
+            from azure.identity import CertificateCredential, ClientSecretCredential
+            from azure.core.exceptions import ClientAuthenticationError
+
+            try:
+                tenant_id, client_id = str(self.pip_identity).split('/', 1)
+            except ValueError as e:
+                self.log.error(e)
+                self.log.error("\nAzure SP authentication for pip authentication specified " +
+                               "but identity was not in the form '<tenant_id>/<client_id>' - " +
+                               "unable to proceed\n")
+                self.exit(1)
+
+            try:
+                if auth == "azure-sp-key":
+                    credential = ClientSecretCredential(
+                        tenant_id=tenant_id,
+                        client_id=client_id,
+                        client_secret=self.pip_secret
+                    )
+                else:
+                    credential = CertificateCredential(
+                        tenant_id=tenant_id,
+                        client_id=client_id,
+                        certificate_data=str(self.pip_secret).encode()
+                    )
+            except ClientAuthenticationError as e:
+                self.log.error("\nAzure SP authentication for pip authentication specified " +
+                               f"but authentication failed: {e}\n")
+                self.exit(1)
+            except Exception as e:
+                self.log.error(f"\nThere was an unexpected issue processing the Azure SP authentication: {e}\n")
+                self.exit(1)
+
+            # 499b84ac-1321-427f-aa17-267ca6975798/.default   (universally ADO)
+            token, _ = credential.get_token("499b84ac-1321-427f-aa17-267ca6975798/.default")
+            url = url._replace(netloc="{}:{}@{}".format("", token, url.hostname))
+
+        return urlunparse(url)
+
     def build(self):
         """
         Build docker image
@@ -835,6 +941,9 @@ def build(self):
                     build_args["REPO_DIR"] = self.target_repo_dir
                 build_args.update(self.extra_build_args)
 
+                # Handle pip authentication
+                build_args["PIP_INDEX_URL"] = self._get_authenticated_pip_index_url()
+
                 if self.dry_run:
                     print(picked_buildpack.render(build_args))
                 else:

repo2docker/buildpacks/base.py

@@ -87,6 +87,9 @@
 {% endfor -%}
 {% endif -%}
 
+# Build arguments for custom/authenticated software repositories
+ARG PIP_INDEX_URL="https://pipy.python.org/simple"
+
 {% if path -%}
 # Special case PATH
 ENV PATH={{ ':'.join(path) }}:${PATH}

setup.py

@@ -61,6 +61,7 @@ def get_identifier(json):
         "semver",
         "toml",
         "traitlets",
+        "azure-identity",
     ],
     python_requires=">=3.6",
     author="Project Jupyter Contributors",