gitRepoUpdates Loop

[wmiller112]

In setting up my first pipelines, I seem to be running into a loop error that I'm not clear how to work around. I've configured a warehouse with image and git subscriptions. The first stage has a gitRepoUpdates promotion mechanism. I then have a number of stages that I'd like to subscribe to that first stage with a argoCDAppUpdates promotion mechanism so they are updated whenever the image gets updated by first stage, OR when manifests change. I havent seen any discussion about it, but I imagine we want the auto sync mechanism in ArgoCD turned off if we want things to be synced via argoCDAppUpdates promotion. If that is the case, and we cant create freight with the image repo and manifests where the image gets updated, how can we auto apply manifest updates?

I track the images in kustomization files in a specific directory, and have the git subscription configured with includePaths that don't include the image directory, but the gitRepoUpdates still error that the commit would cause a loop (hitting this)
Since stages can only subscribe to a single warehouse or multiple stage, it almost seems like I'd need to either

1. split the pipeline, having one that subscribes to a warehouse with image subscription and updates git, and a separate pipeline with all the stages subscribed to a warehouse with git subscription that refreshes and syncs the argo apps? But that cant be right as in that case there doesn't seem to be any point in replacing image updater.
2. Move the image definition kustomizations out to a separate repo

[wmiller112]

This actually falls into #835.

[krancour]

I'm not completely clear on what you're trying to do. I can outline what the common case looks like as well as an alternative that seems it may be close to what you're describing...

Commonly, a Warehouse subscribes to an image repo and a git repo. The former for images, obviously, and the latter for manifests (or something that renders as manifests, like a Helm chart or some manifests + Kustomize overlays). A new piece of Freight is created when either of those subscriptions detects something new.

Since you're using Kustomize, I will focus on that. Typically a Stage subscribed directly to a Warehouse (or indirectly, via upstream Stages) is going to (essentially) check out the desired commit, run kargo edit set image <image-url=desired-tag> in some specified path, take the entire contents of that branch, and completely overwrite the contents of some other, Stage-specific branch. (Which you can think of simply as storage. It may as well be an S3 bucket.)

So imagining your Warehouse found commit X and image Y, those are "mashed together" at each Stage and written to someplace Stage-specific.

This is the common pattern.

Again, I am not entirely sure I've understood what you're trying to do, but it seems you are perhaps shooting for a slightly different pattern that essentially involves two pipelines, with one being very short:

Pipeline 1:

Subscribe only to new images. When one is detected, "promote it" with a Stage that only does the "kustomize edit set image" bit and writes it back to main (for instance) in the git repo (to which it is not subscribed, btw).

Pipeline 2:

Subscribe only to git repo changes. When one is detected, promoted it using whatever promotion mechanisms are desired, but without worrying at all about the kustomize edit set image ... bit, which has already been handled by the other pipeline.

Does this sound like what you're trying to do?

I believe this is already achievable like so:

apiVersion: kargo.akuity.io/v1alpha1
kind: Project
metadata:
  name: example
spec:
  promotionPolicies:
  - stage: image-promoter
    autoPromotionEnabled: true
  - stage: test
    autoPromotionEnabled: true
  # More promotion policies ...
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: image
  namespace: example
spec:
  subscriptions:
  - image:
      repoURL: nginx # Watch for new versions of this image
      semverConstraint: ^1.24.0
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: image-promoter
  namespace: example
spec:
  subscriptions:
    warehouse: image
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/example/examples.git
      readBranch: main
      writeBranch: main # Forms no loop because the Warehouse doesn't also
      kustomize:        # subscribe to this
        images:
        - image: nginx
          path: base # Run kustomize edit set image on this path
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: manifests
  namespace: example
spec:
  subscriptions:
  - git:
      repoURL: https://github.com/example/examples.git
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: test
  namespace: example
spec:
  subscriptions:
    warehouse: manifests
  promotionMechanisms:
    # Whatever you want to do here, such as force a particular Argo CD
    # Application to refresh/sync ...
---
# More Stages downstream from test ...

The one caveat to this is that the UI isn't great about Projects containing multiple Warehouses yet, although we hope to have that addressed in the upcoming v0.7.0 release. If that is a problem, a possible workaround is creating these two pipelines in separate Projects.

Please let me know if this helps at all or if I have completely missed the mark.

[wmiller112]

Sorry for delay. Yes, your example makes sense. I had ended up with the two pipelines as described in two separate projects

[krancour]

@wmiller112 I'm glad to hear that worked. If it matters, you can have both pipelines in the same Project. There's never been a limitation of 1:1 on the back end, although at one time, the UI wasn't good at displaying more than one. That changed in, I think v0.7.0, and it's set to improve further in v0.8.0, which we hope to release this Friday.

[wmiller112]

So I spent today playing a bit more with this, taking into consideration your "common" use case example and its left me with a follow up question that I'm happy to submit a different issue for, but might be easier answered with the context here. I've set up a pipeline with:
warehouse that has 2 sources, image and git
stage (call it render) that has a gitRepoUpdates promotion mechanism with a kustomize image update and a writeBranch that is different from the readBranch
stage (call it apply) with argoCDAppUpdates and an array of app name/namespace. each of the argo Applications has the necessary authorized-stage annotation, and their source.branch is the branch of the render stage's writeBranch. Now, if I add a subscription.upstreamStages: - name: render to this second stage, the promotions generated seem to get stuck "Running", and i get the following error for the applications:

error listing Promotions for Application" app=<app_name> error="cache contained <nil>, which is not an Object" namespace=argocd

However, if I take the other approach that we've been discussing and add a second warehouse with just a git source that uses the repo/writeBack branch defined in the render stage from above, and instead subscribe the apply stage to this second warehouse, the applications are refreshed and synced successfully.

I'd expect them to do the same thing, since the subscription is just the triggering mechanism that creates the promotion of the freight into the stage. So in the first case, the completion of the render stage committing it's updates to git trigger the apply. Is there some requirement that the Argocd Application source repo/branch be the same as the warehouse's git source repo/readBranch?

[krancour]

That error is odd. It occurs when the controller notices a change to an Application's status and attempts to find all running Promotions related to that Application so they can be enqueued for reconciliation. This is how, for instance, a running Promotion might promptly discover that it is now complete.

I'm not sure why you'd be getting that, but before diving too deep down that rabbit hole...

What you're trying to do sounds a bit unconventional. With Stage names like "render" and "apply," It sounds almost as if you are viewing Stages as "steps" in a process and that's not really what they are. They are more similar to "environments," which is the word we used initially (instead of Stages) and abandoned for reasons not worth getting into...

So what the promotion mechanisms in each Stage are really intended to do is define a process for taking a set of artifacts (a piece of Freight) and making them run in that Stage (environment).

You and I discussed a particular pattern where you might have two parallel pipelines. The first of the two is very short and the Warehouse finds images and manifests, a single Stage mashes them together and writes them to somewhere. The second pipeline picks up where the first left off, creating new Freight that will contain a single artifact -- the manifests with the latest image already mashed into them. This is a pattern some people may opt for because they want to do all that "mashing up" of artifacts once up front instead of mashing manifests and and images together at each stage. It's a choice that might sometimes be beneficial.

What you attempted today might seem as if it is equivalent to the pattern we discussed, but it isn't...

With what you have just attempted, your Warehouse is creating pieces of Freight that have two artifacts in them -- manifests and an image. Your "render" Stage has promotion mechanisms that mash those things together and write them to a different branch of the same repo you got the manifests from. Now as that Freight moves on to the second Stage -- "apply" -- keep in mind that it still references the same artifacts -- manifests at a particular commit and image at a particular tag -- that the Warehouse originally produced and was used by your "render" Stage. What is not in that piece of Freight is any reference to the "pre-mashed" manifests that were created during the promotion to your "render" Stage.

In short, a single pipeline should be working with the same artifacts from one end all the way to the other. You basically repackaged your artifacts mid-pipeline.

I understand what you were trying to do and why it might have appeared to be equivalent to the pattern we discussed. And in fact, we anticipate that we eventually will be adding explicit support for re-packaging artifacts mid-pipeline because we want to address use cases like parallel pipelines for the different microservices of an app and have them be completely independent of one another up until, let's say "staging" or something, where one might put a "junction" of sorts to combine the artifacts from multiple pieces of Freight into a new, single, larger uber-Freight that can move the last mile of the pipeline into prod as an atomic unit...

But this sort of mid-pipeline repackaging you're doing just doesn't work yet.

[wmiller112]

Thank you for the very quick, thorough explanation. I do understand its not the intended use of stages, I gave some details about my use case in the discussion about dynamic upstream stages, and why I'm trying to implement it in this way for the time being, but this limitation is clear now. It is again awesome to hear that this is already something that's been considered as the project matures, even if not specifically intended for this case.