[Feature Contribution]: Initial DevOps and github actions clean up

[danuw]

What happened?

This is both an issue and a new feature.
It aims to address duplication in Github actions as well as bring consistency of approach.

First the audit with initial thoughts:

• build-cli-dotnet: build everything and run all tests - remove if duplicated
• build-packages: major functionality as library (Cli + webapi) - package integration test to ensure the library is working under happy path conditions => maybe publically github package as Beta
• build-webapi: build test validate and compare => generated swagger changes should be part of the repo/tracked so obvious as part of the PR
• codeQL-anamlysis: scheduled - security check - static analysis to check for vulnerabilities for updated dependencies
• dev_carbon-aware-api: build test deploy DEV to public api endpoint
• run-sdkCLI-githubaction: CI/CD carbon aware github action => use for CodeQL ACTION
• verify-azure-function-with-packages: function validate the packages / solution deploys correctly and can I request the endpoint successfully => merge with others (e.g. build-packages)

Initial recommendation of first changes to address:

• Rename to more meaningful actions and merge actions where needed
• Separate into the following themes (main build actions):
• - PR build and integration tests
  • do we separate per medium (cli vs webAPI)?
  • do we have separate build for each sample or only some?
• - build (& deploy?) on PR merge
• - Validate dependencies and integration tests (cron - can we use the CI/CD pipeline for this?)

Also (further topics to cover in github actions):

• stale issues: implement git action to auto label and close old tickets (older than X days where X needs to be confirmed - may start with 60 or 90 days minimum and reduce overtime)
• create releases?
• highlight Swagger updates in release notes
• I am also keen to version the builds and make sure it surfaces on the swagger as a relevant build version so people know if things have changed or that they are simply looking at the right thing (hoping that can be used in the analytics)

Thank you @bderusha for your inputs
cc @vaughanknight @Willmish

Code of Conduct

• I agree to follow this project's Code of Conduct

Feature Commitment

• I commit to contributing this feature as a PR and working with the GSF to merge this feature into the Carbon Aware SDK.

[danuw]

To be discussed on Tuesday morning...

[YaSuenag]

I hope following topics are implemented on GHA - they are accepted in ADR-0011:

• Ship WebAPI container images from GitHub packages
• Ship client libraries which are supported by GitHub packages

I think they should be kicked on release-created event on GHA.

[danuw]

@YaSuenag , worth linking #149 here

[YaSuenag]

@danuw I think it is better to upload OpenAPI YAML because SDK user can implement their clients without WebAPI container.

In addition, it is also better to upload documents of client bindings. In case of Java, we can generate Javadoc (API document) from source code comments, and we can integrate generating process into build process.

In my personal Java project, I use GHA to generate both release binary and API documents, then they will be uploaded to GitHub packages and GitHub pages. I'm happy if it helps you. https://github.com/YaSuenag/ffmasm/blob/main/.github/workflows/release.yml#L26-L38

[danuw]

Storing for work in progress diagrams...

How to use?

How to use in standalone solutions

flowchart TD
    %% How to use in standalone solutions
    StandaloneProject[Standalone Dotnet project \n  - eg Function] --> |Included as nuget package| SDK[Carbon Aware SDK \n as .net library]
    SDK --> |built for linux| github[Github packages]
    SDK --> |built for arm64| github[Github packages]
    SDK --> |built for amd64| github[Github packages]

Loading

How to use with Hosted APIs

flowchart TD
    %% How to use with Hosted APIs
    %% Hosting options and methods of deployment
    Project -->|connects via REST| API[Carbon Aware API]
    Project --> |uses| RestClient{Carbon Aware \n OpenAPI Generated \n REST Clients}
    RestClient --> |?| java[JAVA client]
    RestClient --> |NPM| node[node client]
    RestClient --> |Pypi| python[python client]
    RestClient --> |?| rust[rust client]
    RestClient --> |?| go[go client]

Loading

Used in processes

flowchart TD
   %% Used in processes
    Pipeline --> |Command line| CLI

Loading

Hosting options and methods of deployment

flowchart TD
    %% -----------------------------------------
    %% Hosting options and methods of deployment
    CAAPI[Hosted API] --> GSF[GSF Hosted]
    CAAPI --> Self[Self Hosted]
    Self --> Cloud
    Self --> Local
    Cloud --> Container[Container]
    Cloud --> WebApp[Web App]
    Local --> Container2[Container]
    Local --> WebApp2[Web App]

Loading

TODO

• Update to reflect what the GSF may cover and status (planned, in progress, done, not planned)
• Other DevOps processes missing (generic for now)

[YaSuenag]

Thank @danuw for sharing current progress! I have some comments:

• How to use in standalone solutions
  • Can we use "AnyCPU" for .Net library? I think it is prefer to single binary if we can.
  • Can we add workflow to publish WebAPI container to GitHub container registry? It relates to ADR-0011.
• How to use with Hosted APIs
  • Client library for Java conforms with Maven.
  • Would they be uploaded to GitHub packages? They are mentioned in ADR-0011, and Java / JS libraries should be uploaded at least.

[danuw]

Hi @YaSuenag , the above is just work in progress. I think the idea was to help visually explain what the repo enables and what we may or may not cover (that part is not yes illustrated on those graphs)
These are mainly to support and summarise what we are aiming for and will need to document further.

Anyway below to answer your questions.

Can we use "AnyCPU" for .Net library? I think it is prefer to single binary if we can.

yes (I assume this also relates to stated ADR right?)

Can we add workflow to publish WebAPI container to GitHub container registry? It relates to ADR-0011.

yes, do you have anything (documented steps) to test, so we use as first success criteria and covered scope please?

How to use with Hosted APIs

I believe this would not necessarily cover at first but rather document further how to generate if needed (such as through videos etc...). Do you believe more documentation is needed and if yes where shall we start? - we can probably discuss this on Tuesday

Hope above helps... Let me know

[Willmish]

My suggestion is: To also add workflows for testing Code samples of those samples we can test - that way we will keep them up to date with dev branch and not realise when someone tries the CA SDK for the first time that the sample is actually broken. Current samples: https://github.com/Green-Software-Foundation/carbon-aware-sdk/tree/dev/samples

[Willmish]

I hope following topics are implemented on GHA - they are accepted in ADR-0011:

* Ship WebAPI container images from GitHub packages

* Ship client libraries which are supported by GitHub packages

I think they should be kicked on release-created event on GHA.

@YaSuenag Would you be willing to collaborate on this with @danuw , as I can see you originally proposed that ADR?

[YaSuenag]

@YaSuenag Would you be willing to collaborate on this with @danuw , as I can see you originally proposed that ADR?

Sure, I'd like to collaborate on this!
However I think #149 is needed at first because OpenAPI document is needed to build client libraries. So I will start to work after merging #149 .

[danuw]

Following conversation with @YaSuenag around client packages releases, following is a refined plan of actions

• 1- PR (submit new features) (1-pr.yml) - [Feature Contribution]: Clean up and finalise first issue of PR workflows #321
  • CLI and WebApp (Build, unit tests, container integration tests?)
• 2- Pre-Release (2-pre-release.yml) - New pre-release for dev or official release for main (builds, versions and publishes container) for each platform - Tracked with [Feature Contribution]: Roadmap to create releases (main and multi platform packages) #326
  • 2-a Deploy (2.a-deploy.yml) - created a container that gets deployed (should be published to packages as well? - depends on config and secret) - [Feature Contribution]: Clean up and first version of API deployment workflow #322
    • manual approval gate for official release / automated approval gate for pre-release
    • release to public API (pulls container, deploys to API)
  • 2-b Create release packages (2.b-generate-libraries.yml):
    • pulls container and create clients for each "supported" platform (JAVA, JS, dotnet via nuget, python via pypi, etc...)
    • set of (basic?) integration tests per platform and then publishes to "public package repositories" (e.g. nuget, pypi, npm, etc..) - tracking ticket TODO
    • at first, all packages are pre-release (long term, element of official release once we have had enough testing/feedback)
• 3- Regular Verification (3.regular-checks.yml): (nighty/daily CA jobs that run check to ensure everything works and CodeQL etc... vulnerabilities) - tracking ticket TODO
• 4- Release (4-release.yml): TODO
  • Trigger is main branch, no retention logic, tag should be manual (or coming from release)

---

[WIP] Work in progress for diagrams below

Numbered are workflows

flowchart TD

    subgraph PreRel[" "]
    direction LR
    PR[1 - PR] --> |merges to| DEV[Dev branch]
    DEV --> | triggers | PreRelease[2 - Pre-Release]
    PreRelease  --> |pushes to| GHP(GitHub Packages)
    PreRelease -->  Deploy[2.a Deploy]
    Deploy -->  | deploys to | myAPI[Public API - Dev]
    PreRelease --> | | libs[2.b Generate libraries]
    Deploy -.-> |pulls from| GHP
    libs -.-> |pulls from| GHP
    AnyClient[Any Client] -.-> |pulls from| GHP
    end

Loading

NOTE: Originally my main thought is that the above can be derived to official release by using the main branch. In all cases, releases would be automated from manual creation of pull requests.

I will need to experiment with GitHub Releases to confirm what I have in mind for path to official release. In the meantime here are some examples of release pages for different repos.

• https://github.com/pimoroni/pimoroni-pico/releases: .uf2 derivatives per available board types (pico, picolipo_16MB, picolipo_4MB, picow_enviro, picow_galactic`...) + source code
• https://github.com/zephyrproject-rtos/zephyr/releases: spdx + source code
• https://github.com/dotnet/core/releases: source code
• https://github.com/raspberrypi/rpi-imager/releases: per platform (exe, dmg, deb, ddeb) installers + source code

Reference docs:

• https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes
• https://www.youtube.com/watch?v=_ueJ3LrRqPU&list=PLtvmFy4Ed_DL4Vd2uup0ZT6o_HFSaF-yW&index=9
• Sharing Docker containers between jobs in a workflow docker/build-push-action#225 (pass docker image between jobs)

[danuw]

Thank you for your time @YaSuenag.

What we discussed:

• Issue with github package visibility -> action on me (@danuw ) to create an issue and talk to @vaughanknight
• issue with docker image not working on MacOS (silicon) -> action on @danuw to research and propose a fix
• Prioritising a public official release package (priority 1 for NTT) -> action on @YaSuenag to propose a new workflow for that
• Lower priority for libraries generation (workflow 2.b described above) -> action on @YaSuenag to update previous proposed changes to reflect latest repo updates (docker image now coming from GH packages)

cc @Willmish @vaughanknight

[Willmish]

Update in #355 : this is collecting updates on devops side in repo, @danuw suggested @YaSuenag to open PR for flow buliding packages for Java and JavaScript libraries for the CA SDK Web API.

[Willmish]

Update in #369: @danuw to update with most up to date info, first cleanup and changes merged in, in #361

[Willmish]

#381 Related

[github-actions]

This issue has not had any activity in 120 days. Please review this issue and ensure it is still relevant. If no more activity is detected on this issue for the next 20 days, it will be closed automatically.

[danuw]

Will need to refresh this.

May be worth closing and creating a new one...

[danuw]

this has been addressed by v.1.2 and v.13