Skip to content

Commit

Permalink
Merge branch 'main' into pin-versions-for-lts
Browse files Browse the repository at this point in the history
  • Loading branch information
zhumin8 committed Apr 8, 2024
2 parents 158e0ef + ba648ce commit 1edf867
Show file tree
Hide file tree
Showing 18 changed files with 552 additions and 273 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ RUN rm $(which python3)
RUN ln -s $(which python3.11) /usr/local/bin/python
RUN ln -s $(which python3.11) /usr/local/bin/python3
RUN python -m pip install --upgrade pip
RUN cd /src && python -m pip install -r requirements.in
RUN cd /src && python -m pip install -r requirements.txt
RUN cd /src && python -m pip install .

# set dummy git credentials for empty commit used in postprocessing
Expand Down
13 changes: 1 addition & 12 deletions .github/workflows/create_additional_release_tag.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
permissions: write-all
steps:
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Git
Expand All @@ -34,14 +34,3 @@ jobs:
git tag $TAG_NAME
git push origin $TAG_NAME
done
# Generate a tag for unmanaged dependencies check.
# Use fixed tag so that checks in handwritten libraries do not need to
# update the version.
CHECK_LATEST_TAG="unmanaged-dependencies-check-latest"
# delete and create the tag locally.
git tag --delete ${CHECK_LATEST_TAG}
git tag ${CHECK_LATEST_TAG}
# delete the tag in remote repo and push again.
git push --delete origin ${CHECK_LATEST_TAG}
git push origin ${CHECK_LATEST_TAG}
14 changes: 3 additions & 11 deletions .github/workflows/verify_library_generation.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,9 @@ on:
name: verify_library_generation
jobs:
integration_tests:
strategy:
matrix:
java: [ 11 ]
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
java-version: ${{ matrix.java }}
distribution: temurin
cache: maven
- uses: actions/setup-python@v5
with:
python-version: 3.11
Expand All @@ -41,7 +33,7 @@ jobs:
run: |
set -ex
pushd library_generation
pip install -r requirements.in
pip install -r requirements.txt
pip install .
popd
- name: Run integration tests
Expand Down Expand Up @@ -71,7 +63,7 @@ jobs:
run: |
set -ex
pushd library_generation
pip install -r requirements.in
pip install -r requirements.txt
popd
- name: install synthtool
shell: bash
Expand Down Expand Up @@ -115,7 +107,7 @@ jobs:
run: |
set -ex
pushd library_generation
pip install -r requirements.in
pip install -r requirements.txt
popd
- name: Lint
shell: bash
Expand Down
67 changes: 40 additions & 27 deletions library_generation/README.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,45 @@
# Generate a repository containing GAPIC Client Libraries

The script, `generate_repo.py`, allows you to generate a repository containing
GAPIC client libraries (a monorepo, for example, google-cloud-java) from a
configuration file.
The script, `entry_point.py`, allows you to generate a repository containing
GAPIC client libraries with googleapis commit history (a monorepo, for example,
google-cloud-java) from a configuration file.

## Environment

- OS: Linux
- Java runtime environment (8 or above)
- Apache Maven (used in formatting source code)
- Python (3.11.6 or above)
- Docker
- Git

## Prerequisite

In order to generate a version for each library, a versions.txt has to exist
in `repository_path`.
Please refer to [Repository path](#repository-path--repositorypath---optional) for more information.

## Parameters to generate a repository using `generate_repo.py`
## Parameters to generate a repository using `entry_point.py`

### Generation configuration yaml (`generation_config_yaml`)
### Baseline generation configuration yaml (`baseline_generation_config`)

A path to a configuration file containing parameters to generate the repository.
Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository)
for more information.

### Target library API shortname (`target_library_api_shortname`), optional
An absolute or relative path to a generation_config.yaml.
This config file is used for commit history generation, not library
generation.

If specified, the libray whose `api_shortname` equals to `target_library_api_shortname`
will be generated; otherwise all libraries in the configuration file will be
generated.
This can be useful when you just want to generate one library for debugging
purposes.
### Current generation configuration yaml (`current_generation_config`)

The default value is an empty string, which means all libraries will be generated.
An absolute or relative path to a configuration file containing parameters to
generate the repository.
Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository)
for more information.

### Repository path (`repository_path`), optional

The path to where the generated repository goes.

The default value is the current working directory when running the script.
For example, `cd google-cloud-java && python generate_repo.py ...` without
For example, `cd google-cloud-java && python entry_point.py ...` without
specifying the `--repository_path` option will modify the `google-cloud-java`
repository the user `cd`'d into.

Expand All @@ -49,7 +48,9 @@ right version for each library.
Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info
of versions.txt.

## Output of `generate_repo.py`
## Output of `entry_point.py`

### GAPIC libraries

For each module (e.g. `google-cloud-java/java-asset`), the following files/folders
will be created/modified:
Expand All @@ -73,6 +74,11 @@ will be created/modified:
| pom.xml (repo root dir) | Always generated from inputs |
| versions.txt | New entries will be added if they don’t exist |

### googleapis commit history

If both `baseline_generation_config` and `current_generation_config` are
specified, and they contain different googleapis commit, the commit history will
be generated into `pr_description.txt` in the `repository_path`.

## Configuration to generate a repository

Expand Down Expand Up @@ -184,20 +190,27 @@ libraries:
- proto_path: google/cloud/asset/v1p7beta1
```
## An example to generate a repository using `generate_repo.py`
# Local Environment Setup before running `entry_point.py`

1. Assuming Python 3 is installed, follow official guide from [Python.org](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments) to create a virtual environment. The virtual environment can be installed to any folder, usually it is recommended to be installed under the root folder of the project(`sdk-platform-java` in this case).
2. Assuming the virtual environment is installed under `sdk-platform-java`. Run the following command under the root folder of `sdk-platform-java` to install the dependencies of `library_generation`
```bash
# install python module (allows the `library_generation` module to be imported from anywhere)
python -m pip install -r library_generation/requirements.in
# generate the repository
python -m library_generation/generate_repo.py generate \
--generation-config-yaml=/path/to/config-file \
--repository-path=/path/to/repository
python -m pip install -r library_generation/requirements.txt
```
3. Run the following command to install `library_generation` as a module, which allows the `library_generation` module to be imported from anywhere
```bash
python -m pip install library_generation/
```

## An example of generated repository using `generate_repo.py`
## An example to generate a repository using `entry_point.py`

If you run `generate_repo.py` with the example [configuration](#an-example-of-generation-configuration)
```bash
python library_generation/entry_point.py generate \
--baseline-generation-config=/path/to/baseline_config_file \
--current-generation-config=/path/to/current_config_file \
--repository-path=path/to/repository
```
If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration)
shown above, the repository structure is:
```
$repository_path
Expand Down
141 changes: 141 additions & 0 deletions library_generation/cli/entry_point.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os

import click as click
from library_generation.generate_pr_description import generate_pr_descriptions
from library_generation.generate_repo import generate_from_yaml
from library_generation.model.generation_config import from_yaml
from library_generation.utils.generation_config_comparator import compare_config


@click.group(invoke_without_command=False)
@click.pass_context
@click.version_option(message="%(version)s")
def main(ctx):
pass


@main.command()
@click.option(
"--baseline-generation-config",
required=False,
default=None,
type=str,
help="""
Absolute or relative path to a generation_config.yaml.
This config file is used for commit history generation, not library
generation.
""",
)
@click.option(
"--current-generation-config",
required=False,
default=None,
type=str,
help="""
Absolute or relative path to a generation_config.yaml that contains the
metadata about library generation.
""",
)
@click.option(
"--repository-path",
type=str,
default=".",
show_default=True,
help="""
The repository path to which the generated files
will be sent.
If not specified, the repository will be generated to the current working
directory.
""",
)
def generate(
baseline_generation_config: str,
current_generation_config: str,
repository_path: str,
):
"""
Compare baseline generation config and current generation config and
generate changed libraries based on current generation config with commit
history.
If baseline generation config is not specified but current generation
config is specified, generate all libraries based on current generation
config without commit history.
If current generation config is not specified but baseline generation
config is specified, raise FileNotFoundError because current generation
config should be the source of truth of library generation.
If both baseline generation config and current generation config are not
specified, generate all libraries based on the default generation config,
which is generation_config.yaml in the current working directory. Raise
FileNotFoundError if the default config does not exist.
The commit history, if generated, will be available in
repository_path/pr_description.txt.
"""
default_generation_config = f"{os.getcwd()}/generation_config.yaml"

if baseline_generation_config is None and current_generation_config is None:
if not os.path.isfile(default_generation_config):
raise FileNotFoundError(
f"{default_generation_config} does not exist. "
"A valid generation config has to be passed in as "
"current_generation_config or exist in the current working "
"directory."
)
current_generation_config = default_generation_config
elif current_generation_config is None:
raise FileNotFoundError(
"current_generation_config is not specified when "
"baseline_generation_config is specified. "
"current_generation_config should be the source of truth of "
"library generation."
)

current_generation_config = os.path.abspath(current_generation_config)
repository_path = os.path.abspath(repository_path)
if not baseline_generation_config:
# Execute full generation based on current_generation_config if
# baseline_generation_config is not specified.
# Do not generate pull request description.
generate_from_yaml(
config=from_yaml(current_generation_config),
repository_path=repository_path,
)
return

# Compare two generation configs and only generate changed libraries.
# Generate pull request description.
baseline_generation_config = os.path.abspath(baseline_generation_config)
config_change = compare_config(
baseline_config=from_yaml(baseline_generation_config),
current_config=from_yaml(current_generation_config),
)
generate_from_yaml(
config=config_change.current_config,
repository_path=repository_path,
target_library_names=config_change.get_changed_libraries(),
)
generate_pr_descriptions(
config=config_change.current_config,
baseline_commit=config_change.baseline_config.googleapis_commitish,
description_path=repository_path,
)


if __name__ == "__main__":
main()
Loading

0 comments on commit 1edf867

Please sign in to comment.