New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Support for Blob Type (s3 and GCS) #879

Open

khrm wants to merge 4 commits into tektoncd:main from khrm:blobLogs

Contributor

khrm commented Nov 10, 2024

Added support for reading forwarded logs to s3 and GCS.
If vector forwards logs to s3 and GCS, these logs can be read using
Go CDK's blob library.
Also, if result logging mechanism is changed from v1alpha2 to v1alpha3
which doesn't support watcher, we can read old logs if they exist.

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Tested your changes locally (if this is a code change)
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Added support for reading forwarded logs to s3 and GCS.
If vector forwards logs to s3 and GCS, these logs can be read using
Go CDK's blob library.
Also, if result logging mechanism is changed from v1alpha2 to v1alpha3
which doesn't support watcher, we can read old logs if they exist.

tekton-robot added the release-note label

tekton-robot requested review from dibyom and enarha

November 10, 2024 06:22

tekton-robot added the size/XXL label

tekton-robot commented Nov 10, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.3%	8.4%	0.2
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/apis/v1alpha3/types.go	100.0%	0.0%	-100.0
pkg/apis/v1alpha3/types.go	Do not exist	42.9%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm force-pushed the blobLogs branch from 273dab3 to 955400a Compare

November 10, 2024 07:13

tekton-robot commented Nov 10, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.3%	8.4%	0.2
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/apis/v1alpha3/types.go	100.0%	0.0%	-100.0
pkg/apis/v1alpha3/types.go	Do not exist	42.9%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm force-pushed the blobLogs branch 2 times, most recently from 3f5b1b8 to c91eaaa Compare

November 11, 2024 08:31

khrm added this to the v0.13.0 milestone

khrm commented

View reviewed changes

Contributor Author

khrm left a comment

/assign @enarha @vdemeester

tekton-robot assigned enarha and vdemeester


          Vendor Dependencies for Blob Logging

eb72f5d

khrm force-pushed the blobLogs branch from c91eaaa to 9e48acf Compare

November 11, 2024 08:40

tekton-robot commented Nov 11, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	100.0%	14.3%	-85.7
pkg/apis/v1alpha3/types.go	Do not exist	33.3%
pkg/apis/v1alpha3/types.go	Do not exist	100.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm force-pushed the blobLogs branch from 9e48acf to 3e0ad8e Compare

November 11, 2024 22:49

tekton-robot commented Nov 11, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	0.0%
pkg/apis/v1alpha3/types.go	Do not exist	66.7%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot commented Nov 11, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm force-pushed the blobLogs branch from 0addfd0 to d4688f4 Compare

November 11, 2024 23:35

tekton-robot commented Nov 11, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm force-pushed the blobLogs branch from d4688f4 to 1ea49b3 Compare

November 12, 2024 00:19

tekton-robot commented Nov 12, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/gcs.go	42.9%	33.9%	-8.9
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

khrm force-pushed the blobLogs branch from 1ea49b3 to d67030f Compare

November 12, 2024 00:41

Contributor Author

khrm commented Nov 12, 2024

/test pull-tekton-results-unit-tests

tekton-robot commented Nov 12, 2024

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

vdemeester approved these changes

View reviewed changes

tekton-robot commented Nov 12, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tekton-robot added the approved label


          Add Support for Blob Type (s3 and GCS)

bafcf17

Added support for reading forwarded logs to s3 and GCS.
If vector forwards logs to s3 and GCS, these logs can be read using
Go CDK's blob library.
Also, if result logging mechanism is changeg from v1alpha2 to v1alpha3
which doesn't support watcher, we can read old logs if they exist.

khrm added 2 commits

November 12, 2024 17:14


          Add setup for S3 compatible log forwarding in Results

533ea6f


          Fix Build tests failure in s3 due to deprecation

a67ad1b

khrm force-pushed the blobLogs branch from d67030f to a67ad1b Compare

November 12, 2024 11:44

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved release-note size/XXL