Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(openEuler): Add openEuler CVE database #397

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

wjunLu
Copy link

@wjunLu wjunLu commented Apr 8, 2024

Description

What's openEuler?

openEuler is an open source, free Linux distribution platform. The platform provides an open community for global developers to build an open, diversified, and architecture-inclusive software ecosystem. openEuler is also an innovative platform that
encourages everyone to propose new ideas, explore new approaches, and practice new solutions.

Learn more, please visit https://www.openeuler.org/en/

Trivy does not support openEuler

We can see that the operating systems currently supported by trivy for security detection does not include openEuler(see https://aquasecurity.github.io/trivy/v0.50/docs/coverage/os/).

image

To support openEuler

Now, openEuler has 2,345,659 users, 18,072 contributors and 1,501 organization members(see https://datastat.openeuler.org/en/overview). It is necessary to support such a very mature open source operating system.

Discussion

Our discussion is here aquasecurity/trivy#6400

Relatived PRs

aquasecurity/vuln-list-update#284

@wjunLu wjunLu requested a review from knqyf263 as a code owner April 8, 2024 11:08
@CLAassistant
Copy link

CLAassistant commented Apr 8, 2024

CLA assistant check
All committers have signed the CLA.

@wjunLu wjunLu changed the title Add openEuler CVE database feat(openEuler): Add openEuler CVE database Apr 11, 2024
@wjunLu
Copy link
Author

wjunLu commented May 17, 2024

The testing result follows:

=== RUN   TestVulnSrc_Update
=== RUN   TestVulnSrc_Update/happy_path_with_openEuler
2024/05/17 17:07:02 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(dir_doesn't_exist)
2024/05/17 17:07:02 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(failed_to_decode)
2024/05/17 17:07:02 Saving openEuler CVRF
--- PASS: TestVulnSrc_Update (0.19s)
    --- PASS: TestVulnSrc_Update/happy_path_with_openEuler (0.12s)
    --- PASS: TestVulnSrc_Update/sad_path_(dir_doesn't_exist) (0.02s)
    --- PASS: TestVulnSrc_Update/sad_path_(failed_to_decode) (0.04s)
=== RUN   TestVulnSrc_Get
=== RUN   TestVulnSrc_Get/happy_path_1
=== RUN   TestVulnSrc_Get/happy_path_2
=== RUN   TestVulnSrc_Get/no_advisories_are_returned
=== RUN   TestVulnSrc_Get/GetAdvisories_returns_an_error
--- PASS: TestVulnSrc_Get (0.36s)
    --- PASS: TestVulnSrc_Get/happy_path_1 (0.09s)
    --- PASS: TestVulnSrc_Get/happy_path_2 (0.09s)
    --- PASS: TestVulnSrc_Get/no_advisories_are_returned (0.09s)
    --- PASS: TestVulnSrc_Get/GetAdvisories_returns_an_error (0.09s)
=== RUN   TestSeverityFromThreat
--- PASS: TestSeverityFromThreat (0.00s)
=== RUN   TestGetOSVersion
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4
2024/05/17 17:07:02 invalid openEuler version: 20.03-LTS-LTS-SP4
=== RUN   TestGetOSVersion/cpe:/a:openEuler:23.09
--- PASS: TestGetOSVersion (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:23.09 (0.00s)
PASS
ok      github.com/aquasecurity/trivy-db/pkg/vulnsrc/openeuler

@julien-faye
Copy link

+1 to add support for openEuler OS!
It is a great alternative for CentOS!

@wjunLu
Copy link
Author

wjunLu commented Jun 28, 2024

Could someone run tests for this PR? Thanks!

@wjunLu
Copy link
Author

wjunLu commented Jul 10, 2024

@knqyf263 Could you please review this PR currently? Thank you!

pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler_test.go Outdated Show resolved Hide resolved
@wjunLu
Copy link
Author

wjunLu commented Jul 11, 2024

@DmitriyLewen Thank you very much!
I have changed the code, and the testing result looks good as following:

=== RUN   TestVulnSrc_Update
=== RUN   TestVulnSrc_Update/happy_path_with_openEuler
2024/07/11 09:44:50 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(dir_doesn't_exist)
2024/07/11 09:44:50 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(failed_to_decode)
2024/07/11 09:44:50 Saving openEuler CVRF
--- PASS: TestVulnSrc_Update (0.20s)
    --- PASS: TestVulnSrc_Update/happy_path_with_openEuler (0.16s)
    --- PASS: TestVulnSrc_Update/sad_path_(dir_doesn't_exist) (0.02s)
    --- PASS: TestVulnSrc_Update/sad_path_(failed_to_decode) (0.02s)
=== RUN   TestVulnSrc_Get
=== RUN   TestVulnSrc_Get/happy_path
=== RUN   TestVulnSrc_Get/no_advisories_are_returned
=== RUN   TestVulnSrc_Get/GetAdvisories_returns_an_error
--- PASS: TestVulnSrc_Get (0.26s)
    --- PASS: TestVulnSrc_Get/happy_path (0.09s)
    --- PASS: TestVulnSrc_Get/no_advisories_are_returned (0.09s)
    --- PASS: TestVulnSrc_Get/GetAdvisories_returns_an_error (0.08s)
=== RUN   TestSeverityFromThreat
--- PASS: TestSeverityFromThreat (0.00s)
=== RUN   TestGetOSVersion
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4
2024/07/11 09:44:50 invalid openEuler version: 20.03-LTS-LTS-SP4
=== RUN   TestGetOSVersion/cpe:/a:openEuler:23.09
--- PASS: TestGetOSVersion (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:23.09 (0.00s)
PASS
coverage: 84.0% of statements
ok      github.com/aquasecurity/trivy-db/pkg/vulnsrc/openeuler  2.264s  coverage: 84.0% of statements

pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
@wjunLu
Copy link
Author

wjunLu commented Jul 11, 2024

@DmitriyLewen I have changed the code as we discussed above. The result looks good

=== RUN   TestVulnSrc_Update
=== RUN   TestVulnSrc_Update/happy_path_with_openEuler
2024/07/11 20:30:30 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(dir_doesn't_exist)
2024/07/11 20:30:30 Saving openEuler CVRF
=== RUN   TestVulnSrc_Update/sad_path_(failed_to_decode)
2024/07/11 20:30:30 Saving openEuler CVRF
--- PASS: TestVulnSrc_Update (0.11s)
    --- PASS: TestVulnSrc_Update/happy_path_with_openEuler (0.08s)
    --- PASS: TestVulnSrc_Update/sad_path_(dir_doesn't_exist) (0.01s)
    --- PASS: TestVulnSrc_Update/sad_path_(failed_to_decode) (0.02s)
=== RUN   TestVulnSrc_Get
=== RUN   TestVulnSrc_Get/happy_path
=== RUN   TestVulnSrc_Get/no_advisories_are_returned
=== RUN   TestVulnSrc_Get/GetAdvisories_returns_an_error
--- PASS: TestVulnSrc_Get (0.15s)
    --- PASS: TestVulnSrc_Get/happy_path (0.05s)
    --- PASS: TestVulnSrc_Get/no_advisories_are_returned (0.05s)
    --- PASS: TestVulnSrc_Get/GetAdvisories_returns_an_error (0.04s)
=== RUN   TestSeverityFromThreat
--- PASS: TestSeverityFromThreat (0.00s)
=== RUN   TestGetOSVersion
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03
=== RUN   TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4
2024/07/11 20:30:31 invalid openEuler version: 20.03-LTS-LTS-SP4
=== RUN   TestGetOSVersion/cpe:/a:openEuler:23.09
--- PASS: TestGetOSVersion (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:22.03-LTS-SP2 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:21.03 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:openEuler:20.03-LTS-LTS-SP4 (0.00s)
    --- PASS: TestGetOSVersion/cpe:/a:openEuler:23.09 (0.00s)
PASS
coverage: 84.7% of statements
ok      github.com/aquasecurity/trivy-db/pkg/vulnsrc/openeuler  2.057s  coverage: 84.7% of statements

@wjunLu
Copy link
Author

wjunLu commented Jul 12, 2024

@DmitriyLewen Do you have some other suggestions for this PR?

@DmitriyLewen
Copy link
Contributor

Hello @wjunLu
I refactored your code a little:

  • updated logic for arches (788f43b + d396db7)
  • add arch to Get function - 66c9b6b
    Can you take a look and confirm that i didn't miss anything?

If this is okay - please update aquasecurity/trivy#6475 (you can use go mod edit -replace to use commit from your fork).

@wjunLu
Copy link
Author

wjunLu commented Jul 12, 2024

Hello @wjunLu I refactored your code a little:

  • updated logic for arches (788f43b + d396db7)
  • add arch to Get function - 66c9b6b
    Can you take a look and confirm that i didn't miss anything?

If this is okay - please update aquasecurity/trivy#6475 (you can use go mod edit -replace to use commit from your fork).

Thank you very much! I'm checking this.

@wjunLu
Copy link
Author

wjunLu commented Jul 12, 2024

Hello @wjunLu I refactored your code a little:

  • updated logic for arches (788f43b + d396db7)
  • add arch to Get function - 66c9b6b
    Can you take a look and confirm that i didn't miss anything?

If this is okay - please update aquasecurity/trivy#6475 (you can use go mod edit -replace to use commit from your fork).

Thank you again! I have no problem for this! I will update aquasecurity/trivy#6475 soon.

@wjunLu
Copy link
Author

wjunLu commented Jul 12, 2024

Hello @wjunLu I refactored your code a little:

  • updated logic for arches (788f43b + d396db7)
  • add arch to Get function - 66c9b6b
    Can you take a look and confirm that i didn't miss anything?

If this is okay - please update aquasecurity/trivy#6475 (you can use go mod edit -replace to use commit from your fork).

@wjunLu wjunLu closed this Jul 12, 2024
@wjunLu wjunLu reopened this Jul 12, 2024
@wjunLu
Copy link
Author

wjunLu commented Jul 12, 2024

@DmitriyLewen So sorry!
I accidentally closed this PR, please retest it. I have updated aquasecurity/trivy#6475, please check.
Thank you very much!

Copy link
Contributor

@DmitriyLewen DmitriyLewen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wjunLu Thanks for your work!
LGTM

@knqyf263 take a look, please

@wjunLu
Copy link
Author

wjunLu commented Jul 16, 2024

@DmitriyLewen I have updated my branch from upstream, please re-run the tests. Thank you!

@knqyf263
Copy link
Collaborator

I'll take a look today.

return ""
}
version := parts[4]
// e.g. 23.09, 22.03-LTS, 22.03-LTS-SP3
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we treat 22.03-LTS, 22.03-LTS-SP1, and other versions differently? For example, alpine 3.19.0 and 3.19.1 are considered the same version in Trivy from the security advisory perspective, as their fixed versions are the same.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to preserve those versions, since they have different lifecycles (https://www.openeuler.org/en/other/lifecycle/) and different fixed versions.

For example:
The fixed version of kernel in openEuler-22.03-LTS-SP3 is 5.10.0-217.0.0.120, but that in 22.03-LTS-SP4 is kernel-5.10.0-217.0.0.116

see https://repo.openeuler.org/security/data/cvrf/2024/cvrf-openEuler-SA-2024-1792.xml and https://repo.openeuler.org/security/data/cvrf/2024/cvrf-openEuler-SA-2024-1795.xml

pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
pkg/vulnsrc/openeuler/openeuler.go Outdated Show resolved Hide resolved
func getAffectedPackages(productTree ProductTree) []Package {
var pkgs = make(map[string]Package) // pkgID => Package
for _, branch := range productTree.Branches {
// `src` pkgs are not installed in openEuler.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can get a source package name from RPMBDB.

$ docker run --rm -it openeuler/openeuler:20.03-lts 
[root@5462ae7535d3 /]# rpm -qa --qf "%{NAME} %{SOURCERPM}\n" | grep glibc
glibc-common glibc-2.28-36.oe1.src.rpm
glibc glibc-2.28-36.oe1.src.rpm

So, I'm wondering if a source package name and a list of binary package names are equivalent in terms of vulnerability detection. In other words, wouldn't it be enough to save only the source package name and the fixed version in the DB? It has a great benefit from the perspective of DB size.

Let's take an example. Currently, we save all binary names here, like glibc-devel, nscd, etc.
https://github.com/aquasecurity/vuln-list/blob/ecca778da16828205caa70ba6941582f70041104/openeuler/2021/openEuler-SA-2021-1013.json#L79-L210

I'm wondering if storing only glibc is enough.
https://github.com/aquasecurity/vuln-list/blob/ecca778da16828205caa70ba6941582f70041104/openeuler/2021/openEuler-SA-2021-1013.json#L231-L242

We do that for other distributions. If only the binary packages really affected by the vulnerability are listed, they should be preserved, but the current advisory appears to list all binary packages.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to find out this, the result is as the example shows above that the really affected package is only glibc. I will change the code, for aarch64 and x86_64 packages, only the binary like glibc will be preserved, others like glibc-common or glibc-debuginfo ... will not be preserved. Is that OK?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only the binary like glibc will be preserved, others like glibc-common or glibc-debuginfo ... will not be preserved. Is that OK?

I meant we may want to store only source packages. The source glibc and the binary glibc are technically different. In this example, by chance, there is a binary package name identical to the source package name, but there should also be examples where it does not exist.

How about storing the source package name and the affected architectures, since I think there are cases where x86_64 is affected but aarch64 is not?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, such as Affected package: glibc with Arches: [x86_64, aarch64, noaarch]?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. That's my idea. But I'm not confident as I'm not familiar with openEuler. If it turns out we need binary package names in the future, we will be able to fix it without breaking. If you also don't find any issues, we can go with source names.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @DmitriyLewen, could you please check the changes currently? Thank you very much!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DmitriyLewen @knqyf263, I'm still looking forward to your suggestions, please review the new changes as soon as you have time.
Thank you very much!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wjunLu Please be patient. Maintainers have numerous other tasks and might be taking the summer vacation. It is good to remind maintainers, but please do that only once a week. There is not much point in notifying daily.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, sorry for that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I updated the logic a bit to get the arches by OS version - eee8cdb

This is necessary to avoid adding unaffected arches.
See the test in 751c88d

Comment on lines 194 to 198
// e.g. cpe:/a:openEuler:openEuler:22.03-LTS-SP3
parts := strings.Split(cpe, ":")
if len(parts) != 5 || parts[2] != "openEuler" {
return ""
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wjunLu I found an incorrect CPE format - https://github.com/aquasecurity/vuln-list/blob/ecca778da16828205caa70ba6941582f70041104/openeuler/2022/openEuler-SA-2022-1773.json#L97

I checked other advisories. This format only appears in this file.
Do we need to parse this case or is it a typo by openEular and they should fix it on their end?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I confirmed that some files exist such typo, and those errors occurred in the earlier advisories.

I will try to ask the community to fix this, but I think those advisories that have been released may not be easy to modify.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those advisories that have been released may not be easy to modify.

I hope that it is possible to correct one advisory.

If no - we must provide for such cases

Copy link
Author

@wjunLu wjunLu Jul 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If no - we must provide for such cases

How to do this? list such cases in README or some other place?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, i meant we need to update our logic to add the ability to get the OS version for these cases.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DmitriyLewen Thanks!
Please see

if len(parts) != 5 || parts[2] != "openEuler" {

I added codes to use the wrong format getting the OS version, is that OK?

// e.g., cpe:/a:openEuler:openEuler:22.03-LTS-SP3
parts := strings.Split(cpe, ":")
if len(parts) != 5 || parts[2] != "openEuler" {
// e.g., cpe:/a:openEuler:openEuler-22.03-LTS
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DmitriyLewen I did some changes here, to get OS version from the wrong format, please check.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DmitriyLewen Please re-run the tests and tell me what to do next to push this PR merged. Thank you very much!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try not to use regex if possible.
I updated the logic -1dbede2
@wjunLu please take a look.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Your commit is better.

Copy link
Contributor

@DmitriyLewen DmitriyLewen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wjunLu and i updated logic to use src packages.

@knqyf263 take a look, when you have time.

@wjunLu
Copy link
Author

wjunLu commented Aug 12, 2024

@wjunLu and i updated logic to use src packages.

@knqyf263 take a look, when you have time.

@knqyf263 Could please check this?

@wjunLu
Copy link
Author

wjunLu commented Aug 26, 2024

@wjunLu and i updated logic to use src packages.

@knqyf263 take a look, when you have time.

Hi @knqyf263!
If you have some suggestions, please let me know or comment here. Thank you very much!

@wjunLu
Copy link
Author

wjunLu commented Sep 6, 2024

Hi, @DmitriyLewen @knqyf263 !
Are you busy with other higher-priority things lately? I sincerely hope you can review this PR when you have time.
Thank you very much!

@wjunLu
Copy link
Author

wjunLu commented Sep 18, 2024

Hi @knqyf263 ! Please take a look when you have time.
Thank you very much!

@Yikun
Copy link

Yikun commented Sep 23, 2024

Looks like it‘s ready to support openEuler CVE database.

@knqyf263 Would you mind taking a final look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants