feat: Adding locations in CycloneDX reports #3989

Mayankrai449 · 2024-03-29T19:19:45Z

closes feat: Add locations in CycloneDX reports #3815
Added feature to enrich CycloneDX sbom reports with locations utilizing the evidence key as per the official format.
This feat requires the latest version of lib4sbom library, which was recently modified to parse location.

The update is scalable to add locations in more reports.
I have also updated the tests to include locations.
@terriko The locations are extracted using

and populated in ProductInfo class. Do update me if any necessary modifications are required

codecov-commenter · 2024-03-29T20:19:10Z

Codecov Report

Attention: Patch coverage is 71.15385% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 80.43%. Comparing base (d6cbe40) to head (4b26ba4).
Report is 171 commits behind head on main.

Files	Patch %	Lines
cve_bin_tool/sbom_manager/__init__.py	52.00%	4 Missing and 8 partials ⚠️
cve_bin_tool/sbom_manager/cyclonedx_parser.py	0.00%	9 Missing ⚠️
cve_bin_tool/sbom_manager/spdx_parser.py	0.00%	5 Missing ⚠️
cve_bin_tool/output_engine/__init__.py	0.00%	2 Missing ⚠️
cve_bin_tool/version_scanner.py	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3989      +/-   ##
==========================================
+ Coverage   75.41%   80.43%   +5.01%     
==========================================
  Files         808      822      +14     
  Lines       11983    12699     +716     
  Branches     1598     1978     +380     
==========================================
+ Hits         9037    10214    +1177     
+ Misses       2593     2062     -531     
- Partials      353      423      +70

Flag	Coverage Δ
longtests	`79.95% <69.23%> (+4.53%)`	⬆️
win-longtests	`78.41% <57.83%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cve_bin_tool/sbom_manager/__init__.py

cve_bin_tool/version_scanner.py

…i449/cve-bin-tool into cyclonedx_location_feat

cve_bin_tool/sbom_manager/__init__.py

cve_bin_tool/sbom_manager/cyclonedx_parser.py

cve_bin_tool/sbom_manager/spdx_parser.py

terriko

I think we're heading the right direction here, but three requests:

if we now have a minimum lib4sbom version you should indicate that in requirements.txt (if we don't do that, pip won't upgrade people who have older versions already installed)
As well as the mocked data, I'd like at least one test that has an actual location (as in, not location/to/product or NotFound)
I'd like some validation and tests so that if someone puts location like <img url="https://evil.com"> we can be sure it won't get propagated weirdly or result in info leakage.

(edit: I can't count)

Mayankrai449 · 2024-04-03T21:56:23Z

@terriko

I have updated requirements.txt to include latest version of lib4sbom (0.7.0)
I have updated few mock locations to resemble itself better. I have prioritized passing all important tests by simplifying locations data. There is also a more dynamic approach to test variations in locations for different systems.

I have added a test for find_product_location function

I created a test environment by modifying sys.path to mimic potential filesystem paths, verifying if the find_product_location function correctly identifies the module's location.

I've added validation for accepting correct locations. Locations are validated before being appended in ProductInfo inst.

Do let me know if any new addition or improvements are required, especially regarding these tests

edit: The test for find_product_location seems to fail on github test config, probably due to environment mismatch.

It passes on my system and it will pass on everyone's local system. But seems like the mocking filesystem paths doesn't work here. Should I skip it for now?

terriko

Not sure off the top of my head what the correct solution is for the test, but I'm flagging this as needing changes due to merge conflict anyhow.

I'm probably not going to have time to help figure out what to do with this one this week because I'm focusing on doing the 3.3 release stuff, but feel free to ping other devs for ideas on gitter or wherever/

terriko

Quick heads up: you've got a merge conflict so the tests can't re-run right now.

Mayankrai449 · 2024-04-16T16:42:36Z

Quick heads up: you've got a merge conflict so the tests can't re-run right now.

yeah working on it 👍

Mayankrai449 · 2024-04-17T08:55:31Z

@terriko

I have solved the test issue for find_product_location functionality by adding a new robust test.
The test/test_sbom.py has locations set to "NotFound" as it will have different outcomes on different systems and mocking the file paths will make this file quite complex. Instead, the new test for find_product_location will do the job.

Do let me know about any further improvements.

terriko

Okay sounds like this is as good as we're going to get for tests for now. Thank you so much for iterating on this; let's get it merged!

Vishnu-2810 · 2024-04-22T13:57:46Z

I've downloaded the latest whl file 3.3 version ..But still I'm not getting filepath in SBOM also I'm using lib4sbom 0.7.0 (latest version) why? @Mayankrai449 @terriko

Mayankrai449 · 2024-04-22T15:47:46Z

@Vishnu-2810 This feature enriches the SBOM reports, specifically in CycloneDX formats with locations of scanned products if they are available on your system.

Make sure you're using it for CycloneDX reports and whether you see the evidence key being used in your SBOM report.
It searches for the location of a product by iterating through directories specified in sys.path and known installation directories:
Make sure the latest version of lib4sbom is used particularly in your environment and your local codebase is updated.
Execution command for generating CycloneDX reports is "cve-bin-tool --sbom-type cyclonedx --sbom-format json --sbom-output output.sbom.json input_file"

Vishnu-2810 · 2024-04-22T15:54:20Z

Hi Mayank, Thanks for your reply ! I’m running cve bin tool in a python virtual environment . Generating cyclonedx format with latest lib4sbom 0.7.0 as you have mentioned. The only difference is python virtual environment . Is this evidence not available due to this virtual environment ?? @terriko @Mayankrai449

…

On Mon, 22 Apr 2024 at 9:18 PM, Mayank Rai ***@***.***> wrote: @Vishnu-2810 <https://github.com/Vishnu-2810> This feature enriches the SBOM reports, specifically in CycloneDX formats with locations of scanned products if they are available on your system. - Make sure you're using it for CycloneDX reports and whether you see the evidence key being used in your SBOM report. - It searches for the location of a product by iterating through directories specified in sys.path and known installation directories: Screenshot.2024-04-22.210705.png (view on web) <https://github.com/intel/cve-bin-tool/assets/110732414/51926466-de72-4ceb-bc59-4888bab30650> - Make sure the latest version of lib4sbom is used particularly in your environment and your local codebase is updated. - Execution command for generating CycloneDX reports is "cve-bin-tool --sbom-type cyclonedx --sbom-format json --sbom-output output.sbom.json input_file" — Reply to this email directly, view it on GitHub <#3989 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVD75XI6TZMSZCYK5KSYAD3Y6UWLVAVCNFSM6AAAAABFO2JEE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRZHE3DQMZYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Mayankrai449 · 2024-04-22T18:19:18Z

@Vishnu-2810 Check whether the libraries and modules are accessible within the virtual environment with required versions. Sometimes the default installation location can be different from expected locations within the environment. Other than that, I don't see the problem listed. The problem could also arise due to type of file being scanned, whether it has the necessary components or not. Can you share the screenshot of your report and execution for further clarification?

Vishnu-2810 · 2024-04-23T01:48:04Z

I've created a virtual environment and installed cve-bin-tool version 3.3.1 inside the virtual environment . So all the libraries can be accessible from this location. I'm scanning an so file which is outside the virtual environment. I'm providing the command "cve-bin-tool /home/vishnu/mozjpeg_4_1_4/lib/libjpeg.so.62 --sbom-type cyclonedx --sbom-format json --sbom-output out.json" to generate SBOM and got the below mentioned json in the SBOM . @Mayankrai449 Please help me to resolve this issue

Mayankrai449 · 2024-04-23T15:53:13Z

@Vishnu-2810 Dont know if this feature covers the scanning of your filetype currently. Can you share the screenshot of your generate_sbom function in cve_bin_tool/output_engine/init.py

Vishnu-2810 · 2024-04-24T02:06:32Z

I've attached the snippet of generate_sbom function of init.py file. Can you please check this @Mayankrai449

def generate_sbom(
self,
all_product_data,
filename="",
sbom_type="spdx",
sbom_format="tag",
sbom_root="CVE-SCAN",
):
"""Create SBOM package and generate SBOM file."""
# Create SBOM
sbom_packages = {}
sbom_relationships = []
my_package = SBOMPackage()
sbom_relationship = SBOMRelationship()
# Create root package
my_package.initialise()
root_package = f'CVEBINTOOL-{Path(sbom_root).name.replace(".", "-")}'
parent = f"SBOM_{root_package}"
my_package.set_name(root_package)
my_package.set_type("application")
my_package.set_filesanalysis(False)
my_package.set_downloadlocation(sbom_root)
license = "NOASSERTION"
my_package.set_licensedeclared(license)
my_package.set_licenseconcluded(license)
my_package.set_supplier("UNKNOWN", "NOASSERTION")
# Store package data
sbom_packages[(my_package.get_name(), my_package.get_value("version"))] = (
my_package.get_package()
)
sbom_relationship.initialise()
sbom_relationship.set_relationship(parent, "DESCRIBES", root_package)
sbom_relationships.append(sbom_relationship.get_relationship())
# Add dependent products
for product_data in all_product_data:
my_package.initialise()
my_package.set_name(product_data.product)
my_package.set_version(product_data.version)
if product_data.vendor != "UNKNOWN":
my_package.set_supplier("Organization", product_data.vendor)
my_package.set_licensedeclared(license)
my_package.set_licenseconcluded(license)
if not (
(my_package.get_name(), my_package.get_value("version"))
in sbom_packages
and product_data.vendor == "unknown"
):
sbom_packages[
(my_package.get_name(), my_package.get_value("version"))
] = my_package.get_package()
sbom_relationship.initialise()
sbom_relationship.set_relationship(
root_package, "DEPENDS_ON", product_data.product
)
sbom_relationships.append(sbom_relationship.get_relationship())

    # Generate SBOM
    my_sbom = SBOM()
    my_sbom.add_packages(sbom_packages)
    my_sbom.add_relationships(sbom_relationships)
    my_generator = SBOMGenerator(
        sbom_type=sbom_type,
        format=sbom_format,
        application="cve-bin-tool",
        version=VERSION,
    )
    my_generator.generate(parent, my_sbom.get_sbom(), filename=filename)

Mayankrai449 · 2024-04-24T14:02:53Z

@Vishnu-2810 It appears that the generate_sbom function in your local codebase has not been updated to include the recent addition of incorporating locations from this feature. It likely lacks the other recent modifications as well.

Could you please update your local repository to include all the changes and try again? Do let me know if it still didnt work.

feat: Adding locations in CycloneDX reports

f44970f

mastersans suggested changes Mar 31, 2024

View reviewed changes

cve_bin_tool/sbom_manager/__init__.py Outdated Show resolved Hide resolved

cve_bin_tool/version_scanner.py Outdated Show resolved Hide resolved

Mayankrai449 added 5 commits March 31, 2024 16:29

feat: cleaner organization for code

bdaf4ac

feat: improvement

3abd312

feat: improvement

4dd77ff

Merge branch 'cyclonedx_location_feat' of https://github.com/Mayankra…

ff4aba0

…i449/cve-bin-tool into cyclonedx_location_feat

test: update

567573d

mastersans suggested changes Apr 1, 2024

View reviewed changes

test: update

eacbbc6

terriko requested changes Apr 1, 2024

View reviewed changes

Mayankrai449 added 3 commits April 2, 2024 12:37

Merge branch 'main' into cyclonedx_location_feat

743bd1c

Merge branch 'main' into cyclonedx_location_feat

282c749

test: new test and validation update

5bb03d0

Mayankrai449 requested a review from terriko April 3, 2024 21:36

Merge branch 'main' into cyclonedx_location_feat

a05f2b9

Mayankrai449 added 5 commits April 4, 2024 10:52

Merge branch 'main' into cyclonedx_location_feat

a1c8d5d

test: test updates for new merges

d48dec5

test: update

001ec72

test: test update for new merges

7154b4e

feat: validation update

9c730d5

terriko requested changes Apr 10, 2024

View reviewed changes

terriko added this to the 3.3.1 milestone Apr 10, 2024

test: update

81c9dd5

terriko requested changes Apr 16, 2024

View reviewed changes

Mayankrai449 added 3 commits April 16, 2024 22:25

Merge branch 'main' into cyclonedx_location_feat

ad87a7b

docs: new func docstring

6e07a84

Merge branch 'main' into cyclonedx_location_feat

ed7dd5b

Mayankrai449 added 4 commits April 17, 2024 01:57

test: update

148c5ef

Merge branch 'main' into cyclonedx_location_feat

07feefb

Merge branch 'main' into cyclonedx_location_feat

70db334

test: update

793340d

Mayankrai449 requested a review from terriko April 17, 2024 08:59

Merge branch 'main' into cyclonedx_location_feat

4b26ba4

terriko approved these changes Apr 18, 2024

View reviewed changes

terriko merged commit b03681c into intel:main Apr 18, 2024
22 checks passed

Mayankrai449 deleted the cyclonedx_location_feat branch April 18, 2024 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Adding locations in CycloneDX reports #3989

feat: Adding locations in CycloneDX reports #3989

Mayankrai449 commented Mar 29, 2024 •

edited

Loading

codecov-commenter commented Mar 29, 2024 •

edited

Loading

terriko left a comment •

edited

Loading

Mayankrai449 commented Apr 3, 2024 •

edited

Loading

terriko left a comment

terriko left a comment

Mayankrai449 commented Apr 16, 2024 •

edited

Loading

Mayankrai449 commented Apr 17, 2024

terriko left a comment

Vishnu-2810 commented Apr 22, 2024

Mayankrai449 commented Apr 22, 2024

Vishnu-2810 commented Apr 22, 2024 via email •

edited

Loading

Mayankrai449 commented Apr 22, 2024

Vishnu-2810 commented Apr 23, 2024 •

edited

Loading

Mayankrai449 commented Apr 23, 2024

Vishnu-2810 commented Apr 24, 2024

Mayankrai449 commented Apr 24, 2024

feat: Adding locations in CycloneDX reports #3989

feat: Adding locations in CycloneDX reports #3989

Conversation

Mayankrai449 commented Mar 29, 2024 • edited Loading

codecov-commenter commented Mar 29, 2024 • edited Loading

Codecov Report

terriko left a comment • edited Loading

Choose a reason for hiding this comment

Mayankrai449 commented Apr 3, 2024 • edited Loading

I have added a test for find_product_location function

edit: The test for find_product_location seems to fail on github test config, probably due to environment mismatch.

terriko left a comment

Choose a reason for hiding this comment

terriko left a comment

Choose a reason for hiding this comment

Mayankrai449 commented Apr 16, 2024 • edited Loading

Mayankrai449 commented Apr 17, 2024

Do let me know about any further improvements.

terriko left a comment

Choose a reason for hiding this comment

Vishnu-2810 commented Apr 22, 2024

Mayankrai449 commented Apr 22, 2024

Vishnu-2810 commented Apr 22, 2024 via email • edited Loading

Mayankrai449 commented Apr 22, 2024

Vishnu-2810 commented Apr 23, 2024 • edited Loading

Mayankrai449 commented Apr 23, 2024

Vishnu-2810 commented Apr 24, 2024

Mayankrai449 commented Apr 24, 2024

Mayankrai449 commented Mar 29, 2024 •

edited

Loading

codecov-commenter commented Mar 29, 2024 •

edited

Loading

terriko left a comment •

edited

Loading

Mayankrai449 commented Apr 3, 2024 •

edited

Loading

Mayankrai449 commented Apr 16, 2024 •

edited

Loading

Vishnu-2810 commented Apr 22, 2024 via email •

edited

Loading

Vishnu-2810 commented Apr 23, 2024 •

edited

Loading