RecursionError in Dependency Resolution with "sphinx-rtd-theme" in requirements.txt #7960

inderps · 2023-11-29T16:20:08Z

Description

I am encountering a RecursionError during the dependency resolution process when sphinx-rtd-theme==2.0.0 is included in my requirements.txt file. This issue seems to occur specifically when sphinx-rtd-theme has sphinx as a dependency, starting from version 0.4.1 of sphinx-rtd-theme.

Steps to Reproduce

Create a requirements.txt file with the following content:

sphinx-rtd-theme

Run the ORT analyze command on the project.

Expected Behavior

ORT should successfully analyze the dependencies without encountering a RecursionError.

Actual Behavior

A RecursionError is thrown during the dependency resolution process. Here's a snippet of the error log:

Additional Context

This issue seems to be triggered by the presence of sphinx as a dependency in sphinx-rtd-theme.
Environment: Running ORT version DOCKER-SNAPSHOT under Java 17.0.9 on Linux.
Python version: 3.11.5

Possible Solutions

Investigating compatibility issues between sphinx-rtd-theme, its dependencies, and the python-inspector tool within ORT.
Adjustments to dependency resolution logic in ORT to handle such scenarios.

Any insights or suggestions on how to resolve or work around this issue would be greatly appreciated. Thank you!

/        \_______   \__    ___/ The OSS Review Toolkit, version DOCKER-SNAPSHOT.
|    |   | |       _/ |    |
|    |   | |    |   \ |    |    Running 'analyze' as 'ort' under Java 17.0.9 on Linux
\________/ |____|___/ |____|    with 6 CPUs and a maximum of 1964 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/ort/.ort/config
ORT_DATA_DIR = /home/ort/.ort
TERM = xterm
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk

Looking for ORT configuration in the following file:
        /home/ort/.ort/config/config.yml (does not exist)

Looking for analyzer-specific configuration in the following files and directories:
        /project/.ort.yml (does not exist)
        /project/resolutions.yml
The following 25 package manager(s) are enabled:
        Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoDep, GoMod, Gradle, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, SPM, Stack, Unmanaged, Yarn, Yarn2
The following 2 package curation provider(s) are enabled:
        DefaultDir, DefaultFile
Analyzing project path:
        /project
Found 1 PIP definition file(s) at:
        requirements.txt
Found in total 1 definition file(s) from the following 1 package manager(s):
        PIP
15:58:36.180 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.python.Pip - Unable to determine dependencies for definition file '/project/requirements.txt': IOException: Running 'python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/ort-PythonInspector8868100086596520097/python-inspector3154295103495926617.json --analyze-setup-py-insecurely --requirement /project/requirements.txt --setup-py /project/setup.py --verbose' in '/project' failed with exit code 1:
/opt/python/versions/3.11.5/lib/python3.11/site-packages/typecode/magic2.py:195: UserWarning: System libmagic found in typical location is used. Install instead a typecode-libmagic plugin for best support.
  warnings.warn(
Traceback (most recent call last):
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolve_cli.py", line 253, in resolve_dependencies
    resolution_result: Dict = resolver_api(
                              ^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 279, in resolve_dependencies
    resolution, purls = resolve(
                        ^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 341, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 382, in get_resolved_dependencies
    return (format_pdt_tree(resolver_results), package_list)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 749, in format_pdt_tree
    dependencies.append(pdt_dfs(mapping=mapping, graph=graph, src=src))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
[...skipping 2945 lines...]
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded


(Above output is limited to each 20 heading and tailing lines.)
15:58:36.394 [DefaultDispatcher-worker-1] ERROR java.lang.Object - PIP failed to resolve dependencies for path 'requirements.txt': IOException: Running 'python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/ort-PythonInspector8868100086596520097/python-inspector3154295103495926617.json --analyze-setup-py-insecurely --requirement /project/requirements.txt --setup-py /project/setup.py --verbose' in '/project' failed with exit code 1:
/opt/python/versions/3.11.5/lib/python3.11/site-packages/typecode/magic2.py:195: UserWarning: System libmagic found in typical location is used. Install instead a typecode-libmagic plugin for best support.
  warnings.warn(
Traceback (most recent call last):
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolve_cli.py", line 253, in resolve_dependencies
    resolution_result: Dict = resolver_api(
                              ^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 279, in resolve_dependencies
    resolution, purls = resolve(
                        ^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 341, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/api.py", line 382, in get_resolved_dependencies
    return (format_pdt_tree(resolver_results), package_list)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 749, in format_pdt_tree
    dependencies.append(pdt_dfs(mapping=mapping, graph=graph, src=src))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
[...skipping 2945 lines...]
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in pdt_dfs
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/versions/3.11.5/lib/python3.11/site-packages/python_inspector/resolution.py", line 731, in <listcomp>
    dependencies = [pdt_dfs(mapping, graph, c) for c in children]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

The text was updated successfully, but these errors were encountered:

sschuberth · 2023-11-29T16:34:00Z

If you can reproduce this with stand-alone python-inspector, please file this directly to https://github.com/nexB/python-inspector/issues instead.

inderps · 2023-11-29T19:16:05Z

@sschuberth I ran it with standalone python-inspector and it worked fine

python-inspector --requirement requirements.txt -p 3.11 -o macos --json output.json

output.json

sschuberth · 2023-11-29T21:14:04Z

Just to double-check: Did you use the same version of Python (3.11.5) and the same version of python-inspector (0.10.0) in both cases?

inderps · 2023-11-30T07:02:23Z

@sschuberth Yes same python and same python-inspector versions

sschuberth · 2023-11-30T07:08:41Z

python-inspector --requirement requirements.txt -p 3.11 -o macos --json output.json

That's not exactly the same command line that ORT uses, though. Please run exactly like ORT:

python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/ort-PythonInspector8868100086596520097/python-inspector3154295103495926617.json --analyze-setup-py-insecurely --requirement /project/requirements.txt --setup-py /project/setup.py --verbose

(With paths adjusted where needed.)

I you can't run on Linux, you may replace --operating-system linux with --operating-system macos, though it actually could make a difference if platform-specific packages are involved.

inderps · 2023-11-30T09:44:56Z

@sschuberth Great thanks for pointing it out. It does fail with the same error when i run the above command.

I have added a new issue in python_inspector repo
aboutcode-org/python-inspector#164

I'm closing this ticket. Thank you once again

sschuberth added bug Issues that are considered to be bugs analyzer About the analyzer tool labels Nov 29, 2023

inderps closed this as completed Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RecursionError in Dependency Resolution with "sphinx-rtd-theme" in requirements.txt #7960

RecursionError in Dependency Resolution with "sphinx-rtd-theme" in requirements.txt #7960

inderps commented Nov 29, 2023

sschuberth commented Nov 29, 2023

inderps commented Nov 29, 2023 •

edited

Loading

sschuberth commented Nov 29, 2023

inderps commented Nov 30, 2023

sschuberth commented Nov 30, 2023

inderps commented Nov 30, 2023

RecursionError in Dependency Resolution with "sphinx-rtd-theme" in requirements.txt #7960

RecursionError in Dependency Resolution with "sphinx-rtd-theme" in requirements.txt #7960

Comments

inderps commented Nov 29, 2023

Description

Steps to Reproduce

Expected Behavior

Actual Behavior

Additional Context

Possible Solutions

sschuberth commented Nov 29, 2023

inderps commented Nov 29, 2023 • edited Loading

sschuberth commented Nov 29, 2023

inderps commented Nov 30, 2023

sschuberth commented Nov 30, 2023

inderps commented Nov 30, 2023

inderps commented Nov 29, 2023 •

edited

Loading