VirtualFileSystem NPE on indirect import of from pyexpat import * when using openpyxl

[jord1e]

Hi all,

I was trying to use openpyxl to create some Excel sheets and got a HostException (which is an NPE):

     Traceback (most recent call last):
      File "Unnamed", line 1, in <module>
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/__init__.py", line 7, in <module>
        from openpyxl.workbook import Workbook
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/workbook/__init__.py", line 4, in <module>
        from .workbook import Workbook
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/workbook/workbook.py", line 7, in <module>
        from openpyxl.worksheet.worksheet import Worksheet
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/worksheet/worksheet.py", line 24, in <module>
        from openpyxl.cell import Cell, MergedCell
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/cell/__init__.py", line 3, in <module>
        from .cell import Cell, WriteOnlyCell, MergedCell
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/cell/cell.py", line 27, in <module>
        from openpyxl.styles.styleable import StyleableObject
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/styles/styleable.py", line 13, in <module>
        from .builtins import styles
      File "/graalpy_vfs/venv/lib/python3.11/site-packages/openpyxl/styles/builtins.py", line 1346, in <module>
        ('Normal', NamedStyle.from_tree(fromstring(normal))),
      File "/graalpy_vfs/home/lib-python/3/xml/etree/ElementTree.py", line 1337, in XML
        parser = XMLParser(target=TreeBuilder())
      File "/graalpy_vfs/home/lib-python/3/xml/etree/ElementTree.py", line 1518, in __init__
        from xml.parsers import expat
      File "/graalpy_vfs/home/lib-python/3/xml/parsers/expat.py", line 4, in <module>
        from pyexpat import *
    SystemError: HostException: Cannot invoke "org.graalvm.python.embedding.utils.VirtualFileSystem$BaseEntry.getPlatformPath()" because "entry" is null

Reproduction available here (just run with ./gradlew test - on Linux, pyexpat is not implemented on Windows):
https://github.com/jord1e/poc-gradle-openpyxl-error

CI/CD run with error:
https://github.com/jord1e/poc-gradle-openpyxl-error/actions/runs/11363466654

Diff that creates the error

I am using the new GraalPy Gradle plugin (org.graalvm.python) and GraalPyResources.createContext() as described on
https://www.graalvm.org/latest/reference-manual/python/Embedding-Build-Tools/#virtual-filesystem

I've not had time to investigate it further

Maybe another side-question:
How to convert a bytes (which is an unsigned byte array in Python) directly to a byte[] in Java? At the moment I am just doing .as(int[].class) and manually looping through it :)

[msimacek]

Thank you for the report. The test diff doesn't seem to matter, it seems it's always the second test that fails (if you swap the order, then works fails and fails succeeds). openpyxl imports pyexpat which is a native module. We cannot always reliably reset the global state of native modules, so only one context can use native modules directly. Later contexts currently fall back on LLVM emulation. It seems there is some bug in loading this fallback path, that's why it only happens in the second context. @tomasstupka could you please have a look? I could still reproduce this in master.

You can work around the issue if you use just one global context for all the tests.

How to convert a bytes (which is an unsigned byte array in Python) directly to a byte[] in Java? At the moment I am just doing .as(int[].class) and manually looping through it :)

In the current release, you can do .as(ByteSequence.class).toByteArray(), it should work with any object that implements the buffer API. In the next release you should be able to also do .as(byte[].class).

[jord1e]

Edit: NVM saw your comment, will test it

So what I discovered is that it (importing from openpyxl) works the first time, but not after that

Shown in
https://github.com/jord1e/poc-gradle-openpyxl-error/actions/runs/11364945607
(commit jord1e/poc-gradle-openpyxl-error@dd62b8e)

I tried disabling the sources cache

GraalPyResources.contextBuilder()
        .allowExperimentalOptions(true)
        .option("python.WithCachedSources", "false")
        .build();

But this does not seem to fix the issue

Hopefully that helps in identifying the issue