extractcode-libarchive (3.5.1.210122) and extractcode-7z (16.5.210122) not available for macosx_11_0_x86_64

[tiulpin]

Hi! Is it possible to build and put extractcode-libarchive (3.5.1.210122) and extractcode-7z (16.5.210122) on PyPi for new macOS? Without it, installation of scancode-toolkit[full] from pip fails.

This will really help in my local development, can participate if needed.

I used scancode-toolkit[full] without any issues before.

macOS: 11.2 (20D64)

Thanks in advance!

[tiulpin]

Downloading manually the wheels from PyPi, renaming from 10_13 to 11_0 works (and scancode works perfectly).

Why could it be so?

[pombredanne]

Downloading manually the wheels from PyPi, renaming from 10_13 to 11_0 works (and scancode works perfectly).

Great catch and that's a sleek fix! 🙇

Why could it be so?
This is the shenanigan of Python platform tags matching where on macOS many OS versions need to be specified at times e.g. Python thinks that macOS 11.x is not ABI compatible with 10.latest

I will make a new release that will have these tags alright!
FWIW, this is where this is set: https://github.com/nexB/scancode-plugins/blob/develop/builtins/extractcode_libarchive-macosx/setup.cfg#L7 ... and I will just need to add an additional tag there as in:
release = clean --all bdist_wheel --plat-name macosx_10_13_intel --plat-name macosx_11_0 --python-tag py3

[pombredanne]

@tiulpin I pushed new wheels with proper tags. Do you mind trying a pip install in a clean virtualenv on bigsur ?
Thanks!

[tiulpin]

Didn't succeed, seems like the files weren't uploaded to PyPI yet, e.g. https://pypi.org/project/extractcode-7z/#files (don't see 11_0 tag).

I will wait and try again in a few hours if this is a matter of time to get the wheels uploaded there :)

[pombredanne]

@tiulpin thanks for the try and sorry to have wasted your time there! I am stupid as I had not uploaded them to pypi but instead only at https://thirdparty.aboutcode.org/pypi/ which is our poor people PyPI --find-links repo (with all sources an .ABOUT files though) ....
But I have now pushed them on PyPI too!

FWIW these were:

• https://thirdparty.aboutcode.org/pypi/extractcode_7z-16.5.210122-py3-none-macosx_11_0_intel.whl
• https://thirdparty.aboutcode.org/pypi/extractcode_7z-16.5.210122-py3-none-macosx_11_0_intel.whl.ABOUT
• https://thirdparty.aboutcode.org/pypi/extractcode_libarchive-3.5.1.210122-py3-none-macosx_11_0_intel.whl
• https://thirdparty.aboutcode.org/pypi/extractcode_libarchive-3.5.1.210122-py3-none-macosx_11_0_intel.whl.ABOUT
• https://thirdparty.aboutcode.org/pypi/typecode_libmagic-5.39.210122-py3-none-macosx_11_0_intel.whl
• https://thirdparty.aboutcode.org/pypi/typecode_libmagic-5.39.210122-py3-none-macosx_11_0_intel.whl.ABOUT

[pombredanne]

BTW on the test/CI side, https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml does not have yet support for macOS 11.x BUT https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11.0-Readme.md is listing it so we should soon be able to also tests there. (As a side funny note: Microsoft Azure Pipelines doc links to GitHub Actions doc repo, which really means what is no secret that GitHub actions and Azure pipeline are the same thing).

[tiulpin]

• created a new clear environment
• ran pip install scancode-toolkit[full], the installation succesfully finished, thanks! But after running got this:

E   Exception: CRITICAL: libmagic DLL and is magic database are not installed. Unable to continue: you need to install a valid typecode-libmagic plugin with a valid and proper libmagic and magic DB available.

• installed typecode-libmagic – everything works now.

Checked twice, typecode-libmagic is not installed automatically now

[pombredanne]

@tiulpin Thanks. Bummer. We will get to the end of this! Thank you ++ for helping!

[pombredanne]

@tiulpin I wonder what details I would need to get through this. You wrote above:

Downloading manually the wheels from PyPi, renaming from 10_13 to 11_0 works (and scancode works perfectly).

can you tell me which exact wheel name you obtained after renaming?

[tiulpin]

Sorry, forgot to mention: I needed to install typecode-libmagic after that hack too, so I think wheel names do not matter

[pombredanne]

Log from a chat on IRC with @RhinosF1

pombreda> Philippe Ombredanne Howdy,  fierce Authoritative Python Packagers!
10:17 AM I am struggling with macOS 11 trying to find what are the platform tags for it? https://github.com/nexB/scancode-plugins/issues/4
10:18 AM I do not have a macOS 11 install on hand.
10:18 AM so what would be the wheel tag(s) for this -surely lovely- macOS version?
10:20 AM <RhinosF1> Samuel pombreda: I've got big sur installed. Is there a quick way to find out?
10:20 AM 
<pombreda> Philippe Ombredanne RhinosF1: let me see... do you have some Python 3 of sorts installed? AS well as Xcode?
10:20 AM <RhinosF1> Samuel pombreda: I have 3.9
10:23 AM 
<pombreda> Philippe Ombredanne let me assemble simple instructions:
10:23 AM mkdir tmp && cd tmp && curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv/virtualenv.pyz && python3.9 virtualenv.pyz .
10:23 AM source bin/activate
10:23 AM pip wheel bitarray
10:24 AM RhinosF1: this will create an isolated virtualenv for you and build a wheel for bitarray which is a tiny enough package with native C code (that I use extensively) ... so the built wheel will have surely bigsurely tags :P 
10:26 AM <RhinosF1> Samuel pombreda: virtualenv: error: the following arguments are required: dest
10:26 AM 
<pombreda> Philippe Ombredanne RhinosF1:  there is a dot at the end of teh line :)
10:26 AM that's the destination e.g. the current tmp directory you should be in after the cd 
10:28 AM In the output you should see something such as "Created wheel for bitarray: filename=bitarray-1.6.3<some more stuffs>.........."
10:28 AM that's the line I am interested in
10:28 AM <RhinosF1> Samuel pombreda: doing now but apparently xcode-select needs to install. I must have missed it when it corrupted itself and i reset everything
10:29 AM 
<pombreda> Philippe Ombredanne RhinosF1: poor of you! I do not want to corrupt anything on your shiny OS 0;)
10:31 AM I wish some good ole' apt or rpm would be an option there ... (brew is a decent alternative AFAIK on macOS?)
10:36 AM <RhinosF1> Samuel pombreda: filename=bitarray-1.6.3-cp39-cp39-macosx_11_0_x86_64.whl size=59482 sha256=f4ca8952f1d449f7e46388a337349d24cdda55822d6081003493c7aa131caa2d
10:58 AM 
<pombreda> Philippe Ombredanne RhinosF1: you are a mensch!
10:58 AM thank you ++
10:58 AM so this is not "macosx_11_0_intel" as I thought but "macosx_11_0_x86_64" 
10:58 AM 
— pombreda bows to RhinosF1 

[pombredanne]

So the tag to use is not "macosx_11_0_intel" as I thought but "macosx_11_0_x86_64"
Let me respin a build

[pombredanne]

Man... @tiulpin I am an illiterate moron. You ticket title said it "extractcode-libarchive (3.5.1.210122) and extractcode-7z (16.5.210122) not available for macosx_11_0_x86_64" and you were mentioning the tag alright!
I need to take reading lessons.

[tiulpin]

Still, typecode-libmagic is required

[pombredanne]

I think the fact I was using _intel may be the issue. This is for "universal" intel binaries, e.g. 32 and 64 bits.
Let me try with only x86_64 instead

[pombredanne]

Note also for reference I have been bitten by this: pypa/wheel#394

[pombredanne]

Fixed in ScanCode TK v21.2.25

[pombredanne]

@tiulpin Thank you ++ for your help and patience. We should be all good now.

[tiulpin]

Just found out this whole issue could be solved by using a flag export SYSTEM_VERSION_COMPAT=1 – could be mentioned in docs for potential macOS users (got almost the same compatibility issue with the different package).

And everything works now, thanks! (latest 21.2.25)