Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add macos-arm64.yml for setup #1121

Draft
wants to merge 8 commits into
base: tf_2_9
Choose a base branch
from
Draft

Add macos-arm64.yml for setup #1121

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
20a38d3
Create macm1.yml
BrianHenryIE Mar 20, 2022
ed475f1
Update controller.py
BrianHenryIE Jul 19, 2022
9a81b54
Add `pytorch-nightly` note
BrianHenryIE Apr 9, 2023
52058d8
Merge branch 'main' into m1-env-yml
BrianHenryIE Apr 9, 2023
b81161e
Update macm1.yml
BrianHenryIE Apr 9, 2023
e03637f
Merge remote-tracking branch 'upstream/tf_2_9' into m1-env-yml
BrianHenryIE Apr 9, 2023
90a9489
rebase to `tf_2_9` branch
BrianHenryIE Apr 10, 2023
b46875b
Rename to `mac-arm64.yml`
BrianHenryIE Apr 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions install/envs/mac-arm64.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# As posted by @cfox570 on Discord:
# @see https://discord.com/channels/662098530411741184/671604367706554388/926561747983294544

# `tensorflow-macos` requires the following environmental variables set before install:
# `export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1`
# `export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1`
# @see https://github.com/grpc/grpc/issues/25082#issuecomment-988464912

name: donkey

channels:
- defaults
- conda-forge
- pytorch
- fastai

dependencies:
- python=3.9 # 3.9.15
- numpy # 1.23.5
- h5py # 3.7.0
- pillow # 9.4.0
- opencv # 4.6.0
- matplotlib # 3.7.1
- tornado # 6.2
- docopt # 0.6.2
- pandas # 1.5.3
- pylint # 2.16.2
- pytest # 7.1.2
- pytest-cov # 4.0.0
- codecov # 2.1.11
- pip # 23.0.1
- progress # 1.5
- paho-mqtt # 1.6.1
- PrettyTable # 3.5.0
- pyfiglet # 0.8.post1
- mypy # 0.981
- pytorch # 1.12.1
- torchvision=0.12 # 0.12.0
- torchaudio # 0.12.1
- pytorch-lightning>=1.9,<2.0 # 1.9.3
- psutil # 5.9.0
- plotly # 5.9.0
- pyyaml # 6.0
- fastai # 2.7.11
- pynmea2 # 1.19.0
- pyserial # 3.5
- utm # 0.7.0
- albumentations # 1.3.0
# - imgaug # 0.4.0 # This _was_ in mac.yml for an older Donkey version
- pip:
- kivy==2.1.0 # 2.1.0 # Moved from macos.yml dependencies section.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would kivy fail in the conda install, do you know?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried moving it back but got ResolvePackageNotFound:

$ conda env create -f mac-arm64.yml 
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - kivy=2.1.0

With verbose output:

$ conda env create -f mac-arm64.yml -v      
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Traceback (most recent call last):
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/exception_handler.py", line 16, in __call__
    return func(*args, **kwargs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/cli/main.py", line 49, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/notices/core.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/cli/main_create.py", line 163, in execute
    result[installer_type] = installer.install(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda_env/installers/conda.py", line 50, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 214, in solve_for_diff
    final_precs = self.solve_final_state(
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 379, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/core/solve.py", line 732, in _add_specs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 649, in _get_package_pool
    pool = self.get_reduced_index(specs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/common/io.py", line 83, in decorated
    return f(*args, **kwds)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 674, in get_reduced_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "/Users/brianhenry/anaconda3/lib/python3.10/site-packages/conda/resolve.py", line 329, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound: 
  - kivy=2.1.0

With no version specified:

$ conda env create -f mac-arm64.yml         
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - kivy

I moved it back under the pip section and it worked again:

$ conda env create -f mac-arm64.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
                                                                                                                                                                    
Preparing transaction: done                                                                                                                                         
Verifying transaction: done                                                                                                                                         
Executing transaction: done                                                                                                                                         
Installing pip dependencies: / Ran pip subprocess with arguments:                                                                                                   
['/Users/brianhenry/anaconda3/envs/donkey-test/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/brianhenry/Sites/donkey-m1-test/condaenv.yqqhctpo.requirements.txt', '--exists-action=b']                                                                                                                                       
Pip subprocess output:                                                                                                                                              

... cut lots of downloads etc ...

Installing collected packages: tensorboard-plugin-wit, libclang, keras, flatbuffers, werkzeug, termcolor, tensorflow-estimator, tensorboard-data-server, six, simple-pid, pygments, pyasn1, protobuf, proglog, opt-einsum, oauthlib, imageio-ffmpeg, grpcio, gast, docutils, decorator, cachetools, absl-py, tensorflow-metal, rsa, requests-oauthlib, pyasn1-modules, moviepy, markdown, Kivy-Garden, keras-preprocessing, google-pasta, astunparse, kivy, google-auth, keras-vis, google-auth-oauthlib, tensorboard, tensorflow-macos
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
Successfully installed Kivy-Garden-0.1.5 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.1 decorator-4.4.2 docutils-0.20.1 flatbuffers-1.12 gast-0.4.0 google-auth-2.21.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.56.0 imageio-ffmpeg-0.4.8 keras-2.9.0 keras-preprocessing-1.1.2 keras-vis-0.5.0 kivy-2.1.0 libclang-16.0.0 markdown-3.4.3 moviepy-1.0.3 oauthlib-3.2.2 opt-einsum-3.3.0 proglog-0.1.10 protobuf-3.19.6 pyasn1-0.5.0 pyasn1-modules-0.3.0 pygments-2.15.1 requests-oauthlib-1.3.1 rsa-4.9 simple-pid-2.0.0 six-1.15.0 tensorboard-2.9.1 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-estimator-2.9.0 tensorflow-macos-2.9.0 tensorflow-metal-0.5.0 termcolor-2.3.0 werkzeug-2.3.6

done
#
# To activate this environment, use
#
#     $ conda activate donkey-test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Then after activating:

$ conda list | grep kivy
kivy                      2.1.0                    pypi_0    pypi
kivy-garden               0.1.5                    pypi_0    pypi

Regardless, should it be removed since it looks like it will be installed during pip install -e .[pc] in setup.py:

          'pc': [
              'matplotlib',
              'kivy',
              'pandas',
              'plotly',
              'albumentations'
          ],

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BrianHenryIE - it seems that for apple silicon the conda install of kivy is not working for whatever reasons, so what you suggest makes sense (to install it w/ pip). In general, the pip install using setup.py is mimicking the conda install, all packages are listed there too. It means you could (theoretically) install DC w/ pip only, but we don't recommend doing that. If you do the conda install first, then the pip install -e .[pc] will not do anything, except setting the entry point. All packages are already found by pip and nothing should be installed at that step any more.

To give you a quick heads-up, we are currently working on retiring the conda install and will be going w/ pip only very soon (I hope by this month otherwise next). So merging what you have done here, would be short lived. Once we are on pip only, it would be good to have apple silicon support too. It would be great if you could help testing that, as it looks like the only changes are the tf packages as you outlined in your yaml file.

- tensorflow-macos==2.9 # 2.9.0
- tensorflow-metal==0.5.0 # 0.5.0
- git+https://github.com/autorope/keras-vis.git # 0.5.0
- simple-pid # 1.0.1
- moviepy # 1.0.3
# - pygame # 2.3.0