Waterloo internal Python environment - Python 3.8 + packages

[lintool]

Ref: #916

Since we've decided as a group to standardize on Python 3.8 (or, at least there were no objections)... I'm going to ask our admins to do a systemwide upgrade on tuna and orca.

There is, however, something worth discussing: What should we do about packages?

What I've been doing is to set "sensible" system-wide defaults - these happen to be the ones that we run our regressions with, so that if you don't start a virtualenv, you'll get something sensible. So, common packages (sklearn, pandas, tqdm, etc.) will all be available. To override with different versions, you can create your own virtualenv.

The downside here is that I need to sudo pip, which is bad practice: https://stackoverflow.com/questions/21055859/what-are-the-risks-of-running-sudo-pip/21056000

An alternative is to have no system-wide defaults, and force everyone to set up their own virtualenv, even for packages that basically everyone uses, like pandas.

Does anyone have thoughts, comments, etc.?

[lintool]

I tweeted about this - getting some interesting responses already: https://twitter.com/lintool/status/1482365509819678720

Obvious suggestion is to use conda. My thoughts:

• Standardizing on conda forces everyone to use conda. With virtualenv (and just requirements.txt), members can still use conda but are not forced.
• And a bonus to virtualenv is that it aligns better when publishing to PyPI... because you can't assume all artifact users will use conda.

[lintool]

For example, @ben11kehoe suggests:

• Never install anything in system python installs
• Always use a virtualenv

https://ben11kehoe.medium.com/my-python-setup-77c57a2fc4b6

The downside I see here is that if you want something "quick and dirty", you have two options:

1. Create yet another env and install all your basics like pandas (adds friction)
2. Reuse an existing env (possibly polluting the env for an existing project)

[theyorubayesian]

This is interesting. I always rely on conda because it saves me from needing multiple tools as in @ben11kehoe's article.

It's also very easy to use different python versions across different projects while keeping the base environment clean. Then I have a proto environment already created where I do "quick and dirty" work. I've worked this way for years.

The downside, as you mentioned, is forcing everyone to install conda. But it's a one-time cost so.