virtual-environments-tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae726d27",
   "metadata": {},
   "source": [
    "## Learning about virtual environments with conda\n",
    "#### Background information\n",
    "* The standard package manager in python is pip (Pip Installs Packages)\n",
    "* The standard central repository for python packages is PyPI (Python Package Index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e572c3c",
   "metadata": {},
   "source": [
    "### What's wrong with just using one Python installation for everything?\n",
    "#### What if I want to use different versions of Python for different projects?\n",
    "Well, you could install another version of python and add it to your path.\n",
    "\n",
    "Then what happens when you use pip?  You can't really be sure without additional information.  You'd have to know which version of python is first on your path, and maybe specify the whole path to the python you want to use to be sure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c64abb",
   "metadata": {},
   "source": [
    "#### What if I have two projects that depend on conflicting versions of the same package?\n",
    "This is a big one: Python can't handle this on its own.  If ProjectA requires basicpackage 2.0 and ProjectB requires basicpackage 3.0, you're out of luck, they can't both exist on the same python.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e99ddcd",
   "metadata": {},
   "source": [
    "#### How do I document dependencies for other users?\n",
    "You can look at every file for the import statements. Or, you can fail upwards by repeatedly running the code, see which import fails, install that package, and repeat until the software works."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e49ec80",
   "metadata": {},
   "source": [
    "_What if there were a better way?_\n",
    "\n",
    "## Enter the Virtual Environment!\n",
    "#### What is a virtual environment?\n",
    "* A virtual environment is a way to easily create a fresh python installation for your project.  Each one can have its own packages with whatever versions it needs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9a8e906",
   "metadata": {},
   "source": [
    "#### Why should I use one?\n",
    "It is Best Practice™ to have a separate virtual environment for every project you work on because:\n",
    "* It becomes trivial to guarantee that every place you use python (your terminal, IDE debugger and terminal, etc) is using the same python installation.\n",
    "* Isolate dependencies between projects to prevent version conflicts.\n",
    "* Makes environments and imports easy to document and easy to replicate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "530eb6ea",
   "metadata": {},
   "source": [
    "#### How does it work?\n",
    "* A virtual environment creates a new Python installation directory. \n",
    "* In the bin/ subdirectory, it has hardlinks to the binaries of another, \"base\" Python installation directory.\n",
    "* It points to its *own* `site-packages` or equivalent directories where packages can be installed.\n",
    "* When we activate a virtual environment, it sets your environment variables to point to the new, hardlinked python.\n",
    "* When we deactivate it, everything goes back to how it was before."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499a6de6",
   "metadata": {},
   "source": [
    "There are several packages that allow for creating python virtual environments (the default venv is good and easy to use) but my favorite is...\n",
    "## Conda\n",
    "* Conda is an open-source python version manager *and* package manager *and* environment manager that runs on Windows, macOS and Linux.\n",
    "* Conda easily creates, saves, loads and switches between environments on your local computer.\n",
    "* It was created for Python data scientists (and that's where it is most popular), but it can theoretically package and distribute software for any language (whereas pip and PyPI are only for python packages).\n",
    "* It has better dependency conflict resolution than pip\n",
    "    - Pip will install a package that breaks your code (or, worse, silently makes it impossible to replicate your results).\n",
    "    - Conda will find a way to choose compatible versions or tell you if you've asked for the impossible.\n",
    "* Some of conda's data science libraries are optimized with the Intel Math Kernel Library for faster training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "160a1ef0",
   "metadata": {},
   "source": [
    "# Sounds great, how do I get started?\n",
    "\n",
    "##  Installing Conda\n",
    "Download conda and add it to your environment.\n",
    "```\n",
    "wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
    "chmod +x Miniconda3-latest-Linux-x86_64.sh\n",
    "./Miniconda3-latest-Linux-x86_64.sh\n",
    "source ~/.bashrc\n",
    "```\n",
    "When prompted about running `conda init`, select yes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ea83221",
   "metadata": {},
   "source": [
    "## Creating an Environment\n",
    "* `conda create -n [name]` The base command.\n",
    "* `-n` specifies the name of my new environment.\n",
    "    * If you would prefer you can instead specify a path to the environment with `-p /prefix/for/env/location`/\n",
    "* `python=x.x` Specifying a python version is optional.\n",
    "* `packagename=x.x` Immediately install packagename to the environment, downloading it if you don't have it.  Specifying a version is optional; if not specified conda will choose the most recent version that is compatible with your other dependencies.\n",
    "\n",
    "Typical fully-loaded example:\n",
    "```\n",
    "conda create -n my-new-env python=3.6 scipy=0.15.0 astroid babel\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2106e0a",
   "metadata": {},
   "source": [
    "## Activating and Deactivating an Environment\n",
    "\n",
    "It could not be easier!\n",
    "\n",
    "![Switching from Conda's base environment, to my default environment, to the environment I created to make this presentation](activate-envs.png \"Switching Between Conda Environments\")\n",
    "\n",
    "No more worrying about which python installation to use:  Just run `python my_script.py` and the python of your active environment will be used to run it.\n",
    "\n",
    "When you're done working on your project, turn off the current virtual environment with `conda deactivate`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3378625",
   "metadata": {},
   "source": [
    "## Installing More Packages\n",
    "* Find available versions with:\n",
    "```\n",
    "conda search scikit-learn | tail\n",
    "```\n",
    "* With your environment activated, run:\n",
    "```\n",
    "conda install scikit-learn=0.24.1\n",
    "```\n",
    "* Specifying a version is optional.\n",
    "* Despite the warnings about pip's dependency resolution issues above, you can still use pip from within a conda environment, so you're not limited to only Anaconda packages.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e87152",
   "metadata": {},
   "source": [
    "## Documentation\n",
    "![image.png](drake_requirements.jpeg)\n",
    "* When your code is ready, run:\n",
    "    ```\n",
    "    conda env export > environment.yml\n",
    "    ```\n",
    "* Include the `environment.yml` file in your git repository for future users of your code.\n",
    "* To create a new environment from an environment.yml file:\n",
    "    ```\n",
    "    conda env create -n my-env-from-file -f environment.yml\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dffaee87",
   "metadata": {},
   "source": [
    "## Delivering Environments\n",
    "Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment.  This is useful when you want to reproduce an environment with limited or no internet access\n",
    "* Requirements:  Source and target machines are of the same OS type (Mac/Linux/Windows) and both have a miniconda install\n",
    "* Activate the base environment so that the package will be available to all sub-environments, then install conda-pack with:\n",
    "```\n",
    "conda install -c conda-forge conda-pack\n",
    "```\n",
    "    * `-c conda-forge` specifies that the package comes from the conda-forge channel instead of the main Anaconda repository.\n",
    "    \n",
    "* Pack the environment with\n",
    "```\n",
    "conda pack -n my-env-for-customer\n",
    "```\n",
    "* Deliver to the .tar.tgz file to the target computer.  In the target's miniconda envs directory, create a new directory for your new env.  Activate the new environment, unpack it, and deactivate it with:\n",
    "```\n",
    "cd /dir/to/miniconda3/envs/my_env\n",
    "source my_env/bin/activate\n",
    "conda-unpack\n",
    "source my_env/bin/deactivate\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee0b5f8",
   "metadata": {},
   "source": [
    "### I've heard of conda, Anaconda, and Miniconda.  What's the difference?\n",
    "* Conda is the package and environment manager software.\n",
    "* Anaconda and Miniconda are both distributions.  They both have conda and a python installation that is used as the base.\n",
    "* Miniconda comes with a minimal python installation and the conda package and environment manager.\n",
    "* Anaconda is a metapackage of 160 data science packages (numpy, scipy, pandas, etc).\n",
    "* If you have Miniconda, you can install all the Anaconda packages with\n",
    "    ```\n",
    "    conda install anaconda\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cdb59d4",
   "metadata": {},
   "source": [
    "### Conda sounds great, are there literally any downsides?\n",
    "* Publishing your own package to PyPI is trivial, but publishing to a conda repository is more of an involved process (you have to build binaries for all supported OSes instead of just uploading the code).\n",
    "* It's not part of core python: the default package for this functionality is venv.\n",
    "* Outside of the data science world, some people find that conda is overkill for pure python projects.*\n",
    "* A basic miniconda environment takes more disk space than a venv one.\n",
    "* Conda's dependency resolution is better, but does take a little longer.\n",
    "\n",
    "<sub>\\* I think they're biased against its data science reputation and have never had a problem with a conda environment, even when working on non-data science code.</sub>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82be26d6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}