Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contributes script to install IDAES and COIN solvers on Google Colab #1237

Merged
merged 16 commits into from
Aug 17, 2023
Merged
218 changes: 218 additions & 0 deletions scripts/colab_helper.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
#!/usr/bin/env python
###############################################################################
# The Institute for the Design of Advanced Energy Systems Integrated Platform
# Framework (IDAES IP) was produced under the DOE Institute for the
# Design of Advanced Energy Systems (IDAES).
#
# Copyright (c) 2018-2023 by the software owners: The Regents of the
# University of California, through Lawrence Berkeley National Laboratory,
# National Technology & Engineering Solutions of Sandia, LLC, Carnegie Mellon
# University, West Virginia University Research Corporation, et al.
# All rights reserved. Please see the files COPYRIGHT.md and LICENSE.md
# for full copyright and license information.
###############################################################################
"""
Install IDAES, Ipopt, and other solvers on Google Colab

Created by Alex Dowling ([email protected]) and Jeff Kantor at the University of Notre Dame.

To use this script, add the following to a code block in a Jupyter notebook:

```
# Ipopt installer
import sys
# If running on Google Colab, install Ipopt via IDAES
if "google.colab" in sys.modules:
!wget "https://raw.githubusercontent.com/idaes-pse/main/scripts/colab_helper.py"
import colab_helper
colab_helper.install_idaes()
colab_helper.install_ipopt()

# Otherwise, attempt to load idaes which should inlcude Ipopt

Check warning on line 31 in scripts/colab_helper.py

View workflow job for this annotation

GitHub Actions / Check Spelling

"inlcude" should be "include".
# See https://idaes-pse.readthedocs.io/en/stable/tutorials/getting_started/index.html
# for instructions on running idaes get-extensions
else:
try:
import idaes
# Provided idaes extensions are installed, importing idaes provides access to
print("Successfully loaded idaes.")
except:
print("IDAES is not installed!!!")
```

"""

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend to add some versioning information, e.g. by adding a line here with __version__ = "0.23.8" (or any other date-based versioning schema).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion

import shutil
import sys
import os.path
import os
import urllib

import subprocess

def _check_available(executable_name): return (shutil.which(executable_name) or os.path.isfile(executable_name))

def package_available(package_name):

if package_name == "glpk":
return _check_available("gpsol")
else:
return _check_available(package_name)

def on_colab(): return "google.colab" in sys.modules

def command_with_output(command):
r = subprocess.getoutput(command)
print(r)


def install_idaes():
''' Installs latest version of IDAES-PSE via pip

'''

# Check if idaes is available. If not, install it
if not package_available("idaes"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that looking for the IDAES executable is the best way to see if the package has been installed. Rather, I would recommend:

try:
    import idaes
    print("IDAES found! No need to install.")
except ImportError:
    print("Installing idaes via pip...")
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "idaes_pse"], check=True)
    print("idaes was successfully installed")
    subprocess.run(["idaes", "--version'], check=True)

also note in the above:

  • the use of subprocess.run() with check=True
  • the use of an explicit argument list and not a single string
  • the use of sys.executable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great suggestion. I implemented it.

# Tip: "!pip" means run the 'pip' command outside the notebook.
print("Installing idaes via pip...")
os.system("pip install -q idaes_pse")
assert package_available("idaes"), "idaes was not successfully installed."
print("idaes was successfully installed")
os.system('idaes --version')
else:
print("IDAES found! No need to install.")

def install_ipopt():
''' Install Ipopt and possibly other solvers.

If running on Colab, this will install Ipopt, k_aug, and other COIN-OR
solvers via idaes get-extensions.
'''

# Check if Ipopt (solver) is available. If not, install it.
if not package_available("ipopt"):
# Check if we are running on Google Colab
if on_colab():
# Install idaes solvers
print("Running idaes get-extensions to install Ipopt, k_aug, and more...")
os.system("idaes get-extensions")

# Add symbolic link for idaes solvers
# These are needed for Colab to find the solvers
os.system("ln -s /root/.idaes/bin/ipopt ipopt")
os.system("ln -s /root/.idaes/bin/k_aug k_aug")
os.system("ln -s /root/.idaes/bin/couenne couenne")
os.system("ln -s /root/.idaes/bin/bonmin bonmin")
os.system("ln -s /root/.idaes/bin/cbc cbc")
os.system("ln -s /root/.idaes/bin/clp clp")
os.system("ln -s /root/.idaes/bin/ipopt_l1 ipopt_l1")
os.system("ln -s /root/.idaes/bin/dot_sens dot_sens")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only needed for the command_with_output below and if you don't import idaes in your script. I generally don't think that it is a good idea to link / move installed files. Instead it would be better to either update where Pyomo looks for executables, or add /root/.idaes/bin to the os.environ['PATH']

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the PATH alternative to work. Thanks for the great suggestion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I want to make these solvers accessible without import idaes for two reasons:

  • Have the script print the solver versions, which is a good check and helpful for debugging.
  • Reuse this script in workshops/classes/tutorials that introduce Pyomo. In my experience, skipping import idaes is easier with brand-new Pyomo users.


command_with_output('./ipopt -v')
command_with_output('./k_aug -v')
command_with_output('./couenne -v')
command_with_output('./bonmin -v')
#command_with_output('./cbc -v')
#command_with_output('./clp -v')
command_with_output('./ipopt_l1 -v')


# Check again if Ipopt is available
if not package_available("ipopt"):

# As a backup, try copying the executable from AMPL
if on_colab():
print("Installing Ipopt via zip file...")
os.system('wget -N -q "https://ampl.com/dl/open/ipopt/ipopt-linux64.zip"')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that direct (anonymous) wget from ampl.com no longer works: AMPL requires all downloads to first authenticate using a valid user account.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just removed these.

os.system('!unzip -o -q ipopt-linux64')
# Otherwise, try conda
else:
try:
print("Installing Ipopt via conda...")
os.system('conda install -c conda-forge ipopt')
except:
pass


else:
print("Ipopt found! No need to install.")


# Verify Ipopt is now available
assert package_available("ipopt"), "Ipopt is not available"

print("ipopt was successfully installed")

solvers = ["k_aug", "cbc", "clp", "bonmin", "couenne", "ipopt_l1"]
for s in solvers:
if package_available(s):
print(s,"was successfuly installed")

Check warning on line 149 in scripts/colab_helper.py

View workflow job for this annotation

GitHub Actions / Check Spelling

"successfuly" should be "successfully".
print(" ")

def install_cbc():
''' Install CBC via executable from AMPL.

This is not needed if you run install_ipopt() first, as CBC
is available via idaes get-extensions.
'''
if not package_available("cbc") and on_colab():
#print("Installing cbc via apt-get...")
#os.system('apt-get install -y -qq coinor-cbc')
print("Installing cbc via zip file...")
os.system('wget -N -q "https://ampl.com/dl/open/cbc/cbc-linux64.zip"')
os.system('unzip -o -q cbc-linux64')

# Verify package is now available
assert package_available("cbc"), "cbc is not available"

# command_with_output("./cbc -v")


def install_bonmin():
''' Install Bonmin via executable from AMPL.

This is not needed if you run install_ipopt() first, as CBC
is available via idaes get-extensions.
'''

if not package_available("bonmin") and on_colab():
print("Installing bonmin via zip file...")
os.system('wget -N -q "https://ampl.com/dl/open/bonmin/bonmin-linux64.zip"')
os.system('unzip -o -q bonmin-linux64')

# Verify package is now available
assert package_available("bonmin"), "bonmin is not available"

command_with_output("./bonmin -v")

def install_couenne():
''' Install Couenne via executable from AMPL.

This is not needed if you run install_ipopt() first, as CBC
is available via idaes get-extensions.
'''

if not package_available("couenne") and on_colab():
print("Installing couenne via via zip file...")
os.system('wget -N -q "https://ampl.com/dl/open/couenne/couenne-linux64.zip"')
os.system('unzip -o -q couenne-linux64')

# Verify package is now available
assert package_available("couenne"), "couenne is not available"

command_with_output("./couenne -v")


def install_gecode():
''' Install Gecode via executable from AMPL.
'''

if not package_available("gecode") and on_colab():
print("Installing gecode via via zip file...")
os.system('wget -N -q "https://ampl.com/dl/open/gecode/gecode-linux64.zip"')
os.system('unzip -o -q gecode-linux64')

# Verify package is now available
assert package_available("gecode"), "gecode is not available"

command_with_output("./gecode -v")
Loading