Contributes script to install IDAES and COIN solvers on Google Colab #1237

adowling2 · 2023-08-09T12:07:10Z

Contributes script to install IDAES and COIN solvers on Google Colab

Fixes

Summary/Motivation:

Installing IDAES (and solvers via get-extensions) is a little quirky on Colab
We've used (a prior version of) this script for teaching with Google Colab for over a year with >40 student testers

Changes proposed in this PR:

Contributes one file

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Contributes script to install IDAES and COIN solvers on Google Colab

adowling2 · 2023-08-09T12:09:03Z

@jsiirola is another good reviewer.

I'd like feedback on:

How should this get tested?
Are we okay with the API / level of commenting?
Are we okay with leaving the install_CBC, etc. backup methods? The default usage is just to run install_idaes() and then install_ipopt(). install_ipopt will first try get-extensions which actually installs many solvers.

jsiirola

In addition to the specific comments below, there are several general changes that should be made:

fix the identified spelling errors
remove the logic for direct downloads from ampl.com
I would suggest replacing (all?) uses of os.system with subprocess.run

jsiirola · 2023-08-09T15:29:26Z

scripts/colab_helper.py

+    '''
+
+    # Check if idaes is available. If not, install it
+    if not package_available("idaes"):


I don't think that looking for the IDAES executable is the best way to see if the package has been installed. Rather, I would recommend:

try: import idaes print("IDAES found! No need to install.") except ImportError: print("Installing idaes via pip...") subprocess.run([sys.executable, "-m", "pip", "install", "-q", "idaes_pse"], check=True) print("idaes was successfully installed") subprocess.run(["idaes", "--version'], check=True)

also note in the above:

the use of subprocess.run() with check=True

the use of an explicit argument list and not a single string

the use of sys.executable

Thanks for the great suggestion. I implemented it.

jsiirola · 2023-08-09T16:47:03Z

scripts/colab_helper.py

+            # Add symbolic link for idaes solvers
+            # These are needed for Colab to find the solvers
+            os.system("ln -s /root/.idaes/bin/ipopt ipopt")
+            os.system("ln -s /root/.idaes/bin/k_aug k_aug")
+            os.system("ln -s /root/.idaes/bin/couenne couenne")
+            os.system("ln -s /root/.idaes/bin/bonmin bonmin")
+            os.system("ln -s /root/.idaes/bin/cbc cbc")
+            os.system("ln -s /root/.idaes/bin/clp clp")
+            os.system("ln -s /root/.idaes/bin/ipopt_l1 ipopt_l1")
+            os.system("ln -s /root/.idaes/bin/dot_sens dot_sens")


This is only needed for the command_with_output below and if you don't import idaes in your script. I generally don't think that it is a good idea to link / move installed files. Instead it would be better to either update where Pyomo looks for executables, or add /root/.idaes/bin to the os.environ['PATH']

I got the PATH alternative to work. Thanks for the great suggestion.

Also, I want to make these solvers accessible without import idaes for two reasons:

Have the script print the solver versions, which is a good check and helpful for debugging.

Reuse this script in workshops/classes/tutorials that introduce Pyomo. In my experience, skipping import idaes is easier with brand-new Pyomo users.

jsiirola · 2023-08-09T16:48:58Z

scripts/colab_helper.py

+            # As a backup, try copying the executable from AMPL
+            if on_colab():
+                print("Installing Ipopt via zip file...")
+                os.system('wget -N -q "https://ampl.com/dl/open/ipopt/ipopt-linux64.zip"')


Note that direct (anonymous) wget from ampl.com no longer works: AMPL requires all downloads to first authenticate using a valid user account.

I just removed these.

…ATH is a better solution.

adowling2 · 2023-08-10T13:09:28Z

@jsiirola This is ready for you to review.

@dangunter You might also find this helpful/interesting. I've been using a script like this to install Pyomo and IDAES in Google Colab for my graduate elective.

adowling2 · 2023-08-10T13:11:37Z

@ksbeattie @lbianchi-lbl I have black on the most updated version of this one file on my computer; it said no changes. I also confirmed I do not have any uncommitted or unpushed changes. Yet the black test is still failing.

For everyone: I tested this on Google Colab (hence several commits this morning.) I am not sure how to automate testing in Google Colab or if that is really even needed.

andrewlee94 · 2023-08-10T13:13:19Z

@adowling2 black is very temperamental - you need to have the correct version of black installed as the formatting changes slightly from version to version.

adowling2 · 2023-08-10T13:21:54Z

@andrewlee94 Now that I have your attention, do you want to review this quickly?

andrewlee94

This looks OK to me, but I am no expert on this sort of stuff.

codecov · 2023-08-10T14:19:06Z

Codecov Report

Patch coverage has no change and project coverage change: -0.01% ⚠️

Comparison is base (deacf4c) 76.83% compared to head (733c94a) 76.83%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1237      +/-   ##
==========================================
- Coverage   76.83%   76.83%   -0.01%     
==========================================
  Files         390      390              
  Lines       61942    61942              
  Branches    11398    11398              
==========================================
- Hits        47595    47592       -3     
- Misses      11878    11882       +4     
+ Partials     2469     2468       -1

see 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jsiirola

This is getting pretty close.

scripts/colab_helper.py

jsiirola · 2023-08-10T14:22:33Z

scripts/colab_helper.py

+
+def _check_available(executable_name):
+    """Utility to check in an executable is available"""
+    return shutil.which(executable_name) or os.path.isfile(executable_name)


(More of an FYI than an action) This is fine for collab (where we know it is Linux-based). However, a more portable approach would be to use something akin to Pyomo's pyomo.common.fileutils.Executable (which is what Pyomo uses for locating executables)

jsiirola · 2023-08-10T14:23:48Z

scripts/colab_helper.py

+            ["idaes", "--version"], check=True, capture_output=True, text=True
+        )
+        print(v.stdout)
+        print(v.stderr)


If you are just going to print out the output anyway, why bother with capture=True?

I couldn't get the output to show up without using capture. I'm also new to using subprocess.

My quick research (again, I know little about subprocess) suggests I'd need to use Popen to get access to STDOUT or STDERR.

On the terminal, the output comes to the terminal if you don't specify capture_output. I wonder if this is different for the collab environment? If that's the case, we want to at least be consistent and capture/print for all subprocess calls.

I expected that behavior on Colab, but no luck when testing in a cell.

@jsiirola, I like your suggestion. I added a verbose flag to display the captured pip, idaes get-extensions, etc. output

lbianchi-lbl

I think having this script here can be valuable for users of IDAES on Google Colab in addition to and beyond the original audience. A few general thoughts:

I don't think there's a way to test this automatically (without an unreasonable amount of effort)
Since the script is tested and will continue to be used in the future, I think we can pragmatically rely on this "outside testing" to detect possible breaking changes
However, since we're adding code to the repository that's not being tested by the CI, before merging, we should:
- Identify who the maintainer(s) of this script will be, e.g. by adding an entry to the CODEOWNERS file
- Add some kind of simple versioning information to the script itself (since the script won't have access to either the Git history or the idaes version at runtime)

lbianchi-lbl · 2023-08-10T18:40:26Z

scripts/colab_helper.py

+"https://raw.githubusercontent.com/adowling2/idaes-pse/colab-install-script/scripts/colab_helper.py"
+
+"""
+


I'd recommend to add some versioning information, e.g. by adding a line here with __version__ = "0.23.8" (or any other date-based versioning schema).

Great suggestion

adowling2 · 2023-08-10T19:47:22Z

@jsiirola @lbianchi-lbl Thanks for the suggested changes. This should be ready.

Changes:

Capture all subprocess output. Added verbose flag to toggle on output for pip install, idaes get-extensions, etc.
Added a version
Added my name in CODEOWNERS
Retested on Google Colab

adowling2 · 2023-08-15T19:45:54Z

@jsiirola @lbianchi-lbl This is ready for your re-review.

Create colab_helper.py

732898d

Contributes script to install IDAES and COIN solvers on Google Colab

adowling2 requested review from lbianchi-lbl and ksbeattie as code owners August 9, 2023 12:07

Fixed URL in comment block

e1542cb

jsiirola requested changes Aug 9, 2023

View reviewed changes

adowling2 added 11 commits August 10, 2023 07:26

Removed outdated wget commands

0868be6

Ready for testing on Colab. Still need to incorporate PATH suggestion.

35ac82c

Ready for more Colab testing

661ebf8

Ready for more testing

0bb22c0

Debugging printing solver versions

10552e2

Removed symbolic link support which is not working with subprocess. P…

50e59f7

…ATH is a better solution.

Ready for final testing

0c69f84

Minor cleanup

a58cf75

Fixed spelling

93cefb4

Ran black

a711552

Added doc strings.

9304b9f

adowling2 requested a review from jsiirola August 10, 2023 13:09

adowling2 self-assigned this Aug 10, 2023

Ran black (installed via pip instead of conda)

2ccfc83

andrewlee94 approved these changes Aug 10, 2023

View reviewed changes

jsiirola reviewed Aug 10, 2023

View reviewed changes

Moved IDAES to beginning of PATH

e07d9b1

adowling2 requested a review from jsiirola August 10, 2023 16:51

ksbeattie added the Priority:Normal Normal Priority Issue or PR label Aug 10, 2023

lbianchi-lbl requested changes Aug 10, 2023

View reviewed changes

Requested changes

733c94a

adowling2 requested a review from lbianchi-lbl August 10, 2023 19:45

adowling2 mentioned this pull request Aug 15, 2023

Updated Pyomo.DoE tutorial notebook. Pyomo/pyomo#2889

Merged

jsiirola approved these changes Aug 15, 2023

View reviewed changes

lbianchi-lbl merged commit 210844b into IDAES:main Aug 17, 2023
31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Contributes script to install IDAES and COIN solvers on Google Colab #1237

Contributes script to install IDAES and COIN solvers on Google Colab #1237

adowling2 commented Aug 9, 2023 •

edited

Loading

adowling2 commented Aug 9, 2023 •

edited

Loading

jsiirola left a comment

jsiirola Aug 9, 2023

adowling2 Aug 10, 2023

jsiirola Aug 9, 2023

adowling2 Aug 10, 2023

adowling2 Aug 10, 2023

jsiirola Aug 9, 2023

adowling2 Aug 10, 2023

adowling2 commented Aug 10, 2023

adowling2 commented Aug 10, 2023

andrewlee94 commented Aug 10, 2023

adowling2 commented Aug 10, 2023

andrewlee94 left a comment

codecov bot commented Aug 10, 2023 •

edited

Loading

jsiirola left a comment

jsiirola Aug 10, 2023 •

edited

Loading

jsiirola Aug 10, 2023

adowling2 Aug 10, 2023

adowling2 Aug 10, 2023 •

edited

Loading

jsiirola Aug 10, 2023

adowling2 Aug 10, 2023

adowling2 Aug 10, 2023

lbianchi-lbl left a comment

lbianchi-lbl Aug 10, 2023

adowling2 Aug 10, 2023

adowling2 commented Aug 10, 2023 •

edited

Loading

adowling2 commented Aug 15, 2023

		"https://raw.githubusercontent.com/adowling2/idaes-pse/colab-install-script/scripts/colab_helper.py"

		"""

Contributes script to install IDAES and COIN solvers on Google Colab #1237

Contributes script to install IDAES and COIN solvers on Google Colab #1237

Conversation

adowling2 commented Aug 9, 2023 • edited Loading

Fixes

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

adowling2 commented Aug 9, 2023 • edited Loading

jsiirola left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adowling2 commented Aug 10, 2023

adowling2 commented Aug 10, 2023

andrewlee94 commented Aug 10, 2023

adowling2 commented Aug 10, 2023

andrewlee94 left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 10, 2023 • edited Loading

Codecov Report

jsiirola left a comment

Choose a reason for hiding this comment

jsiirola Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adowling2 Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbianchi-lbl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adowling2 commented Aug 10, 2023 • edited Loading

adowling2 commented Aug 15, 2023

adowling2 commented Aug 9, 2023 •

edited

Loading

adowling2 commented Aug 9, 2023 •

edited

Loading

codecov bot commented Aug 10, 2023 •

edited

Loading

jsiirola Aug 10, 2023 •

edited

Loading

adowling2 Aug 10, 2023 •

edited

Loading

adowling2 commented Aug 10, 2023 •

edited

Loading