Install Python-based tools using make install; make changes to `chp…

…l` and `chpldoc` to work better in prefix installs (chapel-lang#24492)

This PR was created with the intention of adding support for installing
`c2chapel` and other Python-based tooling when it has been built.
However, doing so -- particularly in the prefix-based installation case
-- required some additional implementation work in `chpl` and `chpldoc`.

The main problem was the `CHPL_THIRD_PARTY` directory, which is
transitively (via `chpl_home_utils.py`) used by all the `run-in-venv`
scripts and `chpldoc`. Two problems stemmed from the environment
variable:

* This directory is correctly inferred when the environment is
completely blank, but in an explicitly-set `CHPL_HOME`, it points to an
(assumed) home-based directory. Thus, `export CHPL_HOME=$(chpl
--print-chpl-home)` breaks both `chpl` and `chpldoc`.
* It's very hard to find this directory (and therefore `chpldeps`, which
goes through `CHPL_THIRD_PARTY`) when the top-level program is a bash
script invoked from `/bin`. In particular, invoking `chpl_home_utils.py`
etc. requires knowing the path to `/share`, which the first running
script doesn't know. So we need some sort of bootstrapping.

This PR addresses this by adding a `--print-bootstrap-commands`
developer flag to Chapel, which prints a couple of `export` statements
with `CHPL_HOME` and `CHPL_THIRD_PARTY`. These statements are enough to
locate and get proper output from `chpl_homeutils.py`, which I expect to
be the source of truth for subsequent queries. It also fixes the
"incorrect `--prefix` detection" issue by adjusting the `CHPL_HOME`
processing logic.

While adjusting the `CHPL_HOME` logic, I noticed that it lives in two
places: Dyno and the production compiler. I believe that the Dyno
version is a port of the production version; I did my best to simplify
the Dyno version, and then called out to it from the production
compiler. The net result should be that `chpldoc` and `chpl` behave the
same way w.r.t. environment variables. Since in Dyno we use C++ strings
etc., the logic (which until now used `fprintf` and fixed-sized buffers,
and returned on length errors) has been considerably simplified. Only
the production compiler is left to use fixed-size buffers, and one
corresponding length check is left to support that.

Reviewed by @arezaii -- thanks!

## Testing
- [x] paratest
- [x] `test_install`

compiler/main/driver.cpp

@@ -345,6 +345,7 @@ static bool fPrintLicense = false;
 static bool fPrintSettingsHelp = false;
 static bool fPrintVersion = false;
 static bool fPrintChplHome = false;
+static bool fPrintBootstrapCommands = false;
 
 std::string llvmFlags;
 std::string llvmRemarksFilters;
@@ -409,12 +410,6 @@ std::vector<UniqueString> gDynoGenLibSourcePaths;
 // what top-level module names as astrs were requested to be stored in the lib?
 std::unordered_set<const char*> gDynoGenLibModuleNameAstrs;
 
-static bool isMaybeChplHome(const char* path)
-{
-  return chpl::isMaybeChplHome(std::string(path));
-
-}
-
 static void setChplHomeDerivedVars() {
   int rc;
   rc = snprintf(CHPL_RUNTIME_LIB, FILENAME_MAX, "%s/%s",
@@ -470,112 +465,35 @@ static bool restoreChplHomeDerivedFromEnv() {
   return haveAll;
 }
 
+int main(int argc, char* argv[]);
+
 static void setupChplHome(const char* argv0) {
-  const char* chpl_home = getenv("CHPL_HOME");
-  char*       guess     = NULL;
-  bool        installed = false;
   char        majMinorVers[64];
+  std::string foundChplHome;
+  bool        installed = false;
+  bool        fromEnv   = false;
+  std::string diagnosticMsg;
 
   // Get major.minor version string (used below)
   get_major_minor_version(majMinorVers, sizeof(majMinorVers));
 
-  // Get the executable path.
-  guess = findProgramPath(argv0);
-
-  if (guess) {
-    // Determine CHPL_HOME based on the exe path.
-    // Determined exe path, but don't have a env var set
-    // Look for ../../../util/chplenv
-    // Remove the /bin/some-platform/chpl part
-    // from the path.
-    if( guess[0] ) {
-      int j = strlen(guess) - 5; // /bin and '\0'
-      for( ; j >= 0; j-- ) {
-        if( guess[j] == '/' &&
-            guess[j+1] == 'b' &&
-            guess[j+2] == 'i' &&
-            guess[j+3] == 'n' ) {
-          guess[j] = '\0';
-          break;
-        }
-      }
-    }
+  auto err = chpl::findChplHome(argv0, (void*) main, foundChplHome,
+                                installed, fromEnv, diagnosticMsg);
 
-    if( isMaybeChplHome(guess) ) {
-      // OK!
+  if (!diagnosticMsg.empty()) {
+    if (err) {
+      USR_FATAL("%s\n", diagnosticMsg.c_str());
     } else {
-      // Maybe we are in e.g. /usr/bin.
-      free(guess);
-      guess = NULL;
+      USR_WARN("%s\n", diagnosticMsg.c_str());
     }
+  } else if (err) {
+    USR_FATAL("$CHPL_HOME must be set to run chpl");
   }
 
-  if( chpl_home ) {
-    if( strlen(chpl_home) > FILENAME_MAX )
-      USR_FATAL("$CHPL_HOME=%s path too long", chpl_home);
-
-    if( guess == NULL ) {
-      // Could not find exe path, but have a env var set
-    } else {
-      // We have env var and found exe path.
-      // Check that they match and emit a warning if not.
-      if( ! isSameFile(chpl_home, guess) ) {
-        // Not the same. Emit warning.
-        USR_WARN("$CHPL_HOME=%s mismatched with executable home=%s",
-                 chpl_home, guess);
-      }
-    }
-    // Since we have an enviro var, always use that.
-    strncpy(CHPL_HOME, chpl_home, FILENAME_MAX);
-  } else {
-
-    // Check in a default location too
-    if( guess == NULL ) {
-      char TEST_HOME[FILENAME_MAX+1] = "";
-
-      // Check for Chapel libraries at installed prefix
-      // e.g. /usr/share/chapel/<vers>
-      int rc;
-      rc = snprintf(TEST_HOME, FILENAME_MAX, "%s/%s/%s",
-                  get_configured_prefix(), // e.g. /usr
-                  "share/chapel",
-                  majMinorVers);
-      if ( rc >= FILENAME_MAX ) USR_FATAL("Installed pathname too long");
-
-      if( isMaybeChplHome(TEST_HOME) ) {
-        guess = strdup(TEST_HOME);
-
-        installed = true;
-      }
-    }
-
-    if( guess == NULL ) {
-      // Could not find enviro var, and could not
-      // guess at exe's path name.
-      USR_FATAL("$CHPL_HOME must be set to run chpl");
-    } else {
-      int rc;
-
-      if( strlen(guess) > FILENAME_MAX )
-        USR_FATAL("chpl guessed home %s too long", guess);
-
-      // Determined exe path, but don't have a env var set
-      strncpy(CHPL_HOME, guess, FILENAME_MAX);
-      // Also need to setenv in this case.
-      rc = setenv("CHPL_HOME", guess, 0);
-      if( rc ) USR_FATAL("Could not setenv CHPL_HOME");
-    }
-  }
-
-  // Check that the resulting path is a Chapel distribution.
-  if( ! isMaybeChplHome(CHPL_HOME) ) {
-    // Bad enviro var.
-    USR_WARN("CHPL_HOME=%s is not a Chapel distribution", CHPL_HOME);
+  if (foundChplHome.size() > FILENAME_MAX) {
+    USR_FATAL("$CHPL_HOME=%s path too long", foundChplHome.c_str());
   }
-
-  if( guess )
-    free(guess);
-
+  strncpy(CHPL_HOME, foundChplHome.c_str(), FILENAME_MAX);
 
 
   // Get derived-from-home vars
@@ -1472,6 +1390,7 @@ static ArgumentDescription arg_desc[] = {
  {"parse-only", ' ', NULL, "Stop compiling after 'parse' pass for syntax checking", "N", &fParseOnly, NULL, NULL},
  {"parser-debug", ' ', NULL, "Set parser debug level", "+", &debugParserLevel, "CHPL_PARSER_DEBUG", NULL},
  {"debug-short-loc", ' ', NULL, "Display long [short] location in certain debug outputs", "N", &debugShortLoc, "CHPL_DEBUG_SHORT_LOC", NULL},
+ {"print-bootstrap-commands", ' ', NULL, "Print a Bash bootstrap script to be executed by scripts to determine necessary environment variables.", "F", &fPrintBootstrapCommands, NULL,NULL},
  {"print-emitted-code-size", ' ', NULL, "Print emitted code size", "F", &fPrintEmittedCodeSize, NULL, NULL},
  {"print-module-resolution", ' ', NULL, "Print name of module being resolved", "F", &fPrintModuleResolution, "CHPL_PRINT_MODULE_RESOLUTION", NULL},
  {"print-dispatch", ' ', NULL, "Print dynamic dispatch table", "F", &fPrintDispatch, NULL, NULL},
@@ -1674,6 +1593,11 @@ static void printStuff(const char* argv0) {
     printf("%s\n", CHPL_HOME);
     printedSomething = true;
   }
+  if ( fPrintBootstrapCommands ) {
+    printf("export CHPL_HOME='%s'\n", CHPL_HOME);
+    printf("export CHPL_THIRD_PARTY='%s'\n", CHPL_THIRD_PARTY);
+    printedSomething = true;
+  }
   if ( fPrintChplLoc ) {
     char* guess = findProgramPath(argv0);
 

frontend/include/chpl/util/chplenv.h

@@ -67,15 +67,14 @@ bool isMaybeChplHome(std::string path);
 /*
   Try to locate a proper CHPL_HOME value given the `main` executable's name
   and memory address. Output variables chplHomeOut, installed, fromEnv, and
-  warningMessage are used to capture probable CHPL_HOME, whether chpl appears
+  diagnosticMessage are used to capture probable CHPL_HOME, whether chpl appears
   to be installed, whether we got the value of CHPL_HOME from the environment var,
-  and a possible warning message if CHPL_HOME and the chpl executable's location
-  do not match.
+  and a possible diagnostic message if the function needs to report an issue.
 */
-std::error_code findChplHome(char* argv0, void* mainAddr,
+std::error_code findChplHome(const char* argv0, void* mainAddr,
                              std::string& chplHomeOut,
                              bool& installed, bool& fromEnv,
-                             std::string& warningMessage);
+                             std::string& diagnosticMessage);
 
 } // namespace chpl
 

frontend/lib/util/chplenv.cpp

@@ -119,112 +119,89 @@ bool isMaybeChplHome(std::string path) {
   return llvm::sys::fs::exists(path);
 }
 
-std::error_code findChplHome(char* argv0, void* mainAddr,
+std::error_code findChplHome(const char* argv0, void* mainAddr,
                              std::string& chplHomeOut,
                              bool& installed, bool& fromEnv,
-                             std::string& warningMessage) {
+                             std::string& diagnosticMessage) {
   std::string versionString = getMajorMinorVersion();
-  std::string guess = getExecutablePath(argv0, mainAddr);
+  std::string guessFromBinaryPath = getExecutablePath(argv0, mainAddr);
+  chplHomeOut = std::string();
 
-  const char* chpl_home = getenv("CHPL_HOME");
+  const char* chplHomeEnv = getenv("CHPL_HOME");
 
-  if (!guess.empty()) {
+  // First, Try figuring CHPL_HOME out from the binary's location.
+  // If we're running from /path/to/folder/bin/darwin/chpl,
+  // CHPL_HOME might be /path/to/folder.
+  if (!guessFromBinaryPath.empty()) {
     // truncate path at /bin
-    char* tmp_guess = strdup(guess.c_str());
-    if ( tmp_guess[0] ) {
-      int j = strlen(tmp_guess) - 5; // /bin and '\0'
-      for ( ; j >= 0; j-- ) {
-        if ( tmp_guess[j] == '/' &&
-            tmp_guess[j+1] == 'b' &&
-            tmp_guess[j+2] == 'i' &&
-            tmp_guess[j+3] == 'n' ) {
-          tmp_guess[j] = '\0';
-          break;
-        }
-      }
+    auto binIdx = guessFromBinaryPath.rfind("/bin");
+    if (binIdx != std::string::npos) {
+      guessFromBinaryPath.resize(binIdx);
     }
-    guess = std::string(tmp_guess);
-    if (isMaybeChplHome(guess)) {
-      chplHomeOut = guess;
+
+    if (isMaybeChplHome(guessFromBinaryPath)) {
+      chplHomeOut = guessFromBinaryPath;
     } else {
-      guess = "";
+      guessFromBinaryPath.clear();
     }
   }
 
-  if (chpl_home) {
+  // Compute a predefined location based on the prefix. If we find that
+  // the CHPL_HOME lies in this location, we can reason that this is
+  // a prefix-based installation, and install should be true.
+  //
+  // Check for Chapel libraries at installed prefix
+  // e.g. /usr/share/chapel/<vers>
+  std::string guessFromPrefix = std::string()
+    + getConfiguredPrefix() + "/"
+    + "share/chapel/"
+    + versionString;
+  if (!isMaybeChplHome(guessFromPrefix)) {
+    guessFromPrefix.clear();
+  }
+
+  if (chplHomeEnv) {
+    // If the CHPL_HOME environment variable is set, that is our source of
+    // truth. Do some more work to compare it to other guesses and gather info.
+
+    chplHomeOut = chplHomeEnv;
     fromEnv = true;
-    if(strlen(chpl_home) > FILENAME_MAX)
-      // USR_FATAL("$CHPL_HOME=%s path too long", chpl_home);
-      // error_state
-      // TODO: customize error message?
-      return std::make_error_code(std::errc::filename_too_long);
-    if (guess.empty()) {
-      // Could not find exe path, but have a env var set
-      chplHomeOut = std::string(chpl_home);
-    } else {
-      // We have env var and found exe path.
-      // Check that they match and emit a warning if not.
-      if ( ! isSameFile(chpl_home, guess.c_str()) ) {
-        // Not the same. Emit warning.
-        //USR_WARN("$CHPL_HOME=%s mismatched with executable home=%s",
-        //         chpl_home, guess);
-        warningMessage = "$CHPL_HOME=" + std::string(chpl_home) + " is mismatched with executable home=" + guess;
-      }
-      // Since we have an enviro var, always use that.
-      chplHomeOut = std::string(chpl_home);
+
+    if (isSameFile(chplHomeEnv, guessFromPrefix.c_str())) {
+      // The pre-configured prefix path matches the variable in the environment;
+      // this indicates that the CHPL_HOME is from a prefix-based installation.
+      installed = true;
     }
-  } else {
-    // Check in a default location too
-    if (guess.empty()) {
-      char TEST_HOME[FILENAME_MAX+1] = "";
-
-      // Check for Chapel libraries at installed prefix
-      // e.g. /usr/share/chapel/<vers>
-      int rc;
-      rc = snprintf(TEST_HOME, FILENAME_MAX, "%s/%s/%s",
-                    getConfiguredPrefix(), // e.g. /usr
-                    "share/chapel",
-                    versionString.c_str());
-      if (rc >= FILENAME_MAX) {
-        // USR_FATAL("Installed pathname too long");
-        // TODO: return an error here
-        return std::make_error_code(std::errc::filename_too_long);
-      }
-
-      if (isMaybeChplHome(TEST_HOME)) {
-        guess = strdup(TEST_HOME);
-        installed = true;
-       }
+
+    // Emit a warning if we could guess the CHPL_HOME from the binary's path,
+    // but it's not the same path as the environment variable.
+    if (!guessFromBinaryPath.empty() &&
+        !isSameFile(chplHomeEnv, guessFromBinaryPath.c_str())) {
+      diagnosticMessage = "$CHPL_HOME=" + std::string(chplHomeEnv) +
+                       " is mismatched with executable home=" +
+                       guessFromBinaryPath;
     }
+  } else if (guessFromBinaryPath.empty() && !guessFromPrefix.empty()) {
+    // If no environment variable set, and the path-based guess failed,
+    // the last resort is a prefix-based guess; in this case, installed = true.
 
-    if (guess.empty()) {
-      // Could not find enviro var, and could not
-      // guess at exe's path name.
-      // USR_FATAL("$CHPL_HOME must be set to run chpl");
-      // TODO: customize the error message
-      return std::make_error_code(std::errc::no_such_file_or_directory);
-    } else {
-      int rc;
-
-      if (guess.length() > FILENAME_MAX) {
-      // USR_FATAL("chpl guessed home %s too long", guess);
-        return std::make_error_code(std::errc::filename_too_long);
-      }
-      // Determined exe path, but don't have a env var set
-        rc = setenv("CHPL_HOME", guess.c_str(), 0);
-        if ( rc ) {
-          // USR_FATAL("Could not setenv CHPL_HOME");
-          // TODO: customize the error message
-          return std::make_error_code(std::errc::no_such_file_or_directory);
-         }
-         chplHomeOut = guess;
+    installed = true;
+
+    if (setenv("CHPL_HOME", guessFromPrefix.c_str(), 0)) {
+      return std::error_code(errno, std::system_category());
     }
+
+    chplHomeOut = guessFromPrefix;
   }
+
+  if (chplHomeOut.empty()) {
+    diagnosticMessage = "CHPL_HOME must be set";
+    return std::make_error_code(std::errc::no_such_file_or_directory);
+  }
+
   // Check that the resulting path is a Chapel distribution.
   if (!isMaybeChplHome(chplHomeOut.c_str())) {
-    // Bad enviro var.
-    //USR_WARN("CHPL_HOME=%s is not a Chapel distribution", CHPL_HOME);
-    warningMessage = "CHPL_HOME=" + chplHomeOut + " is not a Chapel distribution";
+    diagnosticMessage = "CHPL_HOME=" + chplHomeOut + " is not a Chapel distribution";
   }
   return std::error_code();
 }

tools/c2chapel/c2chapel

@@ -21,8 +21,14 @@
 #
 
 if [ -z "$CHPL_HOME" ]; then
-  echo "Error: CHPL_HOME is not set"
-  exit 1
+  # We need CHPL_HOME to find run-in-venv.bash. Try falling back to a
+  # compiler in PATH.
+  if output=$(chpl --print-bootstrap-commands); then
+    eval "$output"
+  else
+    echo "Error: CHPL_HOME is not set" 1>&2
+    exit 1
+  fi
 fi
 
 exec $CHPL_HOME/util/config/run-in-venv.bash \