Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: filter out Artifactory version & sort versions #402

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ $ brew install tfenv
```

Install via Arch User Repository (AUR)

```console
$ yay --sync tfenv
```
Expand Down Expand Up @@ -220,6 +220,54 @@ functionality will be restored.
$ TFENV_REVERSE_REMOTE=1 tfenv list-remote
```

##### `TFENV_SORT_VERSIONS_REMOTE`

Integer (Default: 0)

When using a custom remote, such as Artifactory, instead of the Hashicorp servers,
the list of terraform versions returned by the curl of the remote directory may be inverted
and sorted alphabetically, and not in a version-aware sort.

For example, you will have something like this:

```
1.0.0
1.0.10
1.0.11
1.0.1
1.0.2
1.0.3
1.0.4
1.0.5
1.0.6
1.0.7
1.0.8
1.0.9
```

instead of:

```
1.0.0
1.0.1
1.0.2
1.0.3
1.0.4
1.0.5
1.0.6
1.0.7
1.0.8
1.0.9
1.0.10
1.0.11
```

To use `sort --version-sort` over all the list of versions, set this environment variable:

```console
$ TFENV_SORT_VERSIONS_REMOTE=1 tfenv list-remote
```

##### `TFENV_CONFIG_DIR`

Path (Default: `$TFENV_ROOT`)
Expand Down
18 changes: 15 additions & 3 deletions libexec/tfenv-list-remote
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,20 @@ remote_versions="$(curlw -sSf "${TFENV_REMOTE}/terraform/")" \

#log 'debug' "Remote versions available: ${remote_versions}"; # Even in debug mode this is too verbose

# Remove Artifactory version
remote_versions="$(grep -v -E "Artifactory/[0-9]+\.[0-9]+\.[0-9]+" <<< "${remote_versions}")"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this, could we update the command to be sed -En 's#.*[a-z]*/([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p'
Then we wouldn't need line 80 either, or the calls to uniq, and it is more generic.

Copy link
Author

@samuel-phan samuel-phan Oct 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it, this doesn't work. It will only capture the Artifactory version.

You can check by yourself with the Artifactory sample page that I shared on #400 .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah my bad! I guess I got too caught up in the regex rather than the actual goal.
sed -En 's#.*[a-z]*_([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p' should work.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't capture anything now. I'm not sure to understand what you want to capture, but to me, the current PR implementation is rather OK:

  • I exclude the particular case where we have the Artifactory/ that prefixes the version.
  • I grep all the versions that I can find (just like before).
  • I process the behaviors depending on other env vars.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why that sed command works on my machine with both BSD and GNU sed. If you replace -En with -rn, does it work?
I'm trying to capture only versions that are relevant to Terraform without hard coding the string 'terraform' (although doing this is probably fine atm as OpenTofu hasn't released anything yet) and without hard coding the string 'Artifactory' as if other mirrors are used, I could foresee a long chain of if/elses being created.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a problem of BSD or GNU sed.

If you take this Artifactory page as input:

Artifactory HTML page
<!DOCTYPE html>
<html>
<head><meta name="robots" content="noindex" />
<title>Index of hashicorp/terraform</title>
</head>
<body>
<h1>Index of hashicorp/terraform</h1>
<pre>Name     Last modified      Size</pre><hr/>
<pre><a href="../">../</a>
<a href="0.13.0/">0.13.0/</a>   15-Jun-2023 15:19    -
<a href="0.13.1/">0.13.1/</a>   15-Jun-2023 15:18    -
<a href="0.13.2/">0.13.2/</a>   15-Jun-2023 15:17    -
<a href="0.13.3/">0.13.3/</a>   15-Jun-2023 15:16    -
<a href="0.13.4/">0.13.4/</a>   15-Jun-2023 15:15    -
<a href="0.13.5/">0.13.5/</a>   15-Jun-2023 15:14    -
<a href="0.13.6/">0.13.6/</a>   15-Jun-2023 15:12    -
<a href="0.13.7/">0.13.7/</a>   15-Jun-2023 15:11    -
<a href="0.14.0/">0.14.0/</a>   15-Jun-2023 15:10    -
<a href="0.14.1/">0.14.1/</a>   15-Jun-2023 15:08    -
<a href="0.14.10/">0.14.10/</a>  15-Jun-2023 14:57    -
<a href="0.14.11/">0.14.11/</a>  15-Jun-2023 14:56    -
<a href="0.14.2/">0.14.2/</a>   15-Jun-2023 15:07    -
<a href="0.14.3/">0.14.3/</a>   15-Jun-2023 15:06    -
<a href="0.14.4/">0.14.4/</a>   15-Jun-2023 15:05    -
<a href="0.14.5/">0.14.5/</a>   15-Jun-2023 15:04    -
<a href="0.14.6/">0.14.6/</a>   15-Jun-2023 15:03    -
<a href="0.14.7/">0.14.7/</a>   15-Jun-2023 15:01    -
<a href="0.14.8/">0.14.8/</a>   15-Jun-2023 15:00    -
<a href="0.14.9/">0.14.9/</a>   15-Jun-2023 14:59    -
<a href="0.15.0/">0.15.0/</a>   15-Jun-2023 14:55    -
<a href="0.15.1/">0.15.1/</a>   15-Jun-2023 14:54    -
<a href="0.15.2/">0.15.2/</a>   15-Jun-2023 14:53    -
<a href="0.15.3/">0.15.3/</a>   15-Jun-2023 14:52    -
<a href="0.15.4/">0.15.4/</a>   15-Jun-2023 14:50    -
<a href="0.15.5/">0.15.5/</a>   15-Jun-2023 14:49    -
<a href="0.5.3/">0.5.3/</a>    07-Jun-2023 16:34    -
<a href="1.0.0/">1.0.0/</a>    15-Jun-2023 14:48    -
<a href="1.0.1/">1.0.1/</a>    15-Jun-2023 14:47    -
<a href="1.0.10/">1.0.10/</a>   15-Jun-2023 14:40    -
<a href="1.0.11/">1.0.11/</a>   15-Jun-2023 13:49    -
<a href="1.0.2/">1.0.2/</a>    15-Jun-2023 13:50    -
<a href="1.0.3/">1.0.3/</a>    15-Jun-2023 14:46    -
<a href="1.0.4/">1.0.4/</a>    15-Jun-2023 14:45    -
<a href="1.0.5/">1.0.5/</a>    15-Jun-2023 14:44    -
<a href="1.0.6/">1.0.6/</a>    15-Jun-2023 14:43    -
<a href="1.0.7/">1.0.7/</a>    15-Jun-2023 14:42    -
<a href="1.0.8/">1.0.8/</a>    15-Jun-2023 14:41    -
<a href="1.0.9/">1.0.9/</a>    15-Jun-2023 14:40    -
<a href="1.1.0/">1.1.0/</a>    15-Jun-2023 14:38    -
<a href="1.1.1/">1.1.1/</a>    15-Jun-2023 14:37    -
<a href="1.1.2/">1.1.2/</a>    15-Jun-2023 14:37    -
<a href="1.1.3/">1.1.3/</a>    15-Jun-2023 14:36    -
<a href="1.1.4/">1.1.4/</a>    15-Jun-2023 14:35    -
<a href="1.1.5/">1.1.5/</a>    15-Jun-2023 14:35    -
<a href="1.1.6/">1.1.6/</a>    15-Jun-2023 14:34    -
<a href="1.1.7/">1.1.7/</a>    24-Mar-2022 17:02    -
<a href="1.1.8/">1.1.8/</a>    22-Apr-2022 15:25    -
<a href="1.1.9/">1.1.9/</a>    22-Apr-2022 15:26    -
<a href="1.2.0/">1.2.0/</a>    19-May-2022 11:20    -
<a href="1.2.1/">1.2.1/</a>    24-May-2022 13:36    -
<a href="1.2.2/">1.2.2/</a>    13-Jun-2022 08:19    -
<a href="1.2.3/">1.2.3/</a>    17-Jun-2022 15:57    -
<a href="1.2.4/">1.2.4/</a>    29-Jun-2022 23:00    -
<a href="1.2.5/">1.2.5/</a>    13-Jul-2022 23:58    -
<a href="1.2.6/">1.2.6/</a>    28-Jul-2022 18:30    -
<a href="1.2.7/">1.2.7/</a>    17-Aug-2022 10:24    -
<a href="1.2.8/">1.2.8/</a>    24-Aug-2022 15:37    -
<a href="1.2.9/">1.2.9/</a>    10-Sep-2022 01:24    -
<a href="1.3.0/">1.3.0/</a>    21-Sep-2022 16:21    -
<a href="1.3.1/">1.3.1/</a>    28-Sep-2022 23:35    -
<a href="1.3.2/">1.3.2/</a>    07-Oct-2022 00:17    -
<a href="1.3.3/">1.3.3/</a>    19-Oct-2022 20:27    -
<a href="1.3.4/">1.3.4/</a>    02-Nov-2022 18:35    -
<a href="1.3.5/">1.3.5/</a>    19-Nov-2022 00:55    -
<a href="1.3.6/">1.3.6/</a>    30-Nov-2022 22:16    -
<a href="1.3.7/">1.3.7/</a>    04-Jan-2023 16:27    -
<a href="1.3.8/">1.3.8/</a>    10-Feb-2023 01:21    -
<a href="1.3.9/">1.3.9/</a>    16-Feb-2023 14:44    -
<a href="1.4.0/">1.4.0/</a>    08-Mar-2023 21:36    -
<a href="1.4.1/">1.4.1/</a>    15-Mar-2023 15:43    -
<a href="1.4.2/">1.4.2/</a>    16-Mar-2023 14:31    -
<a href="1.4.3/">1.4.3/</a>    30-Mar-2023 17:40    -
<a href="1.4.4/">1.4.4/</a>    30-Mar-2023 22:02    -
<a href="1.4.5/">1.4.5/</a>    14-Apr-2023 17:40    -
<a href="1.4.6/">1.4.6/</a>    27-Apr-2023 11:24    -
<a href="1.5.0/">1.5.0/</a>    12-Jun-2023 11:38    -
</pre>
<hr/><address style="font-size:small;">Artifactory/7.38.7 Server at XXX Port 8081</address></body></html>

and considering the sed -En 's#.*[a-z]*_([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).*#\1#p',

you're trying to match something:

  • that starts with .*[a-z]*_, i.e. anything ending with an underscore _
  • followed by a version number ([0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?).

There is nothing in the Artifactory HTML page that has a version number prefixed by an underscore _.

I can capture only stuff between <a ...>(version)/</a> but then, I assume that the page returned by HashiCorp, Artifactory or any other mirror MUST follow this format, which is more restrictive and brittle than the current implementation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, my apologies. Feel free to mark this as resolved.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jack-parsons-bjss 🙏

Can we merge this PR then, if all is good?


# Grep the versions
remote_versions="$(grep -o -E "[0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?" <<< "${remote_versions}" | uniq)"

# Sort versions
if [[ "${TFENV_SORT_VERSIONS_REMOTE:-0}" -eq 1 ]]; then
remote_versions="$(sort --version-sort <<< "${remote_versions}")"
fi

# Reverse versions
if [[ "${TFENV_REVERSE_REMOTE:-0}" -eq 1 ]]; then
grep -o -E "[0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?" <<<"${remote_versions}" | uniq | awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }';
else
grep -o -E "[0-9]+\.[0-9]+\.[0-9]+(-(rc|beta|alpha|oci)-?[0-9]*)?" <<<"${remote_versions}" | uniq;
remote_versions="$(awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' <<< "${remote_versions}")"
fi;

cat <<< "${remote_versions}"
4 changes: 2 additions & 2 deletions test/test_install_and_use.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,8 @@ tests__kv=(
"$(tfenv list-remote | grep 'rc' | head -n 1),latest:rc"
"$(tfenv list-remote | grep '^0\.11\.' | head -n 1),latest:^0.11."
'0.11.15-oci,0.11.15-oci'
'0.8.8,latest:^0.8'
'0.7.13,0.7.13'
'0.11.15,latest:^0.11'
'0.11.9,0.11.9'
'0.14.6,v0.14.6'
);

Expand Down
10 changes: 4 additions & 6 deletions test/test_list.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ declare -a errors=();
log 'info' '### List local versions';
cleanup || log 'error' "Cleanup failed?!";

for v in 0.7.2 0.7.13 0.9.1 0.9.2 v0.9.11 0.14.6; do
for v in 0.11.9 0.11.11 v0.11.13 0.14.6; do
log 'info' "## Installing version ${v} to construct list";
tfenv install "${v}" \
&& log 'debug' "Install of version ${v} succeeded" \
Expand All @@ -71,11 +71,9 @@ log 'info' '## Comparing "tfenv list" with default set';
result="$(tfenv list)";
expected="$(cat << EOS
* 0.14.6 (set by $(tfenv version-file))
0.9.11
0.9.2
0.9.1
0.7.13
0.7.2
0.11.13
0.11.11
0.11.9
EOS
)";

Expand Down
12 changes: 6 additions & 6 deletions test/test_symlink.sh
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,14 @@ ln -s "${PWD}"/bin/* "${TFENV_BIN_DIR}";

cleanup || log 'error' 'Cleanup failed?!';

log 'info' '## Installing 0.8.2';
${TFENV_BIN_DIR}/tfenv install 0.8.2 || error_and_proceed 'Install failed';
log 'info' '## Installing 0.11.14';
${TFENV_BIN_DIR}/tfenv install 0.11.14 || error_and_proceed 'Install failed';

log 'info' '## Using 0.8.2';
${TFENV_BIN_DIR}/tfenv use 0.8.2 || error_and_proceed 'Use failed';
log 'info' '## Using 0.11.14';
${TFENV_BIN_DIR}/tfenv use 0.11.14 || error_and_proceed 'Use failed';

log 'info' '## Check-Version for 0.8.2';
check_active_version 0.8.2 || error_and_proceed 'Version check failed';
log 'info' '## Check-Version for 0.11.14';
check_active_version 0.11.14 || error_and_proceed 'Version check failed';

if [ "${#errors[@]}" -gt 0 ]; then
log 'warn' '===== The following symlink tests failed =====';
Expand Down
8 changes: 4 additions & 4 deletions test/test_uninstall.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,18 +63,18 @@ log 'info' '### Test Suite: Uninstall Local Versions';
cleanup || log 'error' 'Cleanup failed?!';

tests__keywords=(
'0.9.1'
'0.11.10'
'0.11.15-oci'
'latest'
'latest:^0.8'
'latest:^0.11'
'v0.14.6'
);

tests__versions=(
'0.9.1'
'0.11.10'
'0.11.15-oci'
"$(tfenv list-remote | head -n1)"
"$(tfenv list-remote | grep -e "^0.8" | head -n1)"
"$(tfenv list-remote | grep -e "^0.11" | head -n1)"
'0.14.6'
);

Expand Down
2 changes: 1 addition & 1 deletion test/test_use_minrequired.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ cleanup || log 'error' 'Cleanup failed?!';

log 'info' '### Install min-required normal version (#.#.#)';

minv='0.8.0';
minv='0.11.9';

echo "terraform {
required_version = \">=${minv}\"
Expand Down