Reverted fetch-cobolcheck and fetch-cobolcheck.ps1 scripts.

[SimaDovakin]

Attempt to fix the problem with cobolcheck fetching.

[github-actions]

This PR touches files which potentially affect the outcome of the tests of an exercise. This will cause all students' solutions to affected exercises to be re-tested.

If this PR does not affect the result of the test (or, for example, adds an edge case that is not worth rerunning all tests for), please add the following to the merge-commit message which will stops student's tests from re-running. Please copy-paste to avoid typos.

[no important files changed]

For more information, refer to the documentation. If you are unsure whether to add the message or not, please ping @exercism/maintainers-admin in a comment. Thank you!

[kapitaali]

what say @ErikSchierboom ?

[ErikSchierboom]

Could you explain why these changes would fix things? It looks to me like they no longer use a shared directory and thus would re-download cobolcheck for each exercise.

[SimaDovakin]

It required root permissions to download cobolcheck to /usr/local/bin, I guess. Maybe we can download cobolcheck to the repo's bin directory and then check if it is already here.

[ErikSchierboom]

Maybe we can download cobolcheck to the repo's bin directory and then check if it is already here.

Yes, either that or store it in $HOME/cobol or $HOME/.cobol or something.

And then update the test.sh scripts.

[kapitaali]

Per the old fetch-cobolcheck source, it fetches it to each repo's ./bin. This PR basically reverts back to the old functionality by adding

  if [[ -d ./bin ]]; then
    output_dir="./bin"
  elif [[ $PWD == */bin ]]; then
    output_dir="$PWD"
  else
    echo "Error: no ./bin directory found. This script should be ran from a repo root." >&2
    return 1
  fi

[SimaDovakin]

Maybe we can download cobolcheck to the repo's bin directory and then check if it is already here.

Yes, either that or store it in $HOME/cobol or $HOME/.cobol or something.

And then update the test.sh scripts.

It makes sense. We can try to fetch cobolcheck and download it to $HOME/cobolcheck. I'll try to do it.

[ErikSchierboom]

Using the repo root is fine too. I don't really care as long as one place is used

[SimaDovakin]

@kapitaali @ErikSchierboom Now cobolcheck is stored in $HOME/cobolcheck/cobolcheck{,exe}. I renamed bin/sync-fetch-cobolcheck-files to bin/sync-exercise-files and added synchronizations for test.sh and test.ps1.

[ErikSchierboom]

Let's try this out

[SimaDovakin]

@kapitaali @ErikSchierboom could we merge it?

[ErikSchierboom]

Done!

[kapitaali]

The tests should work out of the box if you download the exercise with cli. I deleted the anagram directory, downloaded it with cli, copied my source code to src, ran test.sh and got the following:

exercism/cobol/anagram 
❯ sh test.sh
test.sh: 2: Bad substitution
test.sh: 7: [[: not found
Cobolcheck not found, try to fetch it.
curl: (56) Failure writing output to destination, passed 1369 returned 4294967295
test.sh: 16: ./bin/cobolcheck: not found
COMPILE AND RUN TEST
cobc: test.cob: No such file or directory

I'm in favor of reverting back to the old functionality without any modifications where test.sh downloads the cobolcheck binary to the local ./bin directory in the exercise tree.

[SimaDovakin]

@kapitaali Did you try ./test.sh instead of sh test.sh?

[kapitaali]

It should make no difference, but trying ./test.sh gives me

exercism/cobol/anagram 
|➜ ./test.sh
bash: ./test.sh: Permission denied

[SimaDovakin]

I got Permission denied too but after chmod +x test.sh it works.

[kapitaali]

Doing chmod +x test.sh allows me to execute it by running ./test.sh, but the errors persist:

exercism/cobol/anagram 
|➜ chmod +x test.sh 

exercism/cobol/anagram 
|➜ ./test.sh 
Cobolcheck not found, try to fetch it.
curl: (56) Failure writing output to destination, passed 1369 returned 4294967295
./test.sh: line 16: ./bin/cobolcheck: No such file or directory
COMPILE AND RUN TEST
cobc: test.cob: No such file or directory

[SimaDovakin]

Exercism COBOL documentation says that you have to run bash test.sh for running tests https://exercism.org/docs/tracks/cobol/tests. Actually it does make differences because sh and bash are a little different now.

[SimaDovakin]

Doing chmod +x test.sh allows me to execute it by running ./test.sh, but the errors persist:

exercism/cobol/anagram 
|➜ chmod +x test.sh 

exercism/cobol/anagram 
|➜ ./test.sh 
Cobolcheck not found, try to fetch it.
curl: (56) Failure writing output to destination, passed 1369 returned 4294967295
./test.sh: line 16: ./bin/cobolcheck: No such file or directory
COMPILE AND RUN TEST
cobc: test.cob: No such file or directory

Hmm.. It's weird. It works for me. What's your OS?

[kapitaali]

This is Pop!_OS, an Ubuntu variant.

I am just wondering why would we need to touch Linux shell scripts at all if the original intention was to do something about Powershell.

[SimaDovakin]

We changed test.sh too to be able to fetch cobolcheck only once if it doesn't already exist in the home directory. I use Linux Mint and it works for me

[kapitaali]

We now have 1 datapoint that works and 1 that doesn't. Need more. I have a clean debian install, going to try with that. Just a sec.

[kapitaali]

Fails on debian too:

~/exercism/cobol/anagram$ ./test.sh
Cobolcheck not found, try to fetch it.
curl: (23) Failure writing output to destination
./test.sh: line 16: ./bin/cobolcheck: No such file or directory
COMPILE AND RUN TEST
cobc: test.cob: No such file or directory

[SimaDovakin]

It's really weird. It seems like you don't have permissions to write to home directory. This is my try:

[kapitaali]

I believe it works with your system, but running ./test.sh should work everywhere and be configuration independent.

Not a single user should have to make any configurations to their system to run test.sh and COBOL tests succesfully. That was how the testing script worked a while ago.

[SimaDovakin]

Could you share your test.sh please? In this case, maybe, we really should revert changes.

[kapitaali]

Yeah it's here:

#!/usr/bin/env bash
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )/
SLUG=${1:-$(basename "${SCRIPT_DIR}")}
COBOLCHECK=./bin/cobolcheck

WHICH_COBOLCHECK=$(which cobolcheck)
if [[ $? -eq 0 ]] ; then
    echo "Found cobolcheck, using $COBOLCHECK"
    COBOLCHECK=$WHICH_COBOLCHECK
elif [ ! -f $SCRIPT_DIR/bin/cobolcheck ]; then
    echo "Cobolcheck not found, try to fetch it."
    cd $SCRIPT_DIR/bin/
    bash fetch-cobolcheck
fi
cd $SCRIPT_DIR
$COBOLCHECK -p $SLUG

# compile and run
echo "COMPILE AND RUN TEST"
cobc -xj test.cob

[SimaDovakin]

This is an old version of test.sh. There is new one on the main branch.

[ErikSchierboom]

@SimaDovakin Could you look at this: https://forum.exercism.org/t/test-setup-failure-through-online-editor-for-cobol-exercise-secret-handshake/12710

[SimaDovakin]

@ErikSchierboom sure. It is weird. It works locally but it can't resolve api.github.com.

[SimaDovakin]

@SimaDovakin Could you look at this: https://forum.exercism.org/t/test-setup-failure-through-online-editor-for-cobol-exercise-secret-handshake/12710

@kapitaali @ErikSchierboom Could it be connected with DNS resolving on the server? And why?

[kapitaali]

EDIT: there was nothing wrong apparently? testing works now

[SimaDovakin]

By the errors, it seems that curl try to resolve api.github.com but it can't for some reason.

[kapitaali]

I deleted cobolcheck and ran test.sh and fetch-cobolcheck worked fine here too. So it's the test environment that is failing.

[kapitaali]

I ran tests on the website and they're working. So it must have been a temporary hiccup that is now fixed.

[SimaDovakin]

Hmm.. interesting

[kapitaali]

The exercise that I tried that worked was anagram, the one that was added last. But updating the atbash-cipher exercise on exercism.org gave me the error that was reported:

Cobolcheck not found, try to fetch it.
curl: (6) Could not resolve host: api.github.com
curl: (6) Could not resolve host: api.github.com
curl: (6) Could not resolve host: api.github.com
curl: (6) Could not resolve host: api.github.com
curl: (3) URL using bad/illegal format or missing URL
chmod: cannot access '/root/cobolcheck/cobolcheck': No such file or directory
Cobolcheck has been downloaded to /root/cobolcheck/cobolcheck
/mnt/exercism-iteration/test.sh: line 12: /root/cobolcheck/cobolcheck: No such file or directory
COMPILE AND RUN TEST
cobc: test.cob: No such file or directory  
SOMETHING WENT WRONG DURING TEST SETUP, PLEASE OPEN A TICKET AT: https://github.com/exercism/cobol/issues/new

I'm going to try downloading an updated exercise with the cli to see if this happens locally.

[kapitaali]

Running locally the updated exercise that has been downloaded with cli works fine. cobolcheck gets downloaded and tests are run.

[ErikSchierboom]

The test runner can't download anything as it does not have internet. Cobolcheck thus must be downloaded to the right location in the test runner dockerfile. Can you check that?

[SimaDovakin]

I checked that locally earlier and it works fine. I ran test runner locally in container and without it.

[ErikSchierboom]

What command did you use?

[kapitaali]

There are things that have been paused in the workflow: https://github.com/exercism/cobol/actions/runs/10566009241/workflow

[SimaDovakin]

What command did you use?

As I remember I used ./bin/run.sh ... and ./bin/run-in-docker.sh ....

[ErikSchierboom]

@SimaDovakin I think the remove of this part is what breaks the test runner. The cobol test runner downloads the cobolcheck file to: https://github.com/exercism/cobol-test-runner/blob/main/Dockerfile#L16

What I would suggest to do is:

• Either revert the WHICH part that was deleted, or
• Change the location of cobolcheck in the test runner (and also update the test.sh files in the tests/* directories to match what is used here.

[SimaDovakin]

Hmm... Ok, I'll return "WHICH" part back.

[SimaDovakin]

@ErikSchierboom @kapitaali I reverted WHICH part here #162. But I'm still wondering why the test runner can't resolve api.github.com. Is it because it doesn't have the appropriate DNS settings inside the container?