Sheep compatiability #21

shardros · 2022-08-17T08:43:24Z

This PR will be merged when there is all of the parts which sheep needs to run working.

fenjalien · 2022-08-19T17:32:12Z

Running this branch on my pi produces this error when the start button is pressed in the editor:

INFO:     127.0.0.1:44546 - "POST /upload/upload HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 394, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 216, in app
    raw_response = await run_endpoint_function(
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/fastapi/routing.py", line 151, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/home/pi/.cache/pypoetry/virtualenvs/shepherd-2-TGIJcPDa-py3.10/lib/python3.10/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/home/pi/.pyenv/versions/3.10.6/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/pi/shepherd-2/./app/routers.py", line 55, in upload_file
    app.upload.process_uploaded_file(uploaded_file)
  File "/home/pi/shepherd-2/./app/upload.py", line 110, in process_uploaded_file
    shutil.rmtree(config.round_path)
  File "/home/pi/.pyenv/versions/3.10.6/lib/python3.10/shutil.py", line 714, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/home/pi/.pyenv/versions/3.10.6/lib/python3.10/shutil.py", line 712, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/shepherd-2/usercode/round'

The later run request gives a 409

INFO:     127.0.0.1:49338 - "POST /run/start HTTP/1.1" 409 Conflict

The output of running pytest is

test/test_editor.py ..                                                             [ 20%]
test/test_home.py .                                                                [ 30%]
test/test_runner.py x..F                                                           [ 70%]
test/test_upload.py FFF                                                            [100%]

It looks like the failures are also due to the directory usercode/round not exsisting, what is the purpose of this folder @shardros ?

shardros · 2022-08-19T20:35:18Z

Fixed and tested on fresh install

shardros · 2022-08-19T20:43:02Z

It looks like the failures are also due to the directory usercode/round not exsisting, what is the purpose of this folder @shardros ?

This is correct. This is the folder where the usercode which is to be run in the next round is uploaded to. It is now automatically created by shepherd. Logs aren't done in this pr but you can see that the code works by looking logs.txt

fenjalien · 2022-08-20T13:29:56Z

This is the folder where the usercode which is to be run in the next round is uploaded to.

Ah so usercode/ is all the code that has been written by the user and usercode/round/ is the code that will be run when /run/start is hit?

shardros · 2022-08-20T13:30:42Z

This is the folder where the usercode which is to be run in the next round is uploaded to.

Ah so usercode/ is all the code that has been written by the user and usercode/round/ is the code that will be run when /run/start is hit?

Correct :)

fenjalien

From running the tests:

test/test_editor.py ..        [ 20%]
test/test_home.py .         [ 30%]
test/test_runner.py xF..   [ 70%]
test/test_upload.py .FF    [100%]

test_start fails at line 20 due to a timeout. Looks like it comes from this line:

type(runner).state.__set__(runner, States.RUNNING)

Why is it like this? In fact all of the rest of the tests fail due to this.

app/editor.py

app/config.py

app/routers.py

install.sh

fenjalien · 2022-08-20T15:35:15Z

I've just tried changing this:

shepherd-2/app/routers.py

Line 33 in 18c9b54

type(runner).state.__set__(runner, States.RUNNING)

to this:

runner.state = States.RUNNING

and all tests pass apart from larg_zip_upload by timeout. I'm assuming this is due to the shear size its timeout will need to be much longer.

This test looks at the logs and checks they are as they should be. However if shepherd crashes in this test then the usercode continues running and continues writing to the logs. This commit uses context managers to harden against this but does not completely solve the problem

…d-2 into sheep-compatiability

shardros · 2022-08-28T19:40:13Z

@fenjalien ready for another round of review

shardros · 2022-08-28T19:42:19Z

You can still retroactively do the code review but we need to move fast

serve sheep and docs

9902110

This was referenced Aug 17, 2022

Serve sheep #7

Closed

Serve Docs #8

Closed

shardros added 3 commits August 17, 2022 10:28

Added install script

3878147

make script gracefully fail

3a188cd

tests for files/

9837d53

fenjalien added this to the Shepherd-1 Feature parity milestone Aug 19, 2022

shardros added 2 commits August 19, 2022 15:52

uploading and running compatiability with sheep

e13f469

add tests for creation and deletion

d328fc8

This was linked to issues Aug 19, 2022

Upload parity with shepherd-1 #2

Closed

Run parity with shepherd-1 #3

Closed

shardros marked this pull request as ready for review August 19, 2022 16:42

shardros added 2 commits August 19, 2022 17:46

include test_editor.py

4144d52

include test_editor.py

6a7bded

shardros added 4 commits August 19, 2022 20:32

automate python install in install.sh

6caa63d

fetch new packages

9a6e12b

Document install.sh

f4b54c9

fix problems with running on fresh install

ddc7604

allow install.sh to fix broken env

18c9b54

fenjalien requested changes Aug 20, 2022

View reviewed changes

shardros and others added 4 commits August 20, 2022 17:09

fix tests

361fdca

basic framework for starting usercode (#27)

5d1e0f6

create requirements.txt

8f57339

remove hashes from requirements.txt

1e6ac46

shardros and others added 18 commits August 23, 2022 17:41

add logging

f67c9c9

add logging convience method

039d580

automate python install in install.sh

018ebb6

fetch new packages

37326af

Document install.sh

75dbbb6

fix problems with running on fresh install

fe1f957

allow install.sh to fix broken env

8dd50b2

fix tests

0f45d41

basic framework for starting usercode (#27)

45214cd

create requirements.txt

e16d673

remove hashes from requirements.txt

0780c3b

add logging

b699d8a

add logging convience method

41decea

Merge branch 'sheep-compatiability' of github.com:systemetric/shepher…

2341afa

…d-2 into sheep-compatiability

kill user code in tests using psutil

3ac9c33

Tidy up for review

980520f

shardros requested a review from fenjalien August 28, 2022 18:00

shardros merged commit 8491081 into master Aug 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sheep compatiability #21

Sheep compatiability #21

shardros commented Aug 17, 2022 •

edited

Loading

fenjalien commented Aug 19, 2022

shardros commented Aug 19, 2022

shardros commented Aug 19, 2022

fenjalien commented Aug 20, 2022

shardros commented Aug 20, 2022

fenjalien left a comment

fenjalien commented Aug 20, 2022

shardros commented Aug 28, 2022

shardros commented Aug 28, 2022

Sheep compatiability #21

Sheep compatiability #21

Conversation

shardros commented Aug 17, 2022 • edited Loading

fenjalien commented Aug 19, 2022

shardros commented Aug 19, 2022

shardros commented Aug 19, 2022

fenjalien commented Aug 20, 2022

shardros commented Aug 20, 2022

fenjalien left a comment

Choose a reason for hiding this comment

fenjalien commented Aug 20, 2022

shardros commented Aug 28, 2022

shardros commented Aug 28, 2022

shardros commented Aug 17, 2022 •

edited

Loading