Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compatibility for TES 1.1.0 #62

Open
stxue1 opened this issue May 13, 2024 · 1 comment
Open

Add compatibility for TES 1.1.0 #62

stxue1 opened this issue May 13, 2024 · 1 comment

Comments

@stxue1
Copy link

stxue1 commented May 13, 2024

py-tes on pypi currently has 2 prereleases tagged as 1.1.0. It seems like some of the requests that py-tes calls aren't working though. With py-tes==1.1.0rc2 and Funnel 0.11.0, attempting to run toil-wdl-runner with the TES batchsystem plugin results in TES seeing an unexpected argument:

Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 115, in unmarshal
    output = o(**r)
             ^^^^^^
TypeError: Input.__init__() got an unexpected keyword argument 'streamable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/bin/toil-wdl-runner", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 141, in decorated
    return decoratee(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 3156, in main
    output_bindings = toil.start(root_job)
                      ^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/common.py", line 930, in start
    return self._runMainLoop(rootJobDescription)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/common.py", line 1414, in _runMainLoop
    jobCache=self._jobCache).run()
                             ^^^^^
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 257, in run
    self.innerLoop()
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 782, in innerLoop
    self.checkForDeadlocks()
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 819, in checkForDeadlocks
    totalRunningJobs = len(self.batchSystem.getRunningBatchJobIDs())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil_batch_system_tes/src/toil_batch_system_tes/tes_batch_system.py", line 412, in getRunningBatchJobIDs
    task = self.tes.get_task(tes_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/client.py", line 190, in get_task
    return unmarshal(response.json(), Task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 111, in unmarshal
    field = _unmarshal(v, obj)
            ^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 100, in _unmarshal
    field.append(unmarshal(item, obj))
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 120, in unmarshal
    raise UnmarshalError(msg)
tes.utils.UnmarshalError: {'description': '', 'name': '', 'path': '/root/.aws', 'streamable': False, 'type': 'DIRECTORY', 'url': 'file:///home/heaucques/.aws'} could not be unmarshalled to type: Input
TypeError: Input.__init__() got an unexpected keyword argument 'streamable'

A similar result happens with 1.1.0rc1, where the service_info() function tries to request at the wrong URL:

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:8000/v1/tasks/service-info
@vsmalladi
Copy link

Agreed same thing is happening. When trying this out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants