Skip to content

Commit

Permalink
Apply inferface change to duplicated code in the tgis layer (#50)
Browse files Browse the repository at this point in the history
In the upstream code the `init_state()` method was removed from the
Outlines logits processor classes but the call to this method was still
in our tgis layer causing this error:

```
ERROR 06-19 16:12:57 logs.py:77] generate{input=[b'Hello, I am here to'] prefix_id= adapter_id= input_chars=[19] params=decoding { format: JSON }: 'CFGLogitsProcessor' object has no attribute 'init_state'
ERROR 06-19 16:12:57 grpc_server.py:86] Generate failed
ERROR 06-19 16:12:57 grpc_server.py:86] Traceback (most recent call last):
ERROR 06-19 16:12:57 grpc_server.py:86]   File "/opt/vllm/lib/python3.11/site-packages/vllm/entrypoints/grpc/grpc_server.py", line 103, in func_with_log
ERROR 06-19 16:12:57 grpc_server.py:86]     return await func(*args, **kwargs)
ERROR 06-19 16:12:57 grpc_server.py:86]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-19 16:12:57 grpc_server.py:86]   File "/opt/vllm/lib/python3.11/site-packages/vllm/entrypoints/grpc/grpc_server.py", line 153, in Generate
ERROR 06-19 16:12:57 grpc_server.py:86]     sampling_params, deadline = await self._validate_and_convert_params(
ERROR 06-19 16:12:57 grpc_server.py:86]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-19 16:12:57 grpc_server.py:86]   File "/opt/vllm/lib/python3.11/site-packages/vllm/entrypoints/grpc/grpc_server.py", line 430, in _validate_and_convert_params
ERROR 06-19 16:12:57 grpc_server.py:86]     await get_outlines_guided_decoding_logits_processor(decoding,
ERROR 06-19 16:12:57 grpc_server.py:86]   File "/opt/vllm/lib/python3.11/site-packages/vllm/tgis_utils/guided_decoding.py", line 44, in get_outlines_guided_decoding_logits_processor
ERROR 06-19 16:12:57 grpc_server.py:86]     logits_processor.init_state()
ERROR 06-19 16:12:57 grpc_server.py:86]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

---------

Signed-off-by: Max de Bayser <[email protected]>
  • Loading branch information
maxdebayser authored Jun 19, 2024
1 parent 7ee1c10 commit 3122d24
Showing 1 changed file with 1 addition and 7 deletions.
8 changes: 1 addition & 7 deletions vllm/tgis_utils/guided_decoding.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import asyncio
import concurrent.futures
from copy import copy
from re import escape as regex_escape
from typing import Tuple, Union

Expand Down Expand Up @@ -30,7 +29,7 @@ async def get_outlines_guided_decoding_logits_processor(
concurrent.futures.ThreadPoolExecutor(max_workers=2))
loop = asyncio.get_running_loop()

result = await loop.run_in_executor(
return await loop.run_in_executor(
outlines_decoding.global_thread_pool,
_get_logits_processor,
guide,
Expand All @@ -39,11 +38,6 @@ async def get_outlines_guided_decoding_logits_processor(
None, # guided_whitespace_pattern - TBD
)

logits_processor = copy(result)
# reset logits processor's internal state
logits_processor.init_state()
return logits_processor


def _get_guide_and_mode(
decoding_params: DecodingParameters,
Expand Down

0 comments on commit 3122d24

Please sign in to comment.