any plan to support LLMs

[chgxtony]

any plan to support LLMs like openai / llama ? like langchain in java

[frankfliu]

We are working on LLM support, see: #2547

We have no plan to support langchain, however, if you are looking for multimodal support, you can take a look: https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/workflows.md

[sandys]

@frankfliu does #2547 need to be merged before any LLM can be used ?
i was trying to use MPT 7B. Example code here - https://github.com/arakoodev/onnx-djl-example

onnx model loads and all. But cant get it to spit answers.

[frankfliu]

@sandys

If you just want to deploy llama, you can already do it with DJLServing: deepjavalibrary/djl-serving#844

Technically, you can run LLMs with DJL in pure java fashion as well. What's missing is postprocessing and result token search. You can implement it by yourself if you want to.

[sandys]

@frankfliu hey thanks so much for your reply!
Two questions:

If you just want to deploy llama, you can already do it with DJLServing: deepjavalibrary/djl-serving#844

Llama has a licensing issue, so wondering if u have tested on any other LLMs ? like any at all ?

Technically, you can run LLMs with DJL in pure java fashion as well.

happy to try and implement (and contribute back). starting from our early rough work here - https://github.com/arakoodev/onnx-djl-example . But am not sure how to do this. I dont need any optimisation or anything at all. just get it to reply for now. any pointers how to do this ? we are attempting this with MPT-7B, but ill take anything with an open license.