LFX Mentorship Autumn 2024 Pretest - for #126

[hsj576]

Introduction

For those who want to apply for LFX mentorship for #126, this is a selection test for the application. This LFX mentorship aims to implement cloud-edge collaborative speculative decoding based on KubeEdge-Ianvs, an open source cloud-edge collaborative distributed machine learning platform, so as to further improve the LLM inference speed in cloud-edge environment. Based on Ianvs, we designed this pretest to evaluate the candidates.

Requirements

Each applicant of LFX Mentorship can try out the task and gain a score according to completeness. Finally, the one with the highest score will successfully become the mentee of this LFX Mentorship project. All the titles of task output such as pull requests (PRs) should be prefixed with LFX Mentorship Autumn 2024 Pretest.

Task

Content

1. Create a new example on Kubeedge Ianvs for LLM benchmark.
   • You can choose any LLM benchmark dataset to use, such as MT-bench, Humaneval, etc.
2. Implement a cloud-edge collaborative algorithm based on the LLM benchmark example.
   • You can choose to implement any cloud-edge collaborative algorithm, including but not limited to joint-inference, speculative decoding, etc.
3. Submit an experiment report as a PR which includes algorithm design, experiment results and a README document.
   • The README document aims to show instructions for testing and verifying the submitted example for reviewers. An example is available at unseen task recognition readme document.
   • An example of the algorithm design is available at unseen task recognition proposal.
   • An example of the experiment results is available at the leaderboard of single task learning.

Resources

• Ianvs installation guides
• User interfaces to config
• Reference Example: LLM-benchmark

Rating

1. Each item will be scored based on completion and code quality, and the sum of the scores will be the total score for this task.
2. For the examples that can not be run successfully directly through the submitted code and the README instruction document, the total score will be 0 in content 1, 2. So, be cautious about the code and docs!

Item	Score
Content 1	50
Content 2	25
Content 3	25

Deadline

According to the timeline of LFX mentorship Autumn 2024, the admission decision deadline is August 27th, 5:00 PM PDT. Since we have to process the internal review and decide, the final date for PR submissions of the pretest will be August 24th, 5:00 PM PDT.

[aryan0931]

hello @hsj576 I am facing this issue while installing ianvs, please suggest some possible solutions

[FuryMartin]

Hi, I have implemented a demo. Please see details at here

[Sid260303]

hello @hsj576 I am facing this issue while installing ianvs, please suggest some possible solutions

Hey @hsj576 , I am having similar issues but having ModuleNotFoundError: No module named 'colorlog'