Stable upgrades

[rezzie-rich]

I want to thank you guys for all the efforts and contributions. The project has come a long way since it started, and it's making history as we speak off. However, I can't help but notice the desire for adding new features over perfecting the existing ones.

The original codeact v1 and v1.3 is surely at its peak, but since then, there have been 3 other versions, and they not only perform less but also seem to get stuck in loops and generate false outputs.

Not being able to solve an issue is still better than getting stuck in loops or, worse, giving off wrong outputs as they not only cost ineffective computation but also dilute the code base, making the debugging a nightmare.

The project already has more than enough features to make it somewhat complete. I know open source projects dont pay the bills like a regular job, so the efforts that can be given are very limited. So instead of constantly adding more features ( maybe just one more -memgpt), it would be more effective to improve and perfect the existing ones. There's no point in having newer versions of codeact one after another when they all perform less than the original version. Efforts would be more effective if all the available time is spent identifying the issue and upgrading the versions steadily where each version is out performing the previous one otherwise it will make it even harder to pin point whats causing the newer versions to get stuck.

Taking the apple's developing approach rather than the window will not only improve the overall quality but also make it more reliable. That's the key difference between a production level product and an experimental project.

You guys are heroes for your contributions to the community and dying to see the day when knowing how to use opendevin like excel/powerpoint becomes a job skill.

@rbren @xingyaoww @xcodebuild @enyst @neubig @SmartManoj @iFurySt @PierrunoYT @tobitege

[PierrunoYT]

There are lot of other which contributed to this projects but thx you for your feedback.

[rezzie-rich]

I know, but I can't tag 100+ people, lol. I meant everyone 😁

[neubig]

Hey @rezzie-rich , thanks for rooting for us and the feedback! But actually the most recent agent has the highest quality according to benchmark scores of any of the agents yet, so if you try the most recent version (esp. with clause sonnet 3.5) it may be greatly improved!

[rezzie-rich]

@xingyaoww Hmm, i agree. Some features, such as browsing and vision (even with the lack of models), are very crucial for software development and should have their foundation laid at the beginning.

Switching from bash to Python will cause some unusual issues due to obvious reasons. It would be cool if it could be modularize and be able to have both for their combined strength.

I'm working on a phi model with 4.5 tb training materials, including the codeact dataset. However, there's an issue with huggingface dataset streaming. So, I'm well familiar with instability issues with developing libraries 😭. While I'm tryna figure out a way to deal with it while having limited resources and exciting to see the results with OD, there still remain an issue with the context window (context window needs to be irrelevant 🥲🙏). Making context window irrelevant will also make local models perform similar to gpt-4o because even after having open models with up to 1m window, its not practical nor possible to run them locally on consumer grade hardware and in a practical usage context window will always raise issue as the project/codebase gets bigger.

Either way, when codeact 1.7 can run the benchmark without errors or loops like 1 and 1.3, that will be a game changer.

[xingyaoww]

We actually having @enyst working to resolve the context window issue! The idea is to condense the history whenever the agent exceed the limit. We will need to evaluate it before merge it in, but hopefully we will get that in soon!

#2021

[rezzie-rich]

It will be interesting to see the results after the merge. Though I'm more lenient towards memgpt since it's specialized on memory management, plus it's an already built project with massive community support, I'm very curious to see what you guys are cooking.

There's also another aproch i came across where there's 3 states of memory instead of 2, short-term, long-term, and working memory; i think it has higher potential since it might be able to handle memory better when the model hits the context window limit.

[enyst]

@rezzie-rich I completely agree on the context window. IMHO the context window issue won't go away just because 1 million tokens windows exist, specially since there seems to be some evidence out there that models "forget" some stuff the more input they get, or at a minimum, it seems harder to find the sweet spot for precision.

On a side note, IMHO this issue has even a strange relationship with the stuck issue, because our attempts so far to stop the LLM when it goes into an endless loop do not seem to make it take another approach, when it has too much of that endless attempt in the context window. LLMs are autocompletion machines after all, too much of a thing seems to continuously push them to that thing, so it gets harder to stop it in a way. We are working to improve this. As of now, we're getting closer to evaluate the context window solution on its own, then we can iterate upon to improve this.

[rezzie-rich]

Yeah, and longer the window, worse a model performs. Having smaller windows helps it to modularize the output for better recall than to let the key infos get lost in an endless context.

Having a way to flag the faulty outputs and using it during parallel processing might help it to avoid taking the same route and get stuck in a loop.

[enyst]

@rezzie-rich Thank you for the feedback! I was initially a bit confused why you say 1.3 is still better, and I think the with-hint vs no-hint issue explains it. It really matters. As best as I can tell, I think it's very likely that currently 1.7 is better.

The stuck issue is definitely a pain point, and we're working on it. If you can, it would help immensely if you use the 'thumbs up/down' button to send feedback to developers, when that happens for you. Because it sends us the history of that chat (don't use it if you don't want that sent), and the more real-life histories we have, the more it helps us detect the patterns of behavior that make CodeAct prone to it. Thus it helps fixing them.

[rezzie-rich]

I haven't tried the 1.7 yet. My feedback was initially based on the benchmark results. So far, only v1 and v1.3 finishes the test without errors and loops, figuring out what causes that might be the game changer since it's been constant with all 3 versions since 1.3

[rezzie-rich]

If im correct, even 1.3 had that issue until it wasn't anymore

[enyst]

@xingyaoww can correct me if wrong, I think we weren't yet computing errors and stuck at the first passes through v1 and v1.3, or were we? I thought we literally don't have those results, it's not 0, it's 'unknown'.

[xingyaoww]

Yep! We were basically not tracking loop & stuck issue back in v1.3. Let me see if i can update benchmark to better reflect that for confusion

[rezzie-rich]

Hmm