Keeping up with token limit

[doomuch]

The token limit for the Advanced Data Analyses is 8K tokens and keeping in mind system instructions, the conversation limit could run out pretty quickly. Does the length of the conversation impact its quality significantly? Do you track the token count in any way?

[spdustin]

ChatGPT rolls older messages out (but not custom instructions, normally) when the context max is in danger of being overrun. It may also do a generative summary, but as far as I know, that’s not been demonstrated conclusively. With the AutoExpert Developer Edition, the epilogue “history/symbol tree” section does a good job of keeping things moving even if context is overflowed. In Standard Edition (non-coding), it depends more on what you’re doing. The custom instructions themselves take 7% if the 8192 context buffer. The ADA system message isn’t that much more. So, IMO, there’s plenty of space.

I’ve found no reliable way of counting tokens used that doesn’t itself eat up token space. Asking ChatGPT how many tokens have been used is hit-or-miss.

[sangoi-exe]

Hello! Sorry for the intrusion, but regarding the token issue, I can be helpful.

I created a browser extension that has this functionality (token counting), among other things. It's currently outdated, but I can tweak it to help with this issue.

Btw, I have some useful information for using with ADA that can enhance the token matter and other details in your project. You might already know these details, but here we go.

First point: The active environment of ADA is shared among all ADA's conversations. For instance, if I start a new conversation, and have sent files to the environment in a previous conversation, and this environment hasn't been reset yet, the files are available in the new conversation. However, ChatGPT doesn't know about this, so one has to "instruct" it to read the files available in the Python environment to grasp the situation.

Point two: The print command from the Python environment does not consume tokens, but ChatGPT can only read something that was sent in the print if you specifically tell it to do so (I haven't figured out a way to automate this), as the print doesn't consume tokens because it's not placed in the context memory.

Point three: I haven't been able to automate this or fully test its efficacy, but it's possible to place instructions in a JSON and upload it, telling ChatGPT to follow those instructions; I use this for simpler things with the aim of saving tokens on custom instructions. I managed to "automate" something by placing in the custom instructions that the user will send him a JSON with instructions at the beginning of the conversation, and it should follow them.

Point four: Following the logic of the instruction JSON, I also made some progress trying to create a long-term memory using JSON, I instructed the ChatGPT to recurrently save our interactions in a JSON and send it to me when I request. I would upload this JSON at the beginning of a conversation when the topic is relevant to it. I haven't made much progress on this idea yet.

[SeaDude]

Cool stuff! I think the JSON memory thing you mention is what Dustin is doing with the ctags programmatically.

It's pretty fresh that you both were able to come up with functionality before OpenAI could with their own app! Hackerz!!

[spdustin]

I’d like to check that browser extension out, though, especially if you’re ignoring Jupyter cells :) Tiktoken has been compiled with emscripten, IIRC, so that could be useful…

[spdustin]

Point two: The print command from the Python environment does not consume tokens, but ChatGPT can only read something that was sent in the print if you specifically tell it to do so (I haven't figured out a way to automate this), as the print doesn't consume tokens because it's not placed in the context memory.

That’s precisely what autodev.py does (in combination with the custom instructions). It will print() instructions that ChatGPT is told to follow.

As for memory, autodev.py writes a YAML file when asked, and if it’s included in an upload in a new conversation, reads it back out and exposes it to the chat context though a combination with custom instructions.