Allow us to enable the new token-efficient-tools header please

Coming soon!

@bcherny please make it optional, I really enjoy being able to pay more for large context and tool usage.

If anything can we please make it possible to use legacy versions in case an update takes away some of this? I see tons of comments asking to fix / remove some of the features I love most

@bcherny please make it optional, I really enjoy being able to pay more for large context and tool usage. If anything can we please make it possible to use legacy versions in case an update takes away some of this? I see tons of comments asking to fix / remove some of the features I love most

Seconded

@Traddoo @SamKr
This request was just asking for the option.
The title is. "Allow us to enable", which implies an option. Otherwise I would have just said "please add the header". This also wouldn't make sense as it is a beta header at the moment.

However, I'm also confused by what you both seem to be suggesting. Are you saying this option will make results worse? To me the efficient use of tokens the settings sounds like it just won't spam the context window with irrelevant/duplicate context which as far as I'm aware would only lead to worse results.

Somewhat related parts from a recent Karpathy video. He is talking about keeping out irrelevant context. I've heard others in the past say that spamming repeated data can cause the model to focus too much on such details:

https://youtu.be/EWvNQjAaOHw?t=1009

think of the tokens in the context window as a precious resource [...] tokens in this window are expensive and they're expensive in kind of like two ways number one if you have lots of tokens here then the model can actually find it a little bit distracting [...] this is kind of like the working memory of the model the model might be distracted by all the tokens in the in the past when it is trying to sample tokens much later on so it could be distracting and it could actually decrease the accuracy of of the model [...] think of the tokens in the context window as a precious resource. think of that as the working memory of the model and don't overload it with irrelevant information and keep it as short as you can and you can expect that to work faster and slightly better

Hi @gerrywastaken,

You're absolutely right on pointing that out, I didn't mean to imply you were forcing anything onto anyone.

If it leads to the same results with less tokens, then of course that's great. It's just a matter of choice.

For anybody wondering the reason behind the delay. It turns out that things become much trickier when the tools are used with code and so this header would introduce reliability issues. Or at least that is how I'm reading it. See for yourself here:
https://x.com/_catwu/status/1900675353078792546

We've experimented with this, but unfortunately it doesn't work well for editing code in particular. We're going to keep an eye on this and flip it on in the future.

Thanks for the clear response :)

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.