honestly I’m not sold here
this is a phenomenal write up of sound tactics for minimizing cost and I appreciate the nuggets from Thariq
but it’s like a laundry list of “how we make context worse to make requests cheaper”
and I don’t know I just can’t shake the feeling that this will be seen as an anti-pattern in the next generation of agents
I’ve written about this before, if I had to bet on one wrong slice of current conventional wisdom this would be it
I don’t think it’s obvious that saving 90% $ on particularly long requests at the expense of significantly worse context rot will actually save you tokens in the long run. it will introduce more problems into your codebase which will in turn require more tokens to root out. >10x in some cases, and now you’ve saved nothing at all.
Will Manidis wrote a great article last week on “Tool Shaped Objects” and this feels like one of those things where, sure, per-request is cheaper and therefore I can send more requests and feel like I’m doing more work. but a cleaner context at the outset would simply have me sending fewer requests to start
I also can’t help but shake the feeling that this caching economics benefits the lab more than it does the end user, even if they do pass along some amount of the cache cost savings. this is of course particularly true for flat rate plans like Claude Max.
anyone on a flat plan is not seeing a dime of the savings that their worse context is allegedly providing, which means you’re getting a lower quality product for the same price, the lab just takes more margin at the expense of quality
I think designing your context to take advantage of caching is a key factor, but certainly not the most important factor, as described here
- honestly I’m not sold here this is a phenomenal write up of sound tactics for minimizing cost and I appreciate the nuggets from Thariq but it’s like a laundry list of “how we make context worse to make requests cheaper” and I don’t know I just can’t shake the feeling that this will be seen as an anti-pattern in the next generation of agents I’ve written about this before, if I had to bet on one wrong slice of current conventional wisdom this would be it I don’t think it’s obvious that saving 90% $ on particularly long requests at the expense of significantly worse context rot will actually save you tokens in the long run. it will introduce more problems into your codebase which will in turn require more tokens to root out. >10x in some cases, and now you’ve saved nothing at all. Will Manidis wrote a great article last week on “Tool Shaped Objects” and this feels like one of those things where, sure, per-request is cheaper and therefore I can send more requests and feel like I’m doing more work. but a cleaner context at the outset would simply have me sending fewer requests to start I also can’t help but shake the feeling that this caching economics benefits the lab more than it does the end user, even if they do pass along some amount of the cache cost savings. this is of course particularly true for flat rate plans like Claude Max. anyone on a flat plan is not seeing a dime of the savings that their worse context is allegedly providing, which means you’re getting a lower quality product for the same price, the lab just takes more margin at the expense of quality I think designing your context to take advantage of caching is a key factor, but certainly not the most important factor, as described here
