Prompt cache hit rates from Langfuse
If you trace your agents with Langfuse, it already stores how many tokens each model call read from the cache and wrote to it. So you can see your real hit rate on Bedrock without turning on invocation logging. I traced Strands, LiteLLM and LangChain into a real Langfuse server (Langfuse 4.64, October 2026) and checked what it stores for the same two calls: one that writes the cache and one that reads it.
The short version: most setups store the right numbers. Two don't. Strands before 1.54 makes the cache look perfect, and LiteLLM's langfuse callback counts cache writes twice. If you just add up Langfuse's numbers for those two, you get the wrong hit rate.
The setup I'd use
pip install cachecanary export LANGFUSE_PUBLIC_KEY=pk-lf-... export LANGFUSE_SECRET_KEY=sk-lf-... # EU: https://cloud.langfuse.com, or your own server export LANGFUSE_HOST=https://us.cloud.langfuse.com cachecanary langfuse --since 7d --environment production
Use the API keys of the Langfuse project you want to check. CacheCanary uses only token counts, model names and timestamps, and stores nothing. On Langfuse v4 it asks only for those, so prompts and outputs aren't downloaded; older self-hosted servers send whole records, and CacheCanary ignores the rest. It checks each call's numbers against its total, fixes the cases below, and tells you how many calls it fixed. Add --min-hit 0.8 to fail when the hit rate drops under 80%.
What I found
Every setup below made the same two calls to Bedrock: the first wrote 5,848 tokens to the cache, the second read them back, and each had 10 more tokens of new input and 4 of output. So the right answer is the same for all of them: half the cached part was read, a 50% hit rate.
| Sent by | What Langfuse stored for the read call | Hit rate if you just add it up |
|---|---|---|
| Strands 1.61 | input 10, cache read 5,848 | 50% (right) |
| Strands 1.53 (and older) | input 0, cache read 5,848 | 100% |
LiteLLM langfuse callback | input 10, cache read 5,848 | 33% |
LiteLLM langfuse_otel | input 10, cache read 5,848 | 50% (right) |
LangChain (ChatBedrockConverse) | input 10, cache read 5,848 | 50% (right) |
The LiteLLM callback's read call is fine; its write call is the one that's off (finding 2). CacheCanary gives 50% for all five.
1. Strands before 1.54 makes the cache look perfect
Two things go wrong, and both push the hit rate up:
- The input shows as 0. Bedrock reports its input count with the cached tokens already taken out. Strands sent that number to Langfuse as is, and Langfuse takes the cached tokens out again. So on every call that reads the cache, the input comes out as 0.
- Cache writes go missing. Strands reported cache writes only under
cache_write_input_tokens, a name Langfuse doesn't recognise. Langfuse keeps the number but doesn't treat it as a cache write, and its price list has no price for it. A tool that reads Langfuse's usual cache fields never sees the writes.
Together they show 100% in my test, where the truth is 50%. Every Strands version with cache tracking up to 1.53 does this; 1.54 (August 27, 2026) fixed both. Langfuse's own cost figure for those calls comes out low too, because it prices the same numbers.
The total Strands sends is still right, though. Bedrock's total includes the cached tokens, so CacheCanary reads the writes from the name Strands used and rebuilds the real input as the total minus output, cache reads and cache writes.
2. LiteLLM's langfuse callback counts cache writes twice
On the call that writes the cache, LiteLLM's langfuse callback (LiteLLM 1.106) stores input as 5,858: the 10 new tokens plus the 5,848 written to the cache. It also stores those 5,848 again as the cache write. So the same tokens are counted twice, and the hit rate comes out at 33% instead of 50%. LiteLLM's other integration, langfuse_otel, stores the right numbers. If you can switch, switch.
The callback also logs the model as converse/us.anthropic.claude-sonnet-4-6. CacheCanary takes LiteLLM's prefix off, so one model stays one row and a global. model keeps its lower price.
3. Agent steps repeat the numbers of the calls inside them
Strands puts the summed usage of a whole agent run on the agent step as well, and so does the Vercel AI SDK. Add up every step and you count each token twice. CacheCanary only counts the model calls (Langfuse calls them generations).
Smaller things
- When an integration sends no total, Langfuse makes one by adding up the other numbers. A total like that can't catch a mistake, so CacheCanary takes those calls as they are. Hand-written Langfuse calls often look like this.
- If an integration sends no token counts at all, Langfuse estimates them from the text, with no cache numbers. Calls with no cache fields are counted as uncached, and the output tells you how many there were, because nobody can tell "caching was off" from "the integration doesn't report it".
- On Langfuse v4 (Cloud and self-hosted), CacheCanary uses the faster v2 observations API. On older self-hosted servers it falls back to the v1 API, which is slower: it reads 100 calls at a time instead of 1,000.
- Prices are Bedrock's list prices. A model name without an endpoint prefix is priced as an in-Region model. If you pay a different rate, add
--price.
What Langfuse gets right
It stores the cache counts for every call, from almost every integration, and keeps the total the integration sent next to them. That total is what makes the fixes above possible. LangChain, LiteLLM's langfuse_otel and current Strands all store the right numbers.
Check your own project
Run it against your own project with the command above. To see the output first, the repo has the real records from the five setups in the table. This is the real output on them:
$ cachecanary langfuse --file tests/fixtures/langfuse_observations_2026-10.json --min-hit 0.8 us.anthropic.claude-sonnet-4-6: hit 50% over 10 calls (read 29240, write 29240, uncached 100, no usage 0) <-- below threshold input cost $0.13 at list price; about $0.07 of it lost to cache misses (target: 80% hit rate) List prices: Amazon Bedrock on-demand, Oct 2026 (global. IDs at list, others +10%). Use --price for your own rate. Read 10 generations. 2 generation(s): Langfuse's input count did not add up with the total that was sent, so the uncached input was rebuilt from the total (seen with Strands before 1.54 and LiteLLM's langfuse callback).
In plain words: across all ten calls, half the cached part was read, so the hit rate is 50%, under the 80% target, and the command fails. Two calls didn't add up in Langfuse (old Strands and the LiteLLM callback), and CacheCanary fixed them from their totals.
To get an alert when your hit rate drops, run it on a schedule in GitHub Actions with your Langfuse keys as secrets. The setup is in the README.
How I tested this
I ran a real Langfuse 4.64 server with its official Docker setup. Strands 1.61.1 and 1.53.0, LiteLLM 1.106.0 (its langfuse callback and langfuse_otel) and langchain-aws 1.9.0 with Langfuse's LangChain handler each made two calls to a small local server standing in for Bedrock. That server returned the token counts from a real Bedrock capture: 10 new input tokens, 4 output and 5,848 cached, 5,862 in total. Each library sent its traces to Langfuse the normal way, and I read them back through the Langfuse API. The records are in the repo, and the tests check that CacheCanary gives 50% for each setup. I also ran it on a production Langfuse Cloud project, and its numbers matched a separate count of the raw data for every model. The scripts are in the repo: langfuse_capture.
Framework guides: LiteLLM, Strands Agents and LangChain. More on why caching breaks on Bedrock: Five ways Claude prompt caching quietly breaks on Amazon Bedrock.
Want a quick number for your own app? Paste three totals from CloudWatch into the cache checker. It runs in your browser, nothing to install.
Using CacheCanary? I'd love to hear how it's going, good or bad, and whether you'd want the dashboard. Email me at hello@cachecanary.com. I read every message.