How AgentSight reconciles overlapping token usage

The short answer

In v1.0.31, the token report is a projection over AgentSight's materialized view. If you point the report at a saved SQLite capture, it loads that run. With no explicit DB, AgentSight chooses the latest localagentsight-*.db; if none exists, it warns and falls back to recent agent-native sessions.--local forces the native-session path.

The report then calls effective_tokens() before aggregation. That function performs two kinds of reconciliation: source precedence for rows that resolve to the same selection key, and a special Gemini network-versus-stdout aggregate rule. Only the resulting effective rows feed the model/provider/process/directory summaries.

Raw token rows and report totals are different objects

AgentSight can learn token usage at several boundaries. A captured model response can carry provider usage. An otherwise unpaired response can still expose usage. Gemini CLI can print aggregate statistics. Claude telemetry can contain usage. Local agent transcripts can contribute session-level usage. Keeping these as separate observations preserves provenance, while the report layer can reconcile overlap that it can identify safely.

The materialized view therefore keeps raw token-usage rows and derives an effective set for reporting. The exported snapshot summary and token_summary both use that effective set, so the reconciliation rule is shared rather than being a formatting trick specific to the CLI table.

Source precedence is keyed, not a universal join

The v1.0.31 source priority used when two rows share the same selection key is explicit:

Priority	Source	Interpretation
1	`response_usage`	Usage attached to a network-observed model response.
2	`orphan_response_usage`	Response usage observed even when the normal request/call correlation is incomplete.
3	`gemini_cli_stdout_stats`	Gemini CLI aggregate statistics, also subject to the separate aggregate rule below.
4	`claude_telemetry`	Usage from Claude telemetry.
5	agent-native session	Usage imported from a provider-native session.
6	other / unrecognized source	Any source not matched by the named cases above.

For non-Gemini-stdout rows, the selection key is llm_call_id when it is non-empty; otherwise the row ID is used. For Gemini stdout rows, AgentSight builds a synthetic key from PID and model. Within one key, the lower numeric source priority wins. Equal-priority rows are broken by higher confidence and then a stable ID ordering.

The code comment says network-observed response usage is the primary fact source and native session logs are intended to enrich or backfill when no network call was captured. But v1.0.31 does not generally re-key a native session's {session_id}-{model} row to a live response's call ID. If the keys differ, both rows can survive effective_tokens(). Treat source precedence as keyed de-duplication, not as proof that every network/native copy has been correlated and removed.

Gemini needs an aggregate-level reconciliation step

Gemini exposes a harder case because the same process/model pair can have per-response network usage and a CLI stdout total. Those two shapes are not naturally keyed to the same call. AgentSight first computes, for each(pid, model), the sum of network response totals and the maximum observed Gemini stdout total.

If the network sum is at least as large as the stdout total, the stdout aggregate is dropped. If the stdout total is larger, the network rows for that process/model are dropped and the largest stdout total is retained. Smaller intermediate stdout totals are also discarded. This rule is deliberately separate from the call-key precedence above: it chooses one accounting path for this Gemini aggregate instead of adding both views.

Local-session fallback changes the evidence source, not the command

The report loader has one important branch. A DB path loads the saved SQLite view. Without a DB, native mode creates an agent-native materialized view and imports a bounded set of recent sessions—25 in the current implementation. That makes agentsight report token useful even when you did not run a live eBPF recording first.

The reusable agent-session layer normalizes model usage and token totals from provider-native histories. That does not make every agent version equally observable. For example, the current Cursor notes state that Cursor stopped recording per-turn usage locally around March 2026: older sessions that contain usage events can show token totals, while newer sessions can legitimately show none. A missing total in that situation is not evidence that AgentSight lost a captured response.

Codex v1.0.31 baseline: cumulative-session and response-level paths

Codex session JSONL contains token_count events. For the native session's cumulative token summary, v1.0.31 codex_total_token_usage() walks the transcript from the end and returns the latestinfo.total_token_usage object it can parse. It does not add every token-count event as a new billable increment.

There is a separate response-parsing path: when AgentSight constructs the latest parsed LLM response and that response does not already carry a total, the parser can use last_token_usage as a fallback. That fallback does not make last_token_usage the cumulative native-session total. Keeping the two paths distinct is important when a rollout contains one shape but not the other.

Codex discovery can also use ~/.codex/state_5.sqlite. The v1.0.31 native source queries the recentthreads rows, takes the source-reported tokens_used, and prefers a rollout-derived cumulative usage when the rollout summary is available. That state database is a bounded discovery/index path, not another token total that should be added independently to the hydrated session.

v1.0.35 update: why a live Codex token total once lagged the rollout

This field study updates the earlier v1.0.31 accounting analysis using the v1.0.35 session parserand the source, regressions and first-party run evidence in AgentSight PR #224. When Codex writes both legacy event_msg/token_count and newertoken_usage_record entries, which number should a live session show? It should show the latest usable cumulative thread total: not the sum of all usage records, and not an older legacy total merely because it was encountered first.

Codex 0.153+ can emit token_usage_record entries containingthread_token_usage (cumulative) and usage (per-response or per-turn). Legacy event_msg/token_count events may coexist in one rollout. AgentSight has a forward full-session parser and a reverse-tail summary for a changing rollout. Before the correction, a mixed-format regression could give the tail a stale 100-token value when the full parser reported 130. This was a precedence bug, not evidence of a second 30-token purchase.

Rollout observation, in order	Meaning	v1.0.35 selection
Legacy `total_token_usage = 100`	Older cumulative snapshot.	May be superseded by a newer cumulative record.
Newer `thread_token_usage = 130`	Current thread total; its adjacent `usage` is separate.	Choose 130 for the cumulative native-session total.
Trailing legacy empty cumulative object and `last_token_usage = 30`	No new usable total, only a turn-level fallback.	Do not overwrite 130 with 30 or zero.
Null/empty thread counter with nonempty `usage` or `turn_token_usage`	Response/turn usage, not proven cumulative.	Fallback only if no stronger cumulative total exists.

The reverse-tail helper codex_total_token_usage() scans JSONL backwards. It returns the first valid thread_token_usage or legacy total_token_usage it finds, and saves per-response/turn values as weaker fallbacks. A valid token object must havetotal_tokens > 0. The forward parser updates cumulative state when it sees a newer valid record, but does not let response-only usage overwrite an established thread total. This is a format-aware selection rule, not simply taking the maximum integer from the file.

A public, testable mixed-format example

The checked-in regression test builds a synthetic rollout with a legacy 100-token counter, a later cumulative 130-token thread_token_usage, and a trailing empty total plus 30-token last-turn field. Both full parsing and reverse-tail parsing must return 130. The same fixture proves cache accounting: a source input count of 100 includes 4 cached tokens, so AgentSight normalizes the aggregate as 96 noncached input + 4 cache read + 30 output = 130 total.

A second test places cumulative thread records at 15 and then 30. The correct session total is 30, not 45: later cumulative observations replace earlier snapshots. A third fixture keeps two per-response values of 15 while the cumulative thread becomes 30. Response attribution remains attached to its own response; the session total is not redefined by summing or substituting one response record. Null and empty thread counters have separate fallback tests.

cargo test --manifest-path ext/session/Cargo.toml codex_newer_record_updates_totals_after_token_count
cargo test --manifest-path ext/session/Cargo.toml codex_record_response_usage_does_not_replace_thread_total
cargo test --manifest-path ext/session/Cargo.toml codex_record_without_thread_total_uses_nonempty_fallback

These are public parser tests pinned to the v1.0.35 product commit, not anonymized private user transcripts. The source fixture also checks that codex_total_token_usage(content) agrees with the full parse_session_content() projection. Re-run against the pinned commit and inspect the exact assertions before adapting the method to a new Codex rollout format.

Compare live, saved and browser totals without treating them as separate meters

PR #224 documents an isolated real Codex 0.160.0 run, distinct from the synthetic fixtures. Its final native rollout reported 87,106 tokens: 86,904 input (including 79,616 cached) and 202 output. The PR author reports identical source totals in the served snapshot,report summary and report token --json; the browser rounded that to 87.1k. This is first-party product validation recorded by that PR, not a benchmark repeated in this article.

For a session you are authorized to inspect, pin the process or session identity and compareagentsight top --plain --once -p <codex-pid> --limit 20 withagentsight report summary andagentsight report --local token --json over the same native-session source. Verify the cached-input breakdown as well as the total. A process summary, a recent-local-session report and a SQLite-recording report can select different runs; matching model names alone do not make them comparable.

The views agree because they can share one provider-native transcript; their agreement is useful for detecting parsing/projection errors, but is not an independent provider invoice or proof of transcript completeness. A rotated or partial rollout, missing usage fields, runtime incompatibility, or an incomplete capture boundary can still leave a gap. No dollar amount, quota-window usage or complete BPF coverage is implied by these token totals.

Keep three objects distinct: the running session cumulative total,per-response usage, and the effective report rows after wider key/Gemini reconciliation. The Codex v1.0.35 change repairs the first two across mixed formats; it does not turn all network and native observations into one perfectly joined billing ledger.

Grouping happens after reconciliation

Once effective rows are selected, the token report aggregates input, output, cache-creation, cache-read, and total-token fields. The default key is model. --group-by provider groups by provider,comm by process command, pid by process ID, and dir (alsocwd or directory) by the best available session or process working directory.

Session counts are de-duplicated separately from call counts. A group can therefore represent many calls from one session, and the number of token observations is not itself the number of sessions. Unknown grouping metadata is reported as unknown rather than silently reassigned.

Three totals that should not be conflated

Signal	What it means	What it does not prove
Observed/session token usage	Token fields exposed by the selected model-response, telemetry, CLI, or native-session evidence.	An invoice amount or provider subscription quota.
Source-reported capacity window	Provider/agent metadata about a usage or subscription window when a supported source exposes it.	Usage inferred from AgentSight token totals.
Agent Flamegraph token width	Offline semantic aggregation of reported token counts, with bounded estimates only when that profiling path permits them.	A replacement for the raw report or an exact currency cost.

Provider pricing, cache discounts, plan credits, and subscription windows change independently from the accounting rows above. If you need dollars, preserve the token-kind breakdown and join it to a dated pricing source separately. Do not relabel total_tokens as cost.

A reproducible token check takes three queries

Start from one explicit saved run when you are debugging a recorded workload. Compare the default model view with provider and working-directory views, and keep JSON output when another tool will consume the result.

agentsight report token --db ./agentsight-run.db --json
agentsight report token --db ./agentsight-run.db --group-by provider
agentsight report token --db ./agentsight-run.db --group-by dir

# Deliberately ignore saved DBs and use supported local agent histories
agentsight report --local token --group-by model --json

If a total looks surprising, inspect the run's source provenance before doing arithmetic by hand. Rows that share a selection key can replace one another according to source priority; rows with different keys can both survive. Separately, Gemini stdout can replace a smaller network aggregate for the same PID/model. The useful debugging questions are therefore “which rows shared a key?”, “which source won that key?”, and “did the Gemini aggregate rule apply?”

What a zero or missing number can actually mean

A missing usage field can mean the provider did not persist it, the local-session format no longer includes it, a live capture did not observe the response boundary, or the selected source genuinely reported no value. These cases are not interchangeable. The effective-token rules remove overlap in the cases they explicitly reconcile, but they do not manufacture missing accounting or provide a universal cross-source correlation.

Keep negative claims scoped to the evidence source. “This report has no token total for the session” is defensible. “The agent used zero tokens” usually is not unless the source contract itself establishes that.

How to verify the implementation yourself

Read collector/src/main.rs for CLI and DB/native selection, then cli_db.rs for materialized-view loading. The keyed and Gemini reconciliation rules are in effective_tokens() and the adjacent source-priority function. Inspect agent_native.rs for native-session row construction and Codex state_5.sqlite discovery, then ext/session/src/parser.rs for the provider-native token shapes and cumulative Codex helper.

For an investigation, record the AgentSight version, whether the input was a DB or native sessions, the group key, and the JSON result. Those details are enough for another reviewer to distinguish source reconciliation from a real change in the underlying agent usage.