How AgentSight reconciles overlapping token usage

The short answer

In v1.0.31, the token report is a projection over AgentSight's materialized view. If you point the report at a saved SQLite capture, it loads that run. With no explicit DB, AgentSight chooses the latest localagentsight-*.db; if none exists, it warns and falls back to recent agent-native sessions.--local forces the native-session path.

The report then calls effective_tokens() before aggregation. That function performs two kinds of reconciliation: source precedence for rows that resolve to the same selection key, and a special Gemini network-versus-stdout aggregate rule. Only the resulting effective rows feed the model/provider/process/directory summaries.

Raw token rows and report totals are different objects

AgentSight can learn token usage at several boundaries. A captured model response can carry provider usage. An otherwise unpaired response can still expose usage. Gemini CLI can print aggregate statistics. Claude telemetry can contain usage. Local agent transcripts can contribute session-level usage. Keeping these as separate observations preserves provenance, while the report layer can reconcile overlap that it can identify safely.

The materialized view therefore keeps raw token-usage rows and derives an effective set for reporting. The exported snapshot summary and token_summary both use that effective set, so the reconciliation rule is shared rather than being a formatting trick specific to the CLI table.

Source precedence is keyed, not a universal join

The v1.0.31 source priority used when two rows share the same selection key is explicit:

Priority	Source	Interpretation
1	`response_usage`	Usage attached to a network-observed model response.
2	`orphan_response_usage`	Response usage observed even when the normal request/call correlation is incomplete.
3	`gemini_cli_stdout_stats`	Gemini CLI aggregate statistics, also subject to the separate aggregate rule below.
4	`claude_telemetry`	Usage from Claude telemetry.
5	agent-native session	Usage imported from a provider-native session.
6	other / unrecognized source	Any source not matched by the named cases above.

For non-Gemini-stdout rows, the selection key is llm_call_id when it is non-empty; otherwise the row ID is used. For Gemini stdout rows, AgentSight builds a synthetic key from PID and model. Within one key, the lower numeric source priority wins. Equal-priority rows are broken by higher confidence and then a stable ID ordering.

The code comment says network-observed response usage is the primary fact source and native session logs are intended to enrich or backfill when no network call was captured. But v1.0.31 does not generally re-key a native session's {session_id}-{model}row to a live response's call ID. If the keys differ, both rows can survive effective_tokens(). Treat source precedence as keyed de-duplication, not as proof that every network/native copy has been correlated and removed.

Gemini needs an aggregate-level reconciliation step

Gemini exposes a harder case because the same process/model pair can have per-response network usage and a CLI stdout total. Those two shapes are not naturally keyed to the same call. AgentSight first computes, for each(pid, model), the sum of network response totals and the maximum observed Gemini stdout total.

If the network sum is at least as large as the stdout total, the stdout aggregate is dropped. If the stdout total is larger, the network rows for that process/model are dropped and the largest stdout total is retained. Smaller intermediate stdout totals are also discarded. This rule is deliberately separate from the call-key precedence above: it chooses one accounting path for this Gemini aggregate instead of adding both views.

Local-session fallback changes the evidence source, not the command

The report loader has one important branch. A DB path loads the saved SQLite view. Without a DB, native mode creates an agent-native materialized view and imports a bounded set of recent sessions—25 in the current implementation. That makes agentsight report token useful even when you did not run a live eBPF recording first.

The reusable agent-session layer normalizes model usage and token totals from provider-native histories. That does not make every agent version equally observable. For example, the current Cursor notes state that Cursor stopped recording per-turn usage locally around March 2026: older sessions that contain usage events can show token totals, while newer sessions can legitimately show none. A missing total in that situation is not evidence that AgentSight lost a captured response.

Codex has separate cumulative-session and response-level paths

Codex session JSONL contains token_countevents. For the native session's cumulative token summary, v1.0.31 codex_total_token_usage() walks the transcript from the end and returns the latestinfo.total_token_usage object it can parse. It does not add every token-count event as a new billable increment.

There is a separate response-parsing path: when AgentSight constructs the latest parsed LLM response and that response does not already carry a total, the parser can use last_token_usage as a fallback. That fallback does not make last_token_usage the cumulative native-session total. Keeping the two paths distinct is important when a rollout contains one shape but not the other.

Codex discovery can also use ~/.codex/state_5.sqlite. The v1.0.31 native source queries the recentthreads rows, takes the source-reported tokens_used, and prefers a rollout-derived cumulative usage when the rollout summary is available. That state database is a bounded discovery/index path, not another token total that should be added independently to the hydrated session.

Grouping happens after reconciliation

Once effective rows are selected, the token report aggregates input, output, cache-creation, cache-read, and total-token fields. The default key is model. --group-by provider groups by provider,comm by process command, pid by process ID, and dir (alsocwd or directory) by the best available session or process working directory.

Session counts are de-duplicated separately from call counts. A group can therefore represent many calls from one session, and the number of token observations is not itself the number of sessions. Unknown grouping metadata is reported as unknown rather than silently reassigned.

Three totals that should not be conflated

Signal	What it means	What it does not prove
Observed/session token usage	Token fields exposed by the selected model-response, telemetry, CLI, or native-session evidence.	An invoice amount or provider subscription quota.
Source-reported capacity window	Provider/agent metadata about a usage or subscription window when a supported source exposes it.	Usage inferred from AgentSight token totals.
Agent Flamegraph token width	Offline semantic aggregation of reported token counts, with bounded estimates only when that profiling path permits them.	A replacement for the raw report or an exact currency cost.

Provider pricing, cache discounts, plan credits, and subscription windows change independently from the accounting rows above. If you need dollars, preserve the token-kind breakdown and join it to a dated pricing source separately. Do not relabel total_tokens as cost.

A reproducible token check takes three queries

Start from one explicit saved run when you are debugging a recorded workload. Compare the default model view with provider and working-directory views, and keep JSON output when another tool will consume the result.

agentsight report token --db ./agentsight-run.db --json
agentsight report token --db ./agentsight-run.db --group-by provider
agentsight report token --db ./agentsight-run.db --group-by dir

# Deliberately ignore saved DBs and use supported local agent histories
agentsight report --local token --group-by model --json

If a total looks surprising, inspect the run's source provenance before doing arithmetic by hand. Rows that share a selection key can replace one another according to source priority; rows with different keys can both survive. Separately, Gemini stdout can replace a smaller network aggregate for the same PID/model. The useful debugging questions are therefore “which rows shared a key?”, “which source won that key?”, and “did the Gemini aggregate rule apply?”

What a zero or missing number can actually mean

A missing usage field can mean the provider did not persist it, the local-session format no longer includes it, a live capture did not observe the response boundary, or the selected source genuinely reported no value. These cases are not interchangeable. The effective-token rules remove overlap in the cases they explicitly reconcile, but they do not manufacture missing accounting or provide a universal cross-source correlation.

Keep negative claims scoped to the evidence source. “This report has no token total for the session” is defensible. “The agent used zero tokens” usually is not unless the source contract itself establishes that.

How to verify the implementation yourself

Read collector/src/main.rs for CLI and DB/native selection, then cli_db.rs for materialized-view loading. The keyed and Gemini reconciliation rules are in effective_tokens() and the adjacent source-priority function. Inspect agent_native.rs for native-session row construction and Codex state_5.sqlite discovery, then ext/session/src/parser.rs for the provider-native token shapes and cumulative Codex helper.

For an investigation, record the AgentSight version, whether the input was a DB or native sessions, the group key, and the JSON result. Those details are enough for another reviewer to distinguish source reconciliation from a real change in the underlying agent usage.