Skip to content

feat: Add per-metric windows to lock free metrics batcher - #8327

Open
alexander-alderman-webb wants to merge 37 commits into
webb/sliding-window-metrics-batcherfrom
webb/sliding-window-metrics-batcher-2
Open

alexander-alderman-webb wants to merge 37 commits into
webb/sliding-window-metrics-batcherfrom
webb/sliding-window-metrics-batcher-2

Conversation

@alexander-alderman-webb

@alexander-alderman-webb alexander-alderman-webb commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Keeps window-sampled metrics in a thread-local reservoir, and performs uniform sampling once the window is no longer the active window. Segmenting metrics per thread while the window is being modified enables lock-free operations (only a thread touches its state).

Conceptually, each window has thread state, and each thread state has a reservoir of metrics.

There can be multiple consumers if sentry_sdk.flush() runs concurrently with the background flusher. This is expected and handled by IndexError.

Issues

Reminders

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Results 📊

✅ 59490 passed | ⏭️ 3035 skipped | Total: 62525 | Pass Rate: 95.15% | Execution Time: 239m 36s

📊 Comparison with Base Branch

Metric Change
Total Tests 📉 -84875
Passed Tests 📉 -80005
Failed Tests —
Skipped Tests 📉 -4870

All tests are passing successfully.

✅ Patch coverage is 88.89% (112 of 126 changed executable lines covered; target 80%).
Project statement coverage is 90.01% (down 0.21 percentage points from base (721094c) to head (2caf187)).

Changed files with executable lines (2)
File Patch coverage Changed executable lines
sentry_sdk/_lock_free_metrics_batcher.py 90.48% 95/105 covered; missed: 88, 89, 128, 129, 134, 135, 144, 171, 205, 206; partial branches: 127, 133, 138, 170
sentry_sdk/client.py 80.95% 17/21 covered; missed: 144, 149, 153, 161; partial branches: 136, 138, 152
Coverage diff
@@            Coverage Diff             @@
##        master     #8327       +/-##
==========================================
- Coverage    90.22%    90.01%    -0.21%
==========================================
  Files          203       201        -2
  Tracked lines     27349     22351     -4998
  Branches     10252      7826     -2426
==========================================
- Hits         24675     20119     -4556
+ Misses        2674      2232      -442
+ Partials      1557      1292      -265

Generated by Coverage Action

@alexander-alderman-webb alexander-alderman-webb changed the title feat: Add sliding window to lock free metrics batcher feat: Add per-metric windows to lock free metrics batcher Nov 18, 2026
@alexander-alderman-webb
alexander-alderman-webb marked this pull request as ready for review November 19, 2026 09:11
@alexander-alderman-webb
alexander-alderman-webb requested a review from a team as a code owner November 19, 2026 09:11

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@alexander-alderman-webb

Copy link
Copy Markdown
Collaborator Author

I just inlined _merge() as well since that's easier to read.

@sentrivana sentrivana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things:

  • Retiring/harvesting might have a race, see comment on the diff
  • I like the lock-free design with the buckets by thread, it's cool. But it also means that memory usage scales with how many threads are active at a time, which could potentially be a lot. Might need some testing to see if that's an issue.

window = self._retired.popleft()
except IndexError:
break
sampled_out += self._harvest(window, items)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we should decouple the harvesting from the flush somehow, or at least introduce some small delay in between. A recently retired window might probably still have a thread writing to it, if the write was initiated before it was retired, so we might be losing some metrics there.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added another 0.05 seconds delay so there's at least a guaranteed buffer since I couldn't come up with a good synchronization mechanism to strictly not lose any metrics.
See 5e08286

@alexander-alderman-webb

Copy link
Copy Markdown
Collaborator Author

But it also means that memory usage scales with how many threads are active at a time, which could potentially be a lot.

Yes great point, currently each reservoir can hold up to limit metrics, and the number of reservoirs scales with the number of threads. This means that memory scales linearly with numbers threads in the worst case. I made sure that the metrics in the reservoirs still have a global limit, so that the total memory use of the batcher is bounded. For getsentry deployments I'm not aware of any process with unusually many threads (expecting 8-12 threads).

I just realized that we dropped unsampled metrics when we reach MAX_BEFORE_DROP though.
I've changed the logic so that unsampled metrics are dropped when MAX_UNSAMPLED_BEFORE_DROP unsampled metrics are in the batcher, and sampled metrics are dropped when MAX_KEPT_PER_WINDOW sampled metrics are in the corresponding window (these are new constants). See 525cb85

We could add a mechanism where metrics are pruned when the limit is approached across reservoirs so we don't reach the global limit as fast as a follow up later.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 008f662. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants