Repository navigation
gh-160697: Process the QSBR queue from the eval breaker, not from _PyMem_FreeDelayed() - #160698
Conversation
| def test_type_cache_invalidation(self): | ||
| class C: | ||
| x = None | ||
|
|
||
| holders = self.queue_finalizers(lambda: C.x) | ||
| for h in holders: | ||
| h.__dict__ = {} | ||
| C.x | ||
| new = object() | ||
| C.x = new | ||
| self.assertIs(C.x, new) |
There was a problem hiding this comment.
The type cache is unfortunately affected too.
|
I think this is the right fix. Running arbitrary Python code (via the Suggested change: A note on a risk, not to fix here: a thread that stays in C code without reaching the eval breaker (a long C-only loop over shared objects, or an embedder thread that only calls the C API) will now grow its queue without bound until something else drains it. The backstops are the GC, which merges and processes all threads' queues, and |
There was a problem hiding this comment.
Lines 131 to 132 in 0fb8263
The document needs updating too.
| if (buf == NULL) { | ||
| // failed to allocate a buffer, free immediately | ||
| PyObject *to_dealloc = NULL; | ||
| _PyEval_StopTheWorld(tstate->base.interp); | ||
| if (work_item_should_decref(ptr)) { | ||
| PyObject *obj = (PyObject *)(ptr - 1); | ||
| if (!_Py_IsImmortal(obj)) { | ||
| Py_ssize_t refcount = _Py_ExplicitMergeRefcount(obj, -1); | ||
| if (refcount == 0) { | ||
| to_dealloc = obj; | ||
| } | ||
| } | ||
| } | ||
| else { | ||
| PyMem_Free((void *)ptr); | ||
| } | ||
| _PyEval_StartTheWorld(tstate->base.interp); | ||
| if (to_dealloc != NULL) { | ||
| _Py_Dealloc(to_dealloc); | ||
| } | ||
| return; | ||
| } |
There was a problem hiding this comment.
This still runs __del__ synchronously when PyMem_Calloc fails. Should it go through the eval breaker too?
There was a problem hiding this comment.
This one is actually safe because this path only handles the item being freed, not the queue. For a raw buffer it is just PyMem_Free(). Every caller of _PyObject_XDecRefDelayed() (type_setattro, _PyObject_SetDict, _PyObject_SetManagedDict, the function and generator setters) puts it where a plain Py_XSETREF() or Py_XDECREF() used to be. They only delay it for lock-free readers in other threads, and the stop-the-world on this path covers those readers. So running the finalizer there is the same as the old synchronous decref.
|
|
||
| /* Reserve a few bits for future use */ | ||
| #define _PY_EVAL_EVENTS_BITS 8 | ||
| #define _PY_EVAL_EVENTS_BITS 9 |
There was a problem hiding this comment.
cpython/Python/instrumentation.c
Lines 2043 to 2045 in 0fb8263
The
set_events overflows sooner now, is that fine? Above comment needs updating anyway.
There was a problem hiding this comment.
Well, yes, that's true. A tight set_events() toggle loop now hits OverflowError after 2^23 - 1 calls instead of 2^24 - 1.
Unfortunately, all 8 event bits were already in use, so the new flag needs a ninth. Since only the free-threaded build uses this flag, the alternative would be to define it (and the extra bit) only under free-threaded build. I'd rather not have the bit layout differ between builds, so I've just updated the comment to drop the reference to 24 bits.
|
Thank you Neil and Neko for the review! I've removed So, finally limits start working: import resource, sys, threading
L = []
t = threading.Thread(target=L.append, args=(0,))
t.start()
t.join()
chunk = [None] * 150_000
start = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
for _ in range(2000):
L.extend(chunk)
L.clear()
peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(f"peak growth {(peak - start) / 1024:.0f} MiB")
|
johng
left a comment
There was a problem hiding this comment.
Coming from someone who is trying to gain familiarity with the code base I had a question about the test readability
| def test_dict_resize(self): | ||
| d = {} | ||
| make_shared(d.get, 0) | ||
| holders = self.queue_finalizers(lambda: d.__setitem__(object(), 0)) | ||
| for n, h in enumerate(holders): | ||
| h.__dict__ = {} | ||
| if n % 2: | ||
| h.__dict__ = {} | ||
| for k in list(d): | ||
| del d[k] | ||
| for i in range(6): | ||
| d[i] = i | ||
| self.assertEqual(len(d), len(list(d))) |
There was a problem hiding this comment.
I found this test quite hard to reason with, the odd/even behaviour was a bit difficult to reason exactly why that was picked.
I'm still wondering why it was added when it was not part of the original issue, something like this can equally re-create the issue. Simply staging the decrefs for "Evil" and then iterating with the resizing trigger. I feel this is a bit easier to understand the delayed decref mechanism during the resize.
def test_dict_resize(self):
d = {}
make_shared(d.get, 0)
# Queue delayed decrefs whose finalizers mutate d. Use fewer than
# fit in one chunk of work items, so that none of them are released
# before the loop below starts.
holders = self.queue_finalizers(lambda: d.__setitem__(object(), 0),
count=100)
for h in holders:
h.__dict__ = {}
# Every round compacts d, and each compaction delay-frees the old
# keys. From here on that is the only source of delayed frees, so
# if they were released whenever a chunk fills up, the finalizers
# would run in the middle of dictresize().
for _ in range(self.ROUNDS):
for k in list(d):
del d[k]
for i in range(6):
d[i] = i
self.assertEqual(len(d), len(list(d)))
There was a problem hiding this comment.
Hi and thanks for the review!
The odd/event part was there to shift where the chunk boundary falls. I suppose we can't know how many delayed frees are already in the thread's queue when the test starts (although maybe we can write some kind of helper for that) and the bug shows up only when the chunk fills on the free inside the resize.
I've applied your approach for the dict test (it catches the bug on main for every possible queue fill (0-253), because every delayed free in that loop comes from dictresize). But not for the list, because each round there has three delayed frees (clear and two resizes), when the boundary lands on clear, the finalizers run at a safe point and therefore test passes.
| def test_dict_resize(self): | ||
| d = {} | ||
| make_shared(d.get, 0) | ||
| holders = self.queue_finalizers(lambda: d.__setitem__(object(), 0)) | ||
| for n, h in enumerate(holders): | ||
| h.__dict__ = {} | ||
| if n % 2: | ||
| h.__dict__ = {} | ||
| for k in list(d): | ||
| del d[k] | ||
| for i in range(6): | ||
| d[i] = i | ||
| self.assertEqual(len(d), len(list(d))) |
There was a problem hiding this comment.
As a separate comment, changing the assert to assertEqual(len(d), sum(1 for _ in d)) would avoid the test seg faulting, instead failing on AssertionError: 106 != 6
|
Thanks @Eclips4 for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15. |
|
Sorry, @Eclips4 and @kumaraditya303, I could not cleanly backport this to Please backport manually with cherry_picker, see the devguide for more information. |
|
Sorry, @Eclips4 and @kumaraditya303, I could not cleanly backport this to Please backport manually with cherry_picker, see the devguide for more information. |
… not from `_PyMem_FreeDelayed()` (pythonGH-160698) (cherry picked from commit e0d0f2e) Co-authored-by: Kirill Podoprigora <kirill.bast@gmail.com>
… not from `_PyMem_FreeDelayed()` (pythonGH-160698) (cherry picked from commit e0d0f2e) Co-authored-by: Kirill Podoprigora <kirill.bast@gmail.com>
|
GH-162330 is a backport of this pull request to the 3.15 branch. |
|
GH-162331 is a backport of this pull request to the 3.14 branch. |
Uh oh!
There was an error while loading. Please reload this page.