Repository navigation
Fail with exit 3 when a scan comparison returns no artifacts - #461
Draft
David Larsen (dc-larsen) wants to merge 6 commits into
Draft
David Larsen (dc-larsen) wants to merge 6 commits into
David Larsen (dc-larsen) wants to merge 6 commits into
Conversation
When the diff-scan poll returned a finished comparison whose artifacts were null and whose incomplete flag was unset, get_diff_scan_artifacts raised a plain Exception. get_added_and_removed_packages treated that like any other failure of the diff-scans flow and fell back to the streaming comparison, whose response has no incomplete flag. The run then reported on a comparison nobody had checked for completeness. A finished comparison without an artifacts object now raises IncompleteComparisonError, which skips the fallback and maps to the infrastructure exit code (3, --exit-code-on-api-error, or 0 under --disable-blocking). Failures of the flow itself (missing token scopes, transient errors, the polling timeout, a truncated response) still fall back to streaming as before.
David Larsen (dc-larsen)
temporarily deployed
to
socket-firewall
November 20, 2026 20:17 — with
GitHub Actions
Inactive
3 when a scan comparison returns no artifacts
The SDK's diffscans.get returns {} for statuses it doesn't raise on (204, an
unfollowed 3xx). That response says nothing about the comparison, so it falls
back to streaming as it did in 2.13.0 instead of exiting 3 with a message
claiming the API finished the comparison. Only a returned diff_scan whose
artifacts are null, missing or not an object raises the incomplete-comparison
error, and its message now says that.
This was referenced Nov 20, 2026
David Larsen (dc-larsen)
deployed
to
socket-firewall
November 20, 2026 22:18 — with
GitHub Actions
Active
…ts-exit-3 # Conflicts: # CHANGELOG.md
This was referenced Nov 20, 2026
…ts-exit-3 # Conflicts: # CHANGELOG.md
2.13.2 was released from main, so this change needs the next patch version.
David Larsen (dc-larsen)
deployed
to
socket-firewall
November 20, 2026 23:11 — with
GitHub Actions
Active
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the diff-scan poll returns a finished comparison with
artifacts: nullandincompleteunset, 2.13.0 logsand finishes on the streaming comparison. The streaming response has no
incompleteflag, so the run reports on a comparison that nothing has checked for completeness. Withincomplete: truethe same response correctly exits3(#453). #453's description reads as if both cases fail ("including when its artifacts are null or absent"). That holds only whenincompleteis also set. Found while checking 2.13.0 for a customer who previously ran their own null-artifacts check in CI.Root Cause
Core.get_diff_scan_artifactsraisesIncompleteComparisonErrorforincomplete: truebut a plainExceptionfor a missingartifactsobject.Core.get_added_and_removed_packagesre-raises onlyIncompleteComparisonError. Every other exception is treated as "the diff-scans flow is unavailable" and falls back tofullscans.stream_diff.Is the fallback deliberate for this shape?
The fallback (added in #284) is deliberate for failures of the diff-scans flow: a token without the
diff-scans:*scopes (403), deployments without the endpoint, the 30-minute polling backstop, and a response truncated on a huge dependency tree (response.json()fails). All of those still fall back. This PR doesn't change them.A finished (non-202) response that returns the
diff_scanobject without anartifactsobject is not one of those cases:artifactsobject with anincompleteboolean.omit_unchangedsetsartifacts.unchangedto null, notartifacts, and the CLI already handles that (artifacts_dict.get(key) or []; covered by a new test here).diff_scanobject at all (204, unfollowed 3xx) is a flow failure too, and it still falls back.incompletefield, so there is no completeness check to add on that path. Falling back turns a comparison the API couldn't deliver into one nobody checked.So the choice is to fail with the infrastructure code, not to keep the fallback with a check that can't exist.
Fix
diff_scanobject whoseartifactsis missing, null or not an object raisesIncompleteComparisonErrorwith its own message ("The scan comparison returned no results (diff scan ...): the API returned the diff scan without its artifacts object ..."). It reuses the existing class so it skips the fallback and maps throughcli()likeincomplete: true: exit3, the--exit-code-on-api-errorvalue, or0under--disable-blocking. That matches the README's "incomplete scan comparison" row, so the exit table is unchanged.diff_scanobject at all keeps 2.13.0's streaming fallback. The SDK'sdiffscans.getreturns{}for a status it doesn't raise on (not 200/202, below 400: in practice a 204 or an unfollowed 3xx). That says nothing about the comparison, so it's treated like the other flow failures (403, 404, timeout, truncated body) instead of exiting3with a message claiming the API finished. The fallback warning now reads "response has no diff_scan object". A top-levelincomplete: truestill exits3, as before.get_diff_scan_artifactsdocstring, theIncompleteComparisonErrordocstring anddocs/ci-cd.mdupdated.Testing
tests/core/test_exit_codes.py: four response shapes with adiff_scanobject (artifacts: nullwithincomplete: false,artifacts: nullwithincompleteunset,artifactsabsent,artifacts: []) raiseIncompleteComparisonErrorand never callstream_diff; the exit mapping is pinned for default /--exit-code-on-api-error 42/--disable-blocking/ both;unchanged: nullunderomit_unchangedstill returns a result;incomplete: truestill raises. New:{}and{"diff_scan": null}raise a plain exception and reachstream_diff, and a real HTTP 204 run through the SDK's ownDiffScans.get()falls back to streaming. The 8 shape tests fail onmain; the 204 test passes onmainand failed on this PR's previous head.uv run pytest: 1017 passed (1001 onmain).ruff check socketsecurity tests,uv lock --locked --offline,git diff --check, anduv build+twine checkpass.--scm api --integration api --ignore-commit-files --strict-blocking) on two projects: npm (lodash 4.17.11 + handlebars 4.0.11 baseline, the branch adds vm2 3.9.11) and PyPI (pyyaml 5.3 + django 2.2, the branch adds requests 2.19.1). The only change is the final diff-scan poll response, patched in-process (or, for 204, a real 204 response fed through the SDK'sget()). Before = 2.13.0 from PyPI, after = this branch run twice; both after runs were identical on both projects:42/--disable-blockingartifacts: null,incompleteunset42/--disable-blockingartifactsabsent;artifacts: []artifacts: null,incomplete: true42/--disable-blocking{};{"diff_scan": null}42/--disable-blockingcreate_from_ids403Other open 2.13.x fixes
2.13.1 and 2.13.2 were released from
mainon 2026-11-20, so this branch mergesmainatv2.13.2and bumps to2.13.3inpyproject.toml,socketsecurity/__init__.pyanduv.lock, with its CHANGELOG entries under a new## 2.13.3heading. #459, #460, #461, #462, #463, #464 and #465 all make the same bump, so the version files merge cleanly between them, and once one merges the rest ride main's unreleased2.13.3(Version Check accepts a version equal to main's while it is above the latest PyPI release). Merging a second one conflicts only inCHANGELOG.md: keep one## 2.13.3heading and combine the bullets under its subsections (Upgrade notes, Changed, Fixed, Documentation, in that order). There are no code conflicts between them. Ifmainreleases 2.13.3 before this merges, bump to the next patch again.Public Changelog
A scan comparison the API returns without its package lists now fails the run with the infrastructure-error code (3, or the
--exit-code-on-api-errorvalue) instead of falling back to a comparison that can't report whether it is complete.