What a successful install tells you
A one-line code change can alter an entire dependency graph. One package can bring in transitive packages, download binaries, and run with the permissions of a developer machine or CI runner. Finding a name that fails to install and deciding whether to trust a name that installs successfully require separate checks.
When a package name appears in AI-generated code, start by looking it up in the intended registry. An absent entry rules out that route. A result raises more questions. Check whether the API exists in that release, whether the same spelling names a different project in another ecosystem, and whether the distributed version points to the official source. A publisher can control the links in both the README and the registry entry, so compare them with a source you already trust, such as an official project site or internal catalog.
Spracklen and colleagues analyzed package hallucinations in 576,000 Python and JavaScript code samples for USENIX Security 2025. They used 16 code-generation models and two prompt datasets. The findings reflect the models, prompts, and registries in that study. They support checking suggested names against actual distribution channels. Use your own PR records to estimate how often your team sees incorrect suggestions. USENIX Security 2025 paper[1]
Record these identifiers before evaluating the package.
- Installation target. Record the ecosystem, normalized package name, scope, registry, and version. Internal and public registries may use the same name. Record the download location actually chosen, and in a monorepo identify which workspace settings and lockfile apply.
- Trusted origin. Identify the publisher, source repository, release tag or commit, and artifact identifier. Cross-check registry metadata against official release notes or documentation. A change of publisher, repository, or build path deserves fresh review against the evidence used for the preceding release.
- Required function. Identify the import, API, supported runtime, and failure behavior. Check the documentation and source for the version you will use. A correction that swaps in a similarly named package is still a new dependency decision.
Where a repository has registry rules, check the effective configuration on developer machines, PR builds, and release builds. CI may download the expected artifact while a developer's registry precedence selects a different archive. Resolve that discrepancy before comparing runtime results.
Five entries for a dependency review card
Use this table in a PR to review dependency additions. Review the application change and lockfile together so reviewers can assess where the new code can run. Identify owners in advance for security approvals and exceptions.
| Area | Evidence to record in the PR | Hold when |
|---|---|---|
| Need | Requirement, call sites, comparison with existing code or standard library, owner after adoption | There is no call site, or shared code already meets the need |
| Origin | Exact name and version, approved registry, publisher, source and release commit, artifact digest or verifiable provenance | Name or scope differs, origin changes without explanation, or the installed artifact differs from the attestation subject |
| Dependency graph | Manifest and lockfile diff, direct and transitive changes, registries, licenses, runtimes, platform differences | Unrelated mass upgrades, missing lockfile, or unexplained download endpoints |
| Execution | Install, build and runtime scripts or binaries, access paths, destinations and permissions, isolated test results | Reproduction needs production secrets, an unverified executable is downloaded, or unexpected traffic lacks a record |
| Maintenance | Reviewed commit, scan coverage, skipped or timed-out checks, owner, removal or replacement criteria | Findings have no coverage data, an exception lacks an owner or expiry, or installation cannot be reproduced |
NIST's SSDF is a starting point for managing external components. Establish who decides and which evidence they need. The maintainer assesses product fit, the build owner reproduces the dependency graph and execution, and security assesses origin and privilege exceptions. Where one person fills several roles, retain the exact target and reproducible logs or commands for the next reviewer. NIST SP 800-218 SSDF v1.1[2]
Read the lockfile for version and delivery changes
A manifest can specify a version range, while a lockfile records the graph the package manager resolved. A shared lockfile can change resolution in other workspaces. Review added transitive packages, removed versions, and multiple versions of the same package. If the file was broadly reordered, determine whether the tool version or the resolved graph changed.
Follow the delivery path. Check pinned versions, download targets, and integrity data. When a Git URL, local path, mirror, or alternate registry appears, verify which path the build actually uses. A matching integrity value confirms that the bytes match the recorded artifact. Review behavior and trust in the delivery chain separately.
In an npm project, npm ci errors if the manifest and existing lockfile disagree and leaves the lockfile unchanged. Use the same graph-affecting settings that created the lockfile. For other package managers, verify the lockfile and script controls available in the version you use. Record the install tool, runtime version, registry configuration, and PR commit. npm ci documentation[3]
Explain changes to transitive dependencies. If a package causes two versions of a parser to coexist, identify which paths use each. A code copy can remain in the deployment bundle after a version disappears from the dependency graph. Where dependencies are bundled, record whether you inspected the installed packages, deployment artifact, or runtime environment.
Some code runs before you reach your tests
Packages can run code during install, build, test, deploy, or request handling. npm documents lifecycle events for npm ci and npm install. Passing CI tests do not reveal which earlier installation code ran. Review the scripts and actual files shipped in the selected version before execution. npm lifecycle scripts[4]
For a required script, state its purpose. A native module might need a platform-specific binary. Record where it comes from and how you will verify it, which environment variables the script reads, what files it creates, and where it connects. A fixed download URL may serve different contents later. Reproduce the first approved execution in isolation without live customer data or production secrets, then check that it works with the minimum build permissions.
If your package manager can suppress scripts, consider that setting for initial inspection and record what it prevented. Reproduce deployment-required scripts under the approved conditions and inspect their output and network activity. Keep exceptions tied to the exact approved package version. A global change can let unrelated packages run as well.
Verify provenance and inspect behavior
Provenance connects an artifact to verifiable information about its origin and production. SLSA describes that connection. npm provenance links a registered package to source and build instructions. Check the signature, its relationship to the installed version and artifact, and whether the build identity and path meet your team's trust criteria. SLSA v1.2 Provenance[5], npm provenance documentation[6]
A properly attested build can still contain unsafe changes. Review new scripts and artifact-only content even when the expected repository and release tag match. If provenance is absent, identify alternative evidence allowed by your policy and its approver. You might compare an artifact with auditable source or distribute only the reviewed release through an internal registry. If you cannot establish the publisher's identity, revisit the requirement before installing a similarly named substitute.
Verify the paths beyond the happy path — Reconstruct requirements from behavior and exercise exceptions in input, permissions and external calls.
A worked example of approving or dropping an addition
A fictional delivery app needs to validate a short lookup key before calling a partner API. An AI-generated PR adds a utility library and describes the change as text conversion. The team checks the allowed characters and length, and learns that the partner API treats uppercase and lowercase keys distinctly. Lowercasing all inputs could route a valid key to another customer's record.
| Candidate | Test inputs and observations | Basis for the decision |
|---|---|---|
| Existing validation function | Empty input, longest valid key, forbidden character, case variations | Reuse the maintenance path if behavior matches |
| Standard library and a small check | Minimum and maximum lengths, newlines, whitespace, mixed characters, case preservation | Stable, small rule and observable behavior |
| Proposed external library | Valid and boundary inputs on the exact release, transitive packages, lifecycle scripts | Revisit if the requirement genuinely needs additional features such as Unicode normalization |
The team confirms that the requirement is validation and reuses the existing function. It removes the manifest and lockfile changes on the tested commit and reruns the checks. Valid keys keep their original values and invalid keys fail before any external request. The review records the whitespace and case boundaries the old tests missed. A later normalization requirement would reopen the need, version, and origin decision.
For a long-lived parser supporting a complex format, compare the maintenance cost of a library and in-house code. Evaluate the supported cases and failure isolation with tests. Decide in advance which functionality to disable or replace if maintenance ends, fixes become unavailable, or the publisher changes.
How scan results inform installation approval
Record the target commit, resolved lockfile, ecosystems and versions checked, and scan source. Interpret zero findings within that coverage. Repository dependency alerts may be disabled while a separate lockfile lookup succeeds. Record each result with its actual scope so the team knows which check needs to run again.
Public Qoretix documentation describes GitHub security alerts and lockfile-based supply chain lookups for connected repositories. It labels disabled repository checks as unknown and external scanner integrations as preview. Confirm tool, account, and repository coverage for your project. Your build and security policies govern provenance decisions, script approval, and network restrictions. IXC Qoretix overview, Qoretix security documentation[8]
If a check remains open, identify the boundary it affects. A runtime dependency running during installation with an unverified destination is a clear hold. A development dependency can still affect production artifacts if it runs on the build runner. If it already executed, identify the environments and credentials it could access. For keys that could have been exposed, see Responding to repository secret exposure.
Before submitting evidence for merge, confirm that approvals and scans refer to the commit with the reviewed lockfile. Recheck when a later commit changes the package version, and when scanner configuration or registry policies change what will be installed. For the relationship between protected branches and required checks, see Merge criteria for AI-generated pull requests.
