How Python is Fighting Open Source's 'Phantom' Dependencies Problem

Since 2023 the Python Software Foundation has had a Security Developer-in-Residence (sponsored by the Open Source Security Foundation's vulnerability-finding 'Alpha-Omega' project). And he's just published a new 11-page white paper about open source's 'phantom dependencies' problem — suggesting a way to solve it.

'Phantom' dependencies aren't tracked with packaging metadata, manifests, or lock files, which makes them 'not discoverable' by tools like vulnerability scanners or compliance and policy tools. So Python security developer-in-residence Seth Larson authored a recently-accepted Python Enhancement Proposal offering an easy way for packages to provide metadata through Software Bill-of-Materials (SBOMs). From the whitepaper:

Python Enhancement Proposal 770 is backwards compatible and can be enabled by default by tools, meaning most projects won't need to manually opt in to begin generating valid PEP 770 SBOM metadata. Python is not the only software package ecosystem affected by the 'Phantom Dependency' problem. The approach using SBOMs for metadata can be remixed and adopted by other packaging ecosystems looking to record ecosystem-agnostic software metadata...

Within Endor Labs' [2023 dependencies] report, Python is named as one of the most affected packaging ecosystems by the 'Phantom Dependency' problem. There are multiple reasons that Python is particularly affected:

- There are many methods for interfacing Python with non-Python software, such
as through the C-API or FFI. Python can 'wrap' and expose an easy-to-use
Python API for software written in other languages like C, C++, Rust, Fortran,
Web Assembly, and more.
- Python is the premier language for scientific computing and artificial
intelligence, meaning many high-performance libraries written in system
languages need to be accessed from Python code.

- Finally, Python packages have a distribution type called a 'wheel', which is
essentially a zip file that is 'installed' by being unzipped into a directory,
meaning there is no compilation step allowed during installation. This is great
for being able to inspect a package before installation, but it means that all
compiled languages need to be pre-compiled into binaries before installation...

When designing a new package metadata standard, one of the top concerns is reducing the amount of effort required from the mostly volunteer maintainers of packaging tools and the thousands of projects being published to the Python Package Index... By defining PEP 770 SBOM metadata as using a directory of files, rather than a new metadata field, we were able to side-step all the implementation pain...
We'll be working to submit issues on popular open source SBOM and vulnerability scanning tools, and gradually, Phantom Dependencies will become less of an issue for the Python package ecosystem.

The white paper 'details the approach, challenges, and insights into the creation and acceptance of PEP 770 and adopting Software Bill-of-Materials (SBOMs) to improve the measurability of Python packages,' explains an announcement from the Python Software Foundation. And the white paper ends with a helpful note.

'Having spoken to other open source packaging ecosystem maintainers, we have come to learn that other ecosystems have similar issues with Phantom Dependencies. We welcome other packaging ecosystems to adopt Python's approach with PEP 770 and are willing to provide guidance on the implementation.'

Read more of this story at Slashdot.