Refinements based on read-thru

Author: warsaw

peps/pep-0829.rst

@@ -1,6 +1,7 @@
 PEP: 829
 Title: Structured Startup Configuration via .site.toml Files
 Author: Barry Warsaw <barry@python.org>
+Discussions-To: Pending
 Status: Draft
 Type: Standards Track
 Topic: Packaging
@@ -27,40 +28,40 @@ Motivation
 Python's ``.pth`` files (processed by ``Lib/site.py`` at startup)
 support two functions:
 
-#. **Extending** ``sys.path`` -- Lines in this file (excluding
-   comments and lines that start with ``import``) name directories to
-   be appended to ``sys.path``.  Relative paths are implicitly
-   anchored at the site-packages directory.
+* **Extending** ``sys.path`` -- Lines in this file (excluding
+  comments and lines that start with ``import``) name directories to
+  be appended to ``sys.path``.  Relative paths are implicitly
+  anchored at the site-packages directory.
 
-#. **Executing code** -- lines starting with ``import`` (or
-   ``import\\t``) are executed immediately by passing the source string
-   to ``exec()``.
+* **Executing code** -- lines starting with ``import`` (or
+  ``import\\t``) are executed immediately by passing the source string
+  to ``exec()``.
 
 This design has several problems:
 
-#. Code execution is a side effect of the implementation.  Lines that
-   start with ``import`` can be extended by separating multiple
-   statements with a semicolon.  As long as all the code to be
-   executed appears on the same line, it all gets executed when the
-   ``.pth`` file is processed.
+* Code execution is a side effect of the implementation.  Lines that
+  start with ``import`` can be extended by separating multiple
+  statements with a semicolon.  As long as all the code to be
+  executed appears on the same line, it all gets executed when the
+  ``.pth`` file is processed.
 
-#. ``.pth`` files are essentially unstructured, leading to contents
-   which are difficult to reason about or verify, and often even
-   difficult to read.  It mixes two potentially useful features with
-   different security constraints, and no way to separate out these
-   concerns.
+* ``.pth`` files are essentially unstructured, leading to contents
+  which are difficult to reason about or validate, and are often even
+  difficult to read.  It mixes two potentially useful features with
+  different security constraints, and no way to separate out these
+  concerns.
 
-#. The lack of ``.pth`` file structure also means there's no way to
-   express metadata, no future-proofing of the format, and no defined
-   execution or processing order of the contents.
+* The lack of ``.pth`` file structure also means there's no way to
+  express metadata, no future-proofing of the format, and no defined
+  execution or processing order of the contents.
 
-#. Using ``exec()`` on the file contents during interpreter startup is
-   a broad attack surface.
+* Using ``exec()`` on the file contents during interpreter startup is
+  a broad attack surface.
 
-#. There is no explicit concept of an entry point, which is an
-   established pattern in Python packaging.  Packages that require
-   code execution and initialization at startup abuse ``import`` lines
-   rather than explicitly declaring entry points.
+* There is no explicit concept of an entry point, which is an
+  established pattern in Python packaging.  Packages that require
+  code execution and initialization at startup abuse ``import`` lines
+  rather than explicitly declaring entry points.
 
 
 Specification
@@ -76,11 +77,14 @@ option, which disables ``site.py`` also disables
 The standard library ``tomllib`` package is used to read and process
 ``<package>.site.toml`` files.
 
-Any parsing errors cause the entire ``<package>.site.toml`` file to be
-ignored and not processed (but it still supersedes any parallel
-``<package>.pth`` file).  Any errors that occur when importing entry
-point modules or calling entry point functions are reported but do no
-abort the Python executable.
+The presence of a ``<package>.site.toml`` file supersedes a parallel
+``<package>.pth`` file.  This allows for both an easy migration path and
+continued support for older Pythons in parallel.
+
+Any parsing errors cause the entire ``<package>.site.toml`` file to be ignored
+and not processed (but it still supersedes any parallel ``<package>.pth``
+file).  Any errors that occur when importing entry point modules or calling
+entry point functions are reported but do not abort the Python executable.
 
 
 File Naming and Discovery
@@ -96,18 +100,20 @@ File Naming and Discovery
   ``site.py``).
 
 * ``<package>.site.toml`` files live in the same site-packages directories
-  where ``.pth`` files are found today.
+  where ``<package>.pth`` files are found today.
 
 * The discovery rules for ``<package>.site.toml`` files is the same as
-  ``.pth`` files today.  File names that start with a single ``.``
+  ``<package>.pth`` files today.  File names that start with a single ``.``
   (e.g. ``.site.toml``) and files with OS-level hidden attributes (``UF_HIDDEN``,
   ``FILE_ATTRIBUTE_HIDDEN``) are excluded.
 
 * The processing order is alphabetical by filename, matching ``.pth``
   behavior.
 
 * If both ``<package>.site.toml`` and ``<package>.pth`` exist in the same
-  directory, only the ``<package>.site.toml`` file is processed.
+  directory, only the ``<package>.site.toml`` file is processed.  In other
+  words, the presence of a ``<package>.site.toml`` file supersedes a parallel
+  ``<package.pth>`` file, even if the format of the TOML file is invalid.
 
 
 Processing Model
@@ -135,9 +141,9 @@ This two-phase approach (read then process) enables:
 
 Within each site-packages directory, the processing order is:
 
-#. Discover and parse all ``<package>.site.toml`` files (alphabetically).
-#. Process all ``[paths]`` entries from the parsed data.
-#. Execute all ``[entrypoints]`` entries from the parsed data.
+#. Discover and parse all ``<package>.site.toml`` files, sorted alphabetically.
+#. Process all ``[paths]`` entries from the parsed TOML files.
+#. Execute all ``[entrypoints]`` entries from the parsed TOML files.
 #. Process any remaining ``<package>.pth`` files that are not superseded by a
    ``<package>.site.toml`` file.
 
@@ -146,8 +152,8 @@ runs, and that ``<package>.site.toml``-declared paths are available to both
 entry point imports and ``<package>.pth`` import lines.
 
 
-<package>.site.toml file schema
--------------------------------
+TOML file schema
+----------------
 
 A ``<package>.site.toml`` file is defined to have three sections, all of which
 are optional:
@@ -167,8 +173,8 @@ are optional:
     init = ["foo.startup:initialize", "foo.plugins"]
 
 
-``[metadata]``
-''''''''''''''
+The ``[metadata]`` section
+''''''''''''''''''''''''''
 
 This section contains package and/or file metadata.  There are no required
 keys, and no semantics are assigned to any keys in this section *except* for
@@ -178,13 +184,14 @@ permitted and preserved.
 Defined keys:
 
 ``schema_version`` (integer, recommended)
-    The TOML file schema version number.  Must be the integer ``1`` for this
-    specification.  If present, Python guarantees forward-compatible handling:
-    future versions will either process the file according to the declared
-    schema or skip it with a clear diagnostic.  If the
-    ``schema_version`` is present but has an unsupported value, the
-    entire file is skipped.  If ``schema_version`` is omitted, the file is processed
-    on a best-effort basis with no forward-compatibility guarantees.
+    The TOML file schema version number.  Must be the integer ``1``
+    for this specification.  If present, Python guarantees
+    forward-compatible handling: future versions will either process
+    the file according to the declared schema or skip it with clear
+    diagnostics.  If the ``schema_version`` is present but has an
+    unsupported value, the entire file is skipped.  If
+    ``schema_version`` is omitted, the file is processed on a
+    best-effort basis with no forward-compatibility guarantees.
 
 Recommended keys:
 
@@ -201,8 +208,8 @@ Recommended keys:
     ``"aperson@example.com"``.
 
 
-``[paths]``
-'''''''''''
+The ``[paths]`` section
+'''''''''''''''''''''''
 
 Defined keys:
 
@@ -222,8 +229,8 @@ Path entries use a hybrid resolution scheme:
 * **Placeholder variables** are supported using ``{name}`` syntax.  The
   placeholder ``{sitedir}`` expands to the site-packages directory where the
   ``<package>.site.toml`` file was found.  Thus ``{sitedir}/relpath`` and
-  ``relpath`` resolve to the same path and this is the explicit form
-  of the relative path form.
+  ``relpath`` resolve to the same path with the placeholder version being the
+  explicit (and recommended) form of the relative path form.
 
 While only ``{sitedir}`` is defined in this PEP, additional
 placeholder variables (e.g., ``{prefix}``, ``{exec_prefix}``,
@@ -232,13 +239,13 @@ placeholder variables (e.g., ``{prefix}``, ``{exec_prefix}``,
 If ``dirs`` is not a list of strings, a warning is emitted (visible
 with ``-v``) and the section is skipped.
 
-Directories that do not exist on the filesystem are silently skipped,
-matching ``<package>.pth`` behavior.  Duplicate paths are
-de-duplicated, also matching ``<package>.pth`` behavior.
+Directories that do not exist on the filesystem are silently skipped, matching
+``<package>.pth`` behavior.  Paths are de-duplicated, also matching
+``<package>.pth`` behavior.
 
 
-``[entrypoints]``
-'''''''''''''''''
+The ``[entrypoints]`` section
+'''''''''''''''''''''''''''''
 
 ``init`` -- a list of strings specifying entry point references to
 execute at startup.  Each item uses the standard Python entry point
@@ -358,11 +365,11 @@ Continue on error rather than abort
 Backwards Compatibility
 =======================
 
-* ``<package>.pth`` file processing is **not** removed.  Both
-  ``<package>.pth`` and ``<package>.site.toml`` files are discovered
-  in parallel within each site-packages directory.  This preserves
-  backward compatibility for all existing (pre-migration) packages.
-  Deprecation of ``<package>.pth`` files is out-of-scope for this PEP.
+* ``<package>.pth`` file processing is **not** deprecated or removed.  Both
+  ``<package>.pth`` and ``<package>.site.toml`` files are discovered in
+  parallel within each site-packages directory.  This preserves backward
+  compatibility for all existing (pre-migration) packages.  Deprecation of
+  ``<package>.pth`` files is out-of-scope for this PEP.
 
 * When ``<package>.site.toml`` exists alongside ``<package>.pth``, the
   ``<package>.site.toml`` takes precedence and the ``<package>.pth`` file is
@@ -396,10 +403,10 @@ This PEP improves the security posture of interpreter startup:
   importable modules and their attributes, unlike ``exec()`` which can
   run arbitrary code.
 
-The overall attack surface is not eliminated -- a malicious
-``<package>.site.toml`` file can still cause arbitrary code execution via
-``init`` entrypoints, but the mechanism proposed in this PEP is more
-structured, auditable, and amenable to policy controls.
+The overall attack surface is not eliminated -- a malicious package
+can still cause arbitrary code execution via ``init`` entrypoints, but
+the mechanism proposed in this PEP is more structured, auditable, and
+amenable to future policy controls.
 
 
 How to Teach This
@@ -437,25 +444,26 @@ If your ``<package>.pth`` file includes arbitrary code, put that code in a
 start up function and use the ``module:callable`` syntax.
 
 Both ``<package>.pth`` and ``<package>.site.toml`` can coexist during
-migration.  If both exist for the same package, only the ``<package>.site.toml`` is
-processed.  Thus, it is recommended that packages compatible with older
-Pythons ship both files.
+migration.  If both exist for the same package, only the
+``<package>.site.toml`` is processed.  Thus it is recommended that
+packages compatible with older Pythons ship both files.
+
 
 For tool authors
 ----------------
 
- Build backends and installers should generate ``<package>.site.toml`` files
- alongside or instead of ``<package>.pth`` files, depending on the package's
- Python support matrix.  The TOML format is easy to generate programmatically
- using ``tomllib`` (for reading) or string formatting (for writing, since the
- schema is simple).
+Build backends and installers should generate ``<package>.site.toml``
+files alongside or instead of ``<package>.pth`` files, depending on
+the package's Python support matrix.  The TOML format is easy to
+generate programmatically using ``tomllib`` (for reading) or string
+formatting (for writing, since the schema is simple).
 
 
 Reference Implementation
 =========================
 
-A reference implementation is provided as modifications to ``Lib/site.py``,
-adding the following:
+A `reference implementation <https://github.com/python/cpython/pull/147955>`_
+is provided as modifications to ``Lib/site.py``, adding the following:
 
 * ``_SiteTOMLData`` -- a ``__slots__`` class holding parsed data from
   a single ``<package>.site.toml`` file (metadata, dirs, init).
@@ -494,8 +502,7 @@ JSON instead of TOML
     ``pyproject.toml``.
 
 YAML instead of TOML
-    YAML is not in the standard library and has well-documented
-    parsing pitfalls.
+    There is no standard YAML parser in the standard library.
 
 Python instead of TOML
     Python is imperative, TOML is declarative.  Thus TOML files are
@@ -548,6 +555,12 @@ Open Issues
   ``{userbase}``.
 
 
+Change History
+==============
+
+None at this time.
+
+
 Copyright
 =========