PEP 822: d-string draft implementation

.pre-commit-config.yaml

@@ -19,6 +19,7 @@ repos:
         name: Run Ruff (lint) on Lib/test/
         args: [--exit-non-zero-on-fix]
         files: ^Lib/test/
+        exclude: ^Lib/test/test_dstring.py
       - id: ruff-check
         name: Run Ruff (lint) on Platforms/WASI/
         args: [--exit-non-zero-on-fix, --config=Platforms/WASI/.ruff.toml]

Doc/reference/compound_stmts.rst

@@ -862,8 +862,8 @@ A literal pattern corresponds to most
 
 The rule ``strings`` and the token ``NUMBER`` are defined in the
 :doc:`standard Python grammar <./grammar>`.  Triple-quoted strings are
-supported.  Raw strings and byte strings are supported.  :ref:`f-strings`
-and :ref:`t-strings` are not supported.
+supported.  Raw strings, byte strings, and :ref:`d-strings` are supported.
+:ref:`f-strings` and :ref:`t-strings` are not supported.
 
 The forms ``signed_number '+' NUMBER`` and ``signed_number '-' NUMBER`` are
 for expressing :ref:`complex numbers <imaginary>`; they require a real number

Doc/reference/expressions.rst

@@ -298,7 +298,7 @@ To concatenate string expressions at run time, the '+' operator may be used::
    Hello Blaise
 
 Literal concatenation can freely mix raw strings, triple-quoted strings,
-and formatted string literals.
+dedented strings, and formatted string literals.
 For example::
 
    >>> "Hello" r', ' f"{name}!"

Doc/reference/lexical_analysis.rst

@@ -718,13 +718,17 @@ The allowed prefixes are:
 * ``r``: :ref:`Raw string <raw-strings>`
 * ``f``: :ref:`Formatted string literal <f-strings>` ("f-string")
 * ``t``: :ref:`Template string literal <t-strings>` ("t-string")
+* ``d``: :ref:`Dedented multiline string literal <d-strings>` ("d-string")
 * ``u``: No effect (allowed for backwards compatibility)
 
 See the linked sections for details on each type.
 
 Prefixes are case-insensitive (for example, '``B``' works the same as '``b``').
-The '``r``' prefix can be combined with '``f``', '``t``' or '``b``', so '``fr``',
-'``rf``', '``tr``', '``rt``', '``br``', and '``rb``' are also valid prefixes.
+The '``r``' prefix can be combined with '``f``', '``t``', '``b``', or
+'``d``'. The '``d``' prefix can also be combined with '``f``', '``t``', or
+'``b``', so the following prefix families are valid (in any order):
+``d``, ``dr``, ``db``, ``drb``, ``df``, ``drf``, ``dt``, and ``drt``.
+The ``u`` prefix cannot be combined with ``d``.
 
 .. versionadded:: 3.3
    The ``'rb'`` prefix of raw bytes literals has been added as a synonym
@@ -749,7 +753,10 @@ to indicate that an ending quote ends the literal.
    :group: python-grammar
 
    STRING:          [`stringprefix`] (`stringcontent`)
-   stringprefix:    <("r" | "u" | "b" | "br" | "rb"), case-insensitive>
+   stringprefix:    <("r" | "u" | "b" | "br" | "rb" |
+                      "d" | "dr" | "rd" | "db" | "bd" |
+                      "drb" | "dbr" | "rdb" | "rbd" | "bdr" | "brd"),
+                      case-insensitive>
    stringcontent:
       | "'''" ( !"'''" `longstringitem`)* "'''"
       | '"""' ( !'"""' `longstringitem`)* '"""'
@@ -1023,6 +1030,65 @@ that a single backslash followed by a newline is interpreted as those two
 characters as part of the literal, *not* as a line continuation.
 
 
+.. index::
+   single: dedented multiline string literal
+   single: d-string
+
+.. _d-strings:
+.. _dedented-string-literals:
+
+Dedented string literals
+------------------------
+
+.. versionadded:: 3.16
+
+A :dfn:`dedented multiline string literal` or :dfn:`d-string` is a string
+literal prefixed with '``d``' or '``D``'. It must use triple quotes, and the
+opening quotes must be immediately followed by a newline. This newline is not
+part of the resulting string.
+
+The longest prefix consisting only of spaces and tabs that is common to every
+non-blank line and to the line containing the closing quotes is removed from
+each non-blank line. Blank lines are normalized to a single newline and do not
+participate in finding the common indentation. The closing-quotes line always
+participates in that calculation, even when it contains no other characters.
+
+Dedentation is performed on the physical source lines before escape sequences
+are processed. Consequently, a line-continuation backslash is treated as part
+of the source line while its indentation is removed. Spaces and tabs are
+treated as different characters.
+
+For example::
+
+   >>> value = d"""spam"""
+   SyntaxError: d-string must start with a newline
+   >>> value = d"""
+   ...     first
+   ...       second
+   ...     """
+   >>> value
+   'first\n  second\n'
+   >>> value = d"""
+   ...     spam
+   ...     ham
+   ...     egg
+   ...   """
+   >>> value
+   '  spam\n  ham\n  egg\n'
+
+
+The ``d`` prefix can be combined with the compatible ``r``, ``b``, ``f``, and
+``t`` prefixes. In a ``df`` or ``dt`` string, a physical line participates in
+the outer common indentation calculation only if it starts outside a
+replacement field. A line that starts inside a replacement field is ignored
+even if constant text resumes later on that line. Only constant text outside
+replacement fields is dedented. A nested d-string is dedented independently.
+
+.. seealso::
+
+   :pep:`822` -- Dedented Multiline String
+
+
 .. index::
    single: formatted string literal
    single: interpolated string literal
@@ -1253,7 +1319,9 @@ sequences (``f_quote``).
 
    FSTRING_START:      `fstringprefix` ("'" | '"' | "'''" | '"""')
    FSTRING_END:        `f_quote`
-   fstringprefix:      <("f" | "fr" | "rf"), case-insensitive>
+   fstringprefix:      <("f" | "fr" | "rf" | "df" | "fd" | "drf" |
+                          "dfr" | "rdf" | "rfd" | "fdr" | "frd"),
+                          case-insensitive>
    f_debug_specifier:  '='
    f_quote:            <the quote character(s) used in FSTRING_START>
 
@@ -1587,4 +1655,3 @@ is also available in the :mod:`!token` module documentation.
 
 A sequence of three consecutive periods (``...``) has a special
 meaning as an :py:data:`Ellipsis` literal.
-

Include/internal/pycore_unicodeobject.h

@@ -342,6 +342,16 @@ extern PyObject* _PyUnicode_XStrip(
     );
 
 
+/* Search the longest common leading whitespace of all lines in [src, end).
+   Returns the length of the common leading whitespace and sets `*output` to
+   point to the beginning of the common leading whitespace if length > 0.
+   Export for 'test_peg_generator.test_c_parser'.
+   */
+PyAPI_FUNC(Py_ssize_t) _Py_search_longest_common_leading_whitespace(
+    const char *src,
+    const char *end,
+    const char **output);
+
 /* Dedent a string.
    Intended to dedent Python source. Unlike `textwrap.dedent`, this
    only supports spaces and tabs and doesn't normalize empty lines.

Lib/idlelib/colorizer.py

@@ -53,10 +53,19 @@ def make_pat():
     builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b"
     comment = any("COMMENT", [r"#[^\n]*"])
     stringprefix = r"(?i:r|u|f|fr|rf|b|br|rb|t|rt|tr)?"
+    triple_stringprefix = (
+        r"(?i:"
+        r"brd|bdr|rbd|rdb|dbr|drb|"
+        r"frd|fdr|rfd|rdf|dfr|drf|"
+        r"trd|tdr|rtd|rdt|dtr|drt|"
+        r"bd|db|rd|dr|fd|df|td|dt|d|"
+        r"r|u|f|fr|rf|b|br|rb|t|rt|tr"
+        r")?"
+    )
     sqstring = stringprefix + r"'[^'\\\n]*(\\.[^'\\\n]*)*'?"
     dqstring = stringprefix + r'"[^"\\\n]*(\\.[^"\\\n]*)*"?'
-    sq3string = stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?"
-    dq3string = stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?'
+    sq3string = triple_stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?"
+    dq3string = triple_stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?'
     string = any("STRING", [sq3string, dq3string, sqstring, dqstring])
     prog = re.compile("|".join([
                                 builtin, comment, string, kw,

Lib/idlelib/idle_test/test_colorizer.py

@@ -88,6 +88,24 @@ def test_prog(self):
         m = prog.search(line, m.end())
         eq(m.groupdict()['SYNC'], '\n')
 
+    def test_dstring_prefixes(self):
+        dstring_prefixes = (
+            'd', 'bd', 'db', 'rd', 'dr', 'fd', 'df', 'td', 'dt',
+            'brd', 'bdr', 'rbd', 'rdb', 'dbr', 'drb',
+            'frd', 'fdr', 'rfd', 'rdf', 'dfr', 'drf',
+            'trd', 'tdr', 'rtd', 'rdt', 'dtr', 'drt',
+        )
+        for prefix in dstring_prefixes:
+            for quote in ('\'\'\'', '"""'):
+                with self.subTest(prefix=prefix, quote=quote):
+                    source = f'{prefix}{quote}text{quote}'
+                    match = colorizer.prog.fullmatch(source)
+                    self.assertIsNotNone(match)
+                    self.assertEqual(match.groupdict()['STRING'], source)
+
+        # The d prefix is only valid for triple-quoted strings.
+        self.assertIsNone(colorizer.prog.match('d"text"'))
+
     def test_idprog(self):
         idprog = colorizer.idprog
         m = idprog.match('nospace')