gh-148675: Add Zd/Zf formats to array, ctypes, memoryview, struct (#148676)

* Add Zd/Zf format support to array, memoryview and struct.
* ctypes: Replace F/D/G complex format with Zf/Zd/Zg.
* Modify array, ctypes and struct modules to support format strings
  longer than 1 character (such as "Zd").
* Change array.typecodes type from str to tuple.

Author: vstinner

Doc/library/array.rst

@@ -52,6 +52,10 @@ defined:
 +-----------+--------------------+-------------------+-----------------------+-------+
 | ``'D'``   | double complex     | complex           | 16                    | \(4)  |
 +-----------+--------------------+-------------------+-----------------------+-------+
+| ``'Zf'``  | float complex      | complex           | 8                     | \(4)  |
++-----------+--------------------+-------------------+-----------------------+-------+
+| ``'Zd'``  | double complex     | complex           | 16                    | \(4)  |
++-----------+--------------------+-------------------+-----------------------+-------+
 
 
 Notes:
@@ -80,7 +84,7 @@ Notes:
    .. versionadded:: 3.15
 
 (4)
-   Complex types (``F`` and ``D``) are available unconditionally,
+   Complex types (``F``, ``D``, ``Zf`` and ``Zd``) are available unconditionally,
    regardless on support for complex types (the Annex G of the C11 standard)
    by the C compiler.
    As specified in the C11 standard, each complex type is represented by a
@@ -105,7 +109,10 @@ The module defines the following item:
 
 .. data:: typecodes
 
-   A string with all available type codes.
+   A tuple with all available type codes.
+
+   .. versionchanged:: next
+      The type changed from :class:`str` to :class:`tuple`.
 
 
 The module defines the following type:

Doc/library/ctypes.rst

@@ -370,15 +370,19 @@ in both C and ``libffi``, the following complex types are available:
    * - :class:`c_float_complex`
      - :c:expr:`float complex`
      - :py:class:`complex`
-     - ``'F'``
+     - ``'Zf'``
    * - :class:`c_double_complex`
      - :c:expr:`double complex`
      - :py:class:`complex`
-     - ``'D'``
+     - ``'Zd'``
    * - :class:`c_longdouble_complex`
      - :c:expr:`long double complex`
      - :py:class:`complex`
-     - ``'G'``
+     - ``'Zg'``
+
+.. versionchanged:: next
+   The :py:attr:`~_SimpleCData._type_` types ``F``, ``D`` and ``G`` have been
+   replaced with ``Zf``, ``Zd`` and ``Zg``.
 
 
 All these types can be created by calling them with an optional initializer of

Doc/library/struct.rst

@@ -264,6 +264,10 @@ platform-dependent.
 +--------+--------------------------+--------------------+----------------+------------+
 | ``D``  | :c:expr:`double complex` | complex            | 16             | \(10)      |
 +--------+--------------------------+--------------------+----------------+------------+
+| ``Zf`` | :c:expr:`float complex`  | complex            | 8              | \(10)      |
++--------+--------------------------+--------------------+----------------+------------+
+| ``Zd`` | :c:expr:`double complex` | complex            | 16             | \(10)      |
++--------+--------------------------+--------------------+----------------+------------+
 | ``s``  | :c:expr:`char[]`         | bytes              |                | \(9)       |
 +--------+--------------------------+--------------------+----------------+------------+
 | ``p``  | :c:expr:`char[]`         | bytes              |                | \(8)       |
@@ -280,6 +284,9 @@ platform-dependent.
 .. versionchanged:: 3.14
    Added support for the ``'F'`` and ``'D'`` formats.
 
+.. versionchanged:: next
+   Added support for the ``'Zf'`` and ``'Zd'`` formats.
+
 .. seealso::
 
    The :mod:`array` and :ref:`ctypes <ctypes-fundamental-data-types>` modules,
@@ -372,7 +379,7 @@ Notes:
    For the ``'F'`` and ``'D'`` format characters, the packed representation uses
    the IEEE 754 binary32 and binary64 format for components of the complex
    number, regardless of the floating-point format used by the platform.
-   Note that complex types (``F`` and ``D``) are available unconditionally,
+   Note that complex types (``F``/``Zf`` and ``D``/``Zd``) are available unconditionally,
    despite complex types being an optional feature in C.
    As specified in the C11 standard, each complex type is represented by a
    two-element C array containing, respectively, the real and imaginary parts.

Doc/whatsnew/3.15.rst

@@ -671,9 +671,9 @@ Other language changes
   (Contributed by James Hilton-Balfe in :gh:`128335`.)
 
 * The class :class:`memoryview` now supports the :c:expr:`float complex` and
-  :c:expr:`double complex` C types: formatting characters ``'F'`` and ``'D'``
-  respectively.
-  (Contributed by Sergey B Kirpichev in :gh:`146151`.)
+  :c:expr:`double complex` C types: formatting characters ``'F'``/``'Zf'``
+  and ``'D'``/``'Zd'`` respectively.
+  (Contributed by Victor Stinner in :gh:`146151` and :gh:`148675`.)
 
 * Allow the *count* argument of :meth:`bytes.replace` to be a keyword.
   (Contributed by Stan Ulbrych in :gh:`147856`.)
@@ -724,13 +724,17 @@ array
 -----
 
 * Support the :c:expr:`float complex` and :c:expr:`double complex` C types:
-  formatting characters ``'F'`` and ``'D'`` respectively.
-  (Contributed by Sergey B Kirpichev in :gh:`146151`.)
+  formatting characters ``'F'``/``'Zf'`` and ``'D'``/``'Zd'`` respectively.
+  (Contributed by Victor Stinner in :gh:`146151` and :gh:`148675`.)
 
 * Support half-floats (16-bit IEEE 754 binary interchange format): formatting
   character ``'e'``.
   (Contributed by Sergey B Kirpichev in :gh:`146238`.)
 
+* The :data:`array.typecodes` type changed from :class:`str` to :class:`tuple`
+  to support type codes longer than 1 character (``Zf`` and ``Zd``).
+  (Contributed by Victor Stinner in :gh:`148675`.)
+
 
 ast
 ---
@@ -1741,6 +1745,12 @@ ctypes
   which has been deprecated since Python 3.13.
   (Contributed by Bénédikt Tran in :gh:`133866`.)
 
+* Change the :py:attr:`~ctypes._SimpleCData._type_` of
+  :class:`~ctypes.c_float_complex`, :class:`~ctypes.c_double_complex` and
+  :class:`~ctypes.c_longdouble_complex` from ``F``, ``D`` and ``G`` to ``Zf``,
+  ``Zd`` and ``Zg`` for compatibility with numpy.
+  (Contributed by Victor Stinner in :gh:`148675`.)
+
 
 datetime
 --------

Lib/ctypes/__init__.py

@@ -206,13 +206,13 @@ class c_longdouble(_SimpleCData):
 
 try:
     class c_double_complex(_SimpleCData):
-        _type_ = "D"
+        _type_ = "Zd"
     _check_size(c_double_complex)
     class c_float_complex(_SimpleCData):
-        _type_ = "F"
+        _type_ = "Zf"
     _check_size(c_float_complex)
     class c_longdouble_complex(_SimpleCData):
-        _type_ = "G"
+        _type_ = "Zg"
 except AttributeError:
     pass
 

Lib/test/test_array.py

@@ -31,7 +31,10 @@ class ArraySubclassWithKwargs(array.array):
     def __init__(self, typecode, newarg=None):
         array.array.__init__(self)
 
-typecodes = 'uwbBhHiIlLfdqQFDe'
+typecodes = (
+    'u', 'w', 'b', 'B', 'h', 'H', 'i', 'I', 'l', 'L',
+    'f', 'd', 'q', 'Q', 'F', 'D', 'e', 'Zf', 'Zd')
+
 
 class MiscTest(unittest.TestCase):
 
@@ -42,8 +45,9 @@ def test_array_is_sequence(self):
     def test_bad_constructor(self):
         self.assertRaises(TypeError, array.array)
         self.assertRaises(TypeError, array.array, spam=42)
-        self.assertRaises(TypeError, array.array, 'xx')
+        self.assertRaises(ValueError, array.array, 'xx')
         self.assertRaises(ValueError, array.array, 'x')
+        self.assertRaises(ValueError, array.array, 'Z')
 
     @support.cpython_only
     def test_disallow_instantiation(self):
@@ -85,6 +89,12 @@ def __index__(self):
                 with self.assertRaises(TypeError):
                     a.fromlist(lst)
 
+    def test_typecodes(self):
+        self.assertIsInstance(array.typecodes, tuple)
+        for typecode in array.typecodes:
+            self.assertIsInstance(typecode, str)
+            self.assertGreaterEqual(len(typecode), 1)
+
 
 # Machine format codes.
 #
@@ -208,6 +218,14 @@ def test_numbers(self):
              [9006104071832581.0j, float('inf'), complex('1-infj'), -0.0]),
             (['D'], IEEE_754_DOUBLE_COMPLEX_BE, '>DDDD',
              [9006104071832581.0j, float('inf'), complex('1-infj'), -0.0]),
+            (['Zf'], IEEE_754_FLOAT_COMPLEX_LE, '<ZfZfZfZf',
+             [16711938.0j, float('inf'), complex('1-infj'), -0.0]),
+            (['Zf'], IEEE_754_FLOAT_COMPLEX_BE, '>ZfZfZfZf',
+             [16711938.0j, float('inf'), complex('1-infj'), -0.0]),
+            (['Zd'], IEEE_754_DOUBLE_COMPLEX_LE, '<ZdZdZdZd',
+             [9006104071832581.0j, float('inf'), complex('1-infj'), -0.0]),
+            (['Zd'], IEEE_754_DOUBLE_COMPLEX_BE, '>ZdZdZdZd',
+             [9006104071832581.0j, float('inf'), complex('1-infj'), -0.0]),
         )
         for testcase in testcases:
             valid_typecodes, mformat_code, struct_fmt, values = testcase
@@ -1237,6 +1255,9 @@ def test_free_after_iterating(self):
         support.check_free_after_iterating(self, reversed, array.array,
                                            (self.typecode,))
 
+    def test_known_typecode(self):
+        self.assertIn(self.typecode, array.typecodes)
+
 class StringTest(BaseTest):
 
     def test_setitem(self):
@@ -1576,7 +1597,7 @@ def test_nan(self):
     def test_byteswap(self):
         a = array.array(self.typecode, self.example)
         self.assertRaises(TypeError, a.byteswap, 42)
-        if a.itemsize in (1, 2, 4, 8):
+        if a.itemsize in (1, 2, 4, 8, 16):
             b = array.array(self.typecode, self.example)
             b.byteswap()
             if a.itemsize == 1:
@@ -1631,6 +1652,14 @@ class ComplexDoubleTest(CFPTest, unittest.TestCase):
     typecode = 'D'
     minitemsize = 16
 
+class ComplexZfFloatTest(CFPTest, unittest.TestCase):
+    typecode = 'Zf'
+    minitemsize = 8
+
+class ComplexZdDoubleTest(CFPTest, unittest.TestCase):
+    typecode = 'Zd'
+    minitemsize = 16
+
 
 class LargeArrayTest(unittest.TestCase):
     typecode = 'b'

Lib/test/test_buffer.py

@@ -67,7 +67,7 @@
     'h':0, 'H':0, 'i':0, 'I':0,
     'l':0, 'L':0, 'n':0, 'N':0,
     'e':0, 'f':0, 'd':0, 'P':0,
-    'F':0, 'D':0
+    'F':0, 'D':0, 'Zf':0, 'Zd':0,
 }
 
 # NumPy does not have 'n' or 'N':
@@ -95,7 +95,9 @@
     'e':(-65519, 65520),   'f':(-(1<<63), 1<<63),
     'd':(-(1<<1023), 1<<1023),
     'F':(-(1<<63), 1<<63),
-    'D':(-(1<<1023), 1<<1023)
+    'D':(-(1<<1023), 1<<1023),
+    'Zf':(-(1<<63), 1<<63),
+    'Zd':(-(1<<1023), 1<<1023),
 }
 
 def native_type_range(fmt):
@@ -110,9 +112,9 @@ def native_type_range(fmt):
         lh = (-(1<<63), 1<<63)
     elif fmt == 'd':
         lh = (-(1<<1023), 1<<1023)
-    elif fmt == 'F':
+    elif fmt in ('F', 'Zf'):
         lh = (-(1<<63), 1<<63)
-    elif fmt == 'D':
+    elif fmt in ('D', 'Zd'):
         lh = (-(1<<1023), 1<<1023)
     else:
         for exp in (128, 127, 64, 63, 32, 31, 16, 15, 8, 7):
@@ -182,18 +184,28 @@ def randrange_fmt(mode, char, obj):
     if char in 'efd':
         x = struct.pack(char, x)
         x = struct.unpack(char, x)[0]
-    if char in 'FD':
+    if char in ('F', 'D', 'Zf', 'Zd'):
         y = randrange(*fmtdict[mode][char])
         x = complex(x, y)
         x = struct.pack(char, x)
         x = struct.unpack(char, x)[0]
     return x
 
+def split_format(fmt):
+    i = 0
+    while i < len(fmt):
+        if fmt[i] == 'Z':
+            n = 2
+        else:
+            n = 1
+        yield fmt[i:i + n]
+        i += n
+
 def gen_item(fmt, obj):
     """Return single random item."""
     mode, chars = fmt.split('#')
     x = []
-    for c in chars:
+    for c in split_format(chars):
         x.append(randrange_fmt(mode, c, obj))
     return x[0] if len(x) == 1 else tuple(x)
 
@@ -254,9 +266,7 @@ def is_byte_format(fmt):
 
 def is_memoryview_format(fmt):
     """format suitable for memoryview"""
-    x = len(fmt)
-    return ((x == 1 or (x == 2 and fmt[0] == '@')) and
-            fmt[x-1] in MEMORYVIEW)
+    return fmt.removeprefix('@') in MEMORYVIEW
 
 NON_BYTE_FORMAT = [c for c in fmtdict['@'] if not is_byte_format(c)]
 
@@ -648,14 +658,22 @@ def ndarray_from_structure(items, fmt, t, flags=0):
     return ndarray(items, shape=shape, strides=strides, format=fmt,
                    offset=offset, flags=ND_WRITABLE|flags)
 
+# Convert PEP 3118 formats to numpy dtypes
+FORMAT_TO_DTYPE = {
+    'Zf': 'F',
+    'Zd': 'D',
+}
+
 def numpy_array_from_structure(items, fmt, t):
     """Return numpy_array from the tuple returned by rand_structure()"""
     memlen, itemsize, ndim, shape, strides, offset = t
     buf = bytearray(memlen)
     for j, v in enumerate(items):
         struct.pack_into(fmt, buf, j*itemsize, v)
+    # Replace Zd/Zf formats with D/F dtypes
+    dtype = FORMAT_TO_DTYPE.get(fmt, fmt)
     return numpy_array(buffer=buf, shape=shape, strides=strides,
-                       dtype=fmt, offset=offset)
+                       dtype=dtype, offset=offset)
 
 
 # ======================================================================
@@ -3037,7 +3055,7 @@ def test_memoryview_assign(self):
                 continue
             m2 = m1.cast(fmt)
             lo, hi = _range
-            if fmt in "dfDF":
+            if fmt in ("d", "f", "D", "F", "Zd", "Zf"):
                 lo, hi = -2**1024, 2**1024
             if fmt != 'P': # PyLong_AsVoidPtr() accepts negative numbers
                 self.assertRaises(ValueError, m2.__setitem__, 0, lo-1)

Lib/test/test_ctypes/test_c_simple_type_meta.py

@@ -213,14 +213,12 @@ def test_bad_type_message(self):
             class F(metaclass=PyCSimpleType):
                 _type_ = "\0"
         message = str(cm.exception)
-        expected_type_chars = list('cbBhHiIlLdDFGfuzZqQPXOv?g')
-        if not hasattr(ctypes, 'c_float_complex'):
-            expected_type_chars.remove('F')
-            expected_type_chars.remove('D')
-            expected_type_chars.remove('G')
+        expected_type_chars = list('cbBhHiIlLdfuzZqQPXOv?g')
         if not MS_WINDOWS:
             expected_type_chars.remove('X')
         self.assertIn("'" + ''.join(expected_type_chars) + "'", message)
+        if hasattr(ctypes, 'c_float_complex'):
+            self.assertIn("'Zf', 'Zd', 'Zg'", message)
 
     def test_creating_pointer_in_dunder_init_3(self):
         """Check if interfcase subclasses properly creates according internal

Lib/test/test_ctypes/test_numbers.py

@@ -119,15 +119,24 @@ def test_floats(self):
     @unittest.skipUnless(hasattr(ctypes, "c_double_complex"),
                          "requires C11 complex type")
     def test_complex(self):
-        for t in [ctypes.c_double_complex, ctypes.c_float_complex,
-                  ctypes.c_longdouble_complex]:
+        for format, t in [
+            ('Zd', ctypes.c_double_complex),
+            ('Zf', ctypes.c_float_complex),
+            ('Zg', ctypes.c_longdouble_complex),
+        ]:
             self.assertEqual(t(1).value, 1+0j)
             self.assertEqual(t(1.0).value, 1+0j)
             self.assertEqual(t(1+0.125j).value, 1+0.125j)
             self.assertEqual(t(IndexLike()).value, 2+0j)
             self.assertEqual(t(FloatLike()).value, 2+0j)
             self.assertEqual(t(ComplexLike()).value, 1+1j)
 
+            prefix = '>' if sys.byteorder == 'big' else '<'
+            num = t(1.0)
+            self.assertEqual(memoryview(num).format, prefix + format)
+            array = (t * 3)()
+            self.assertEqual(memoryview(array).format, prefix + format)
+
     @unittest.skipUnless(hasattr(ctypes, "c_double_complex"),
                          "requires C11 complex type")
     def test_complex_round_trip(self):