Merge branch 'pybind11clif_main' into iwyu_pragmas_pybind11clif

CMakeLists.txt

@@ -170,7 +170,8 @@ set(PYBIND11_HEADERS
     include/pybind11/stl/filesystem.h
     include/pybind11/trampoline_self_life_support.h
     include/pybind11/type_caster_pyobject_ptr.h
-    include/pybind11/typing.h)
+    include/pybind11/typing.h
+    include/pybind11/warnings.h)
 
 # Compare with grep and warn if mismatched
 if(PYBIND11_MASTER_PROJECT)

README.rst

@@ -1,193 +1,18 @@
-===========================================================================
-pybind11k — A fork of pybind11 set up for sustained innovation & continuity
-===========================================================================
+============================================================================
+pybind11clif — A fork of pybind11 to support the PyCLIF-pybind11 unification
+============================================================================
 
-Warning — New features are still in flux
-========================================
+WARNING — The PyCLIF-pybind11 unification was SUSPENDED.
+========================================================
 
-pybind11k is used Google-internally to build thousands of extensions that are deployed to production and is also regularly tested with all CLANG sanitizers. However, feature stability guarantees are currently limited in this way:
+Possibly, **THIS FORK WILL NO LONGER BE UPDATED**.
 
-0. pybind11k is meant to maximize backward-compatiblity with `pybind11 (master) <https://github.com/pybind/pybind11/tree/master>`_.
+Please use the upstream project: https://github.com/pybind/pybind11
 
-1. The main driving force for adding new features is the PyCLIF-pybind11 integration work (for Googlers: `go/pyclif_pybind11_fusion <http://go/pyclif_pybind11_fusion>`_). Until this work is completed, it is impractical for us to take external use cases into account when evolving new features. This is because we can globally test changes fairly easily internally, but not externally.
+For completeness, this repo was renamed twice:
 
-2. After the PyCLIF-pybind11 work is completed we will commit to greater feature stability and update the documentation.
+* Originally the name was google/pywrapcc (Feb 2023),
 
-Outstanding new features: ``py::native_enum`` (google/pybind11k#30005), ``py::return_value_policy_pack`` (google/pybind11k#30011), enhanced pybind11/functional.h API (google/pybind11k#30022)
+* then google/pybind11k (Apr 2024),
 
-Background / Overview
-=====================
-
-This pybind11k fork originated from the `pybind11 smart_holder <https://github.com/pybind/pybind11/tree/smart_holder>`_ branch. It was created with two important goals in mind:
-
-1. Sustained innovation and proactive bug fixes.
-2. Sustained continuity.
-
-With original pybind11, these two goals are in a conflict. In the early days of pybind11 this was not so much of a problem, but this has been changing with growing adoption.
-
-Regarding goal 1: pybind11 has two serious long-standing bugs (pybind/pybind11#1138, pybind/pybind11#1333) that have never been fixed on the master branch, but were fixed on the smart_holder branch in early-mid 2021.
-
-Regarding goal 2: The original and still current ``pybind11::smart_holder`` implementation is a compromise solution that avoids an `"ABI break" <https://github.com/pybind/pybind11/blob/09db6445d8da6e918c2d2be3aa4e7b0ddd8077c7/include/pybind11/detail/internals.h#L25>`_, concretely, changes to the ``pybind11::details::internals`` ``struct``. Each time the ``internals`` are changed, the ``PYBIND11_INTERNALS_VERSION`` needs to be incremented, cutting off interoperability with existing PyPI wheels based on pybind11, without giving any hint about this problem at runtime. In the meantime, two other PRs were merged on the pybind11 master branch that require changes to the ``internals``, PRs pybind/pybind11#3275 and pybind/pybind11#4254. To avoid ABI breaks, these PRs were effectively held back behind ``#ifdef`` s. This problem came to a breaking point with PR pybind/pybind11#4329, for which hiding the new feature behind ``#ifdef`` s is not a practical option.
-
-Obviously, neither repeatedly breaking interoperability with existing PyPI wheels, nor holding back bug fixes and new features, is desirable. Therefore PR pybind/pybind11#4329 was extended to include a generalization of the ``internals`` approach, under the name ``cross_extension_shared_state``. The fundamental difference is that established shared state is left untouched, and new shared states are added as needed, largely resolving the conflict between innovation & continuity. The price to pay is added complexity managing the evolving shared states, but that is assumed to be a relatively small extra effort for a few developers, resulting in a big usability gain for a much larger number of users. Ultimately, this is just the familiar innovate-deprecate-cleanup life cycle typical for many (all?) long-term successful major projects (e.g. C++, Python). Even pybind11k developers are likely to see this as a win worth paying a price for, because they are more free to innovate.
-
-A direct consequence of goal 2. is that the C++ pybind11 namespace cannot abruptly be changed, because renaming would break both API and ABI compatibility. The intent is to change the API gradually, driven primarily by code health and innovation-related refactoring needs.
-
-Notes regarding the choice of name for this fork
-================================================
-
-The main motivations for this choice of name are:
-
-* We want it to be immediately obvious that pybind11k is rooted in pybind11. Users should not be surprised to find that the main include is still ``pybind11/pybind11.h``, the C++ namespace name is still ``pybind11``, the C++ macros are still prefixed with ``PYBIND11_``, and that ``pip install pybind11k`` (not yet available) produces an installation similar to that of ``pip install pybind11``.
-
-* ``11k`` is short for ``11000``, not to be taken too seriously, but it is meant to signal that we will inevitably need to move beyond the C++11 standard, and that any software needs to evolve to survive long-term.
-
-* And yes, the ``k`` is also playing on `"Py3k" <https://www.python.org/download/releases/3.0/>`_.
-
-ORIGINAL pybind11 README below (to be updated)
-==============================================
-
-**pybind11** is a lightweight header-only library that exposes C++ types
-in Python and vice versa, mainly to create Python bindings of existing
-C++ code. Its goals and syntax are similar to the excellent
-`Boost.Python <http://www.boost.org/doc/libs/1_58_0/libs/python/doc/>`_
-library by David Abrahams: to minimize boilerplate code in traditional
-extension modules by inferring type information using compile-time
-introspection.
-
-The main issue with Boost.Python—and the reason for creating such a
-similar project—is Boost. Boost is an enormously large and complex suite
-of utility libraries that works with almost every C++ compiler in
-existence. This compatibility has its cost: arcane template tricks and
-workarounds are necessary to support the oldest and buggiest of compiler
-specimens. Now that C++11-compatible compilers are widely available,
-this heavy machinery has become an excessively large and unnecessary
-dependency.
-
-Think of this library as a tiny self-contained version of Boost.Python
-with everything stripped away that isn't relevant for binding
-generation. Without comments, the core header files only require ~4K
-lines of code and depend on Python (3.8+, or PyPy) and the C++
-standard library. This compact implementation was possible thanks to
-some C++11 language features (specifically: tuples, lambda functions and
-variadic templates). Since its creation, this library has grown beyond
-Boost.Python in many ways, leading to dramatically simpler binding code in many
-common situations.
-
-Tutorial and reference documentation is provided at
-`pybind11.readthedocs.io <https://pybind11.readthedocs.io/en/latest>`_.
-A PDF version of the manual is available
-`here <https://pybind11.readthedocs.io/_/downloads/en/latest/pdf/>`_.
-And the source code is always available at
-`github.com/pybind/pybind11 <https://github.com/pybind/pybind11>`_.
-
-
-Core features
--------------
-
-
-pybind11 can map the following core C++ features to Python:
-
-- Functions accepting and returning custom data structures per value,
-  reference, or pointer
-- Instance methods and static methods
-- Overloaded functions
-- Instance attributes and static attributes
-- Arbitrary exception types
-- Enumerations
-- Callbacks
-- Iterators and ranges
-- Custom operators
-- Single and multiple inheritance
-- STL data structures
-- Smart pointers with reference counting like ``std::shared_ptr``
-- Internal references with correct reference counting
-- C++ classes with virtual (and pure virtual) methods can be extended
-  in Python
-- Integrated NumPy support (NumPy 2 requires pybind11 2.12+)
-
-Goodies
--------
-
-In addition to the core functionality, pybind11 provides some extra
-goodies:
-
-- Python 3.8+, and PyPy3 7.3 are supported with an implementation-agnostic
-  interface (pybind11 2.9 was the last version to support Python 2 and 3.5).
-
-- It is possible to bind C++11 lambda functions with captured
-  variables. The lambda capture data is stored inside the resulting
-  Python function object.
-
-- pybind11 uses C++11 move constructors and move assignment operators
-  whenever possible to efficiently transfer custom data types.
-
-- It's easy to expose the internal storage of custom data types through
-  Pythons' buffer protocols. This is handy e.g. for fast conversion
-  between C++ matrix classes like Eigen and NumPy without expensive
-  copy operations.
-
-- pybind11 can automatically vectorize functions so that they are
-  transparently applied to all entries of one or more NumPy array
-  arguments.
-
-- Python's slice-based access and assignment operations can be
-  supported with just a few lines of code.
-
-- Everything is contained in just a few header files; there is no need
-  to link against any additional libraries.
-
-- Binaries are generally smaller by a factor of at least 2 compared to
-  equivalent bindings generated by Boost.Python. A recent pybind11
-  conversion of PyRosetta, an enormous Boost.Python binding project,
-  `reported <https://graylab.jhu.edu/Sergey/2016.RosettaCon/PyRosetta-4.pdf>`_
-  a binary size reduction of **5.4x** and compile time reduction by
-  **5.8x**.
-
-- Function signatures are precomputed at compile time (using
-  ``constexpr``), leading to smaller binaries.
-
-- With little extra effort, C++ types can be pickled and unpickled
-  similar to regular Python objects.
-
-Supported compilers
--------------------
-
-1. Clang/LLVM 3.3 or newer (for Apple Xcode's clang, this is 5.0.0 or
-   newer)
-2. GCC 4.8 or newer
-3. Microsoft Visual Studio 2017 or newer
-4. Intel classic C++ compiler 18 or newer (ICC 20.2 tested in CI)
-5. Cygwin/GCC (previously tested on 2.5.1)
-6. NVCC (CUDA 11.0 tested in CI)
-7. NVIDIA PGI (20.9 tested in CI)
-
-About
------
-
-This project was created by `Wenzel
-Jakob <http://rgl.epfl.ch/people/wjakob>`_. Significant features and/or
-improvements to the code were contributed by Jonas Adler, Lori A. Burns,
-Sylvain Corlay, Eric Cousineau, Aaron Gokaslan, Ralf Grosse-Kunstleve, Trent Houliston, Axel
-Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov, Johan Mabille, Tomasz Miąsko,
-Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim
-Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart.
-
-We thank Google for a generous financial contribution to the continuous
-integration infrastructure used by this project.
-
-
-Contributing
-~~~~~~~~~~~~
-
-See the `contributing
-guide <https://github.com/pybind/pybind11/blob/master/.github/CONTRIBUTING.md>`_
-for information on building and contributing to pybind11.
-
-License
-~~~~~~~
-
-pybind11 is provided under a BSD-style license that can be found in the
-`LICENSE <https://github.com/pybind/pybind11/blob/master/LICENSE>`_
-file. By using, distributing, or contributing to this project, you agree
-to the terms and conditions of this license.
+* then google/pybind11clif (Aug 2024).

docs/classes.rst

@@ -557,4 +557,4 @@ The ``name`` property returns the name of the enum value as a unicode string.
 .. note::
 
     ``py::native_enum`` was added as an alternative to ``py::enum_``
-    with http://github.com/google/pybind11k/pull/30005
+    with http://github.com/google/pybind11clif/pull/30005

docs/faq.rst

@@ -247,6 +247,50 @@ been received, you must either explicitly interrupt execution by throwing
         });
     }
 
+What is a highly conclusive and simple way to find memory leaks (e.g. in pybind11 bindings)?
+============================================================================================
+
+Use ``while True`` & ``top`` (Linux, macOS).
+
+For example, locally change tests/test_type_caster_pyobject_ptr.py like this:
+
+.. code-block:: diff
+
+     def test_return_list_pyobject_ptr_reference():
+    +  while True:
+         vec_obj = m.return_list_pyobject_ptr_reference(ValueHolder)
+         assert [e.value for e in vec_obj] == [93, 186]
+         # Commenting out the next `assert` will leak the Python references.
+         # An easy way to see evidence of the leaks:
+         # Insert `while True:` as the first line of this function and monitor the
+         # process RES (Resident Memory Size) with the Unix top command.
+    -    assert m.dec_ref_each_pyobject_ptr(vec_obj) == 2
+    +    # assert m.dec_ref_each_pyobject_ptr(vec_obj) == 2
+
+Then run the test as you would normally do, which will go into the infinite loop.
+
+**In another shell, but on the same machine** run:
+
+.. code-block:: bash
+
+    top
+
+This will show:
+
+.. code-block::
+
+        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
+    1266095 rwgk      20   0 5207496 611372  45696 R 100.0   0.3   0:08.01 test_type_caste
+
+Look for the number under ``RES`` there. You'll see it going up very quickly.
+
+**Don't forget to Ctrl-C the test command** before your machine becomes
+unresponsive due to swapping.
+
+This method only takes a couple minutes of effort and is very conclusive.
+What you want to see is that the ``RES`` number is stable after a couple
+seconds.
+
 CMake doesn't detect the right Python version
 =============================================
 

include/pybind11/cast.h

@@ -1947,6 +1947,18 @@ struct arg : detail::arg_base {
     template <typename T>
     arg_v operator=(T &&value) const;
 
+    /// Same as `arg_base::noconvert()`, but returns *this as arg&, not arg_base&
+    arg &noconvert(bool flag = true) {
+        arg_base::noconvert(flag);
+        return *this;
+    }
+
+    /// Same as `arg_base::noconvert()`, but returns *this as arg&, not arg_base&
+    arg &none(bool flag = true) {
+        arg_base::none(flag);
+        return *this;
+    }
+
     arg &policies(const from_python_policies &policies) {
         m_policies = policies;
         return *this;
@@ -2017,6 +2029,12 @@ struct arg_v : arg {
         return *this;
     }
 
+    /// Same as `arg::policies()`, but returns *this as arg_v&, not arg&
+    arg_v &policies(const from_python_policies &policies) {
+        arg::policies(policies);
+        return *this;
+    }
+
     /// The default value
     object value;
     bool value_is_nullptr = false;

include/pybind11/detail/class.h

@@ -200,7 +200,7 @@ inline bool ensure_base_init_functions_were_called(PyObject *self) {
     return true;
 }
 
-// See google/pybind11k#30095 for background.
+// See google/pybind11clif#30095 for background.
 #if !defined(PYBIND11_INIT_SAFETY_CHECKS_VIA_INTERCEPTING_TP_INIT)                                \
     && !defined(PYBIND11_INIT_SAFETY_CHECKS_VIA_DEFAULT_PYBIND11_METACLASS)
 #    if !defined(PYPY_VERSION)

include/pybind11/detail/function_record_pyobject.h

@@ -2,7 +2,7 @@
 // All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
-// For background see the description of PR google/pybind11k#30099.
+// For background see the description of PR google/pybind11clif#30099.
 
 // IWYU pragma: private, include "third_party/pybind11/include/pybind11/pybind11.h"
 

include/pybind11/detail/struct_smart_holder.h

@@ -217,7 +217,7 @@ struct smart_holder {
         // race conditions, but in the context of Python it is a bug (elsewhere)
         // if the Global Interpreter Lock (GIL) is not being held when this code
         // is reached.
-        // SMART_HOLDER_WIP: IMPROVABLE: assert(GIL is held).
+        // PYBIND11:REMINDER: This may need to be protected by a mutex in free-threaded Python.
         if (vptr.use_count() != 1) {
             throw std::invalid_argument(std::string("Cannot disown use_count != 1 (") + context
                                         + ").");
@@ -279,29 +279,32 @@ struct smart_holder {
         return hld;
     }
 
-    // Caller is responsible for ensuring preconditions (SMART_HOLDER_WIP: details).
+    // Caller is responsible for ensuring the complex preconditions
+    // (see `smart_holder_type_caster_support::load_helper`).
     void disown() {
         reset_vptr_deleter_armed_flag(false);
         is_disowned = true;
     }
 
-    // Caller is responsible for ensuring preconditions (SMART_HOLDER_WIP: details).
+    // Caller is responsible for ensuring the complex preconditions
+    // (see `smart_holder_type_caster_support::load_helper`).
     void reclaim_disowned() {
         reset_vptr_deleter_armed_flag(true);
         is_disowned = false;
     }
 
-    // Caller is responsible for ensuring preconditions (SMART_HOLDER_WIP: details).
+    // Caller is responsible for ensuring the complex preconditions
+    // (see `smart_holder_type_caster_support::load_helper`).
     void release_disowned() { vptr.reset(); }
 
-    // SMART_HOLDER_WIP: review this function.
     void ensure_can_release_ownership(const char *context = "ensure_can_release_ownership") const {
         ensure_is_not_disowned(context);
         ensure_vptr_is_using_builtin_delete(context);
         ensure_use_count_1(context);
     }
 
-    // Caller is responsible for ensuring preconditions (SMART_HOLDER_WIP: details).
+    // Caller is responsible for ensuring the complex preconditions
+    // (see `smart_holder_type_caster_support::load_helper`).
     void release_ownership() {
         reset_vptr_deleter_armed_flag(false);
         release_disowned();

include/pybind11/detail/type_caster_base.h

@@ -478,7 +478,7 @@ inline PyObject *make_new_instance(PyTypeObject *type);
 
 #ifdef PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT
 
-// SMART_HOLDER_WIP: Needs refactoring of existing pybind11 code.
+// PYBIND11:REMINDER: Needs refactoring of existing pybind11 code.
 inline bool deregister_instance(instance *self, void *valptr, const type_info *tinfo);
 
 PYBIND11_NAMESPACE_BEGIN(smart_holder_type_caster_support)
@@ -571,8 +571,7 @@ handle smart_holder_from_unique_ptr(std::unique_ptr<T, D> &&src,
 
     if (static_cast<void *>(src.get()) == src_raw_void_ptr) {
         // This is a multiple-inheritance situation that is incompatible with the current
-        // shared_from_this handling (see PR #3023).
-        // SMART_HOLDER_WIP: IMPROVABLE: Is there a better solution?
+        // shared_from_this handling (see PR #3023). Is there a better solution?
         src_raw_void_ptr = nullptr;
     }
     auto smhldr = smart_holder::from_unique_ptr(std::move(src), src_raw_void_ptr);
@@ -628,8 +627,8 @@ handle smart_holder_from_shared_ptr(const std::shared_ptr<T> &src,
     void *src_raw_void_ptr = static_cast<void *>(src_raw_ptr);
     const detail::type_info *tinfo = st.second;
     if (handle existing_inst = find_registered_python_instance(src_raw_void_ptr, tinfo)) {
-        // SMART_HOLDER_WIP: MISSING: Enforcement of consistency with existing smart_holder.
-        // SMART_HOLDER_WIP: MISSING: keep_alive.
+        // PYBIND11:REMINDER: MISSING: Enforcement of consistency with existing smart_holder.
+        // PYBIND11:REMINDER: MISSING: keep_alive.
         return existing_inst;
     }