Pybind11: Converting List to Vector and Back

In the realm of Python and C++ interoperability, pybind11 is a vital library. It allows you to expose C++ classes and functions to Python seamlessly. One common task that often comes up when working with pybind11 is the conversion between Python lists and C++ std::vector objects. In this comprehensive guide, we will explore different ways to perform this conversion effectively.

Before we dive in, let’s make sure you have a grasp of the prerequisites:

  • Basic understanding of C++ programming
  • Familiarity with Python programming language
  • Knowledge of how to install and use pybind11

Note: For the purpose of this guide, it is assumed that you have already set up a pybind11 project. If you haven’t, you can quickly get started by following the official documentation.

Setting Up the Environment

Prerequisites

Before proceeding, you’ll need to have the following installed:

  1. C++ Compiler (e.g., GCC, Clang)
  2. Python 3.x
  3. CMake
  4. pybind11 library

Installing pybind11

You can install pybind11 using multiple methods. Here are some options:

pip install pybind11

git clone https://github.com/pybind/pybind11.git
cd pybind11
mkdir build
cd build
cmake ..
make install

Sample Project Structure

For the purpose of this guide, let’s assume you have the following directory structure:

my_pybind_project/
├── CMakeLists.txt
└── src/
    └── example.cpp

Note: We will be writing our code in example.cpp.

Basic Conversion: Python List to std::vector

The Basics of Conversion in pybind11

Converting a Python list to a C++ std::vector is straightforward with pybind11, thanks to its automatic type conversion. This allows a C++ function to accept Python lists as arguments, automatically converting them to std::vector types.

Example 1: Converting a List of Integers

Here’s how to write a simple C++ function that takes a Python list (which will be converted to an std::vector<int>) and returns the sum of its elements.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h> // Necessary for handling std::vector
#include <vector>

namespace py = pybind11;

int sum_of_elements(std::vector<int> const &vec) {
    int sum = 0;
    for (const auto &elem : vec) {
        sum += elem;
    }
    return sum;
}

PYBIND11_MODULE(example, m) {
    m.def("sum_of_elements", &sum_of_elements);
}

In the Python interpreter, you can then do:

import example

result = example.sum_of_elements([1, 2, 3, 4])
print(result)  # Output: 10

Important Notes

  1. Header File: Make sure to include <pybind11/stl.h> in your code. This header contains necessary definitions for converting STL containers like std::vector.
  2. Type Safety: pybind11 performs type checking, so if the Python list contains anything other than integers, a TypeError will be thrown.

Advanced Conversion: Python List to std::vector of Custom Objects

When Built-in Types Are Not Enough

The simple examples are neat, but what if you have a Python list of custom objects that you want to convert into a std::vector of custom C++ objects? Here’s how you can do that.

Example 2: Converting a List of Custom Objects

Suppose we have a simple Person class in C++:

class Person {
public:
    Person(const std::string &name, int age) : name(name), age(age) {}

    std::string name;
    int age;
};

You can expose this class to Python using pybind11 like so:

PYBIND11_MODULE(example, m) {
    py::class_<Person>(m, "Person")
        .def(py::init<const std::string &, int>())
        .def_readwrite("name", &Person::name)
        .def_readwrite("age", &Person::age);
}

Now let’s say you want to write a C++ function that takes a Python list of Person objects:

void display_people(const std::vector<Person> &people) {
    for (const auto &person : people) {
        std::cout << "Name: " << person.name << ", Age: " << person.age << std::endl;
    }
}

You can bind this function as well:

PYBIND11_MODULE(example, m) {
    py::class_<Person>(m, "Person")
        .def(py::init<const std::string &, int>())
        .def_readwrite("name", &Person::name)
        .def_readwrite("age", &Person::age);

    m.def("display_people", &display_people);
}

In Python, you can then call this function with a list of Person objects:

import example

person1 = example.Person("Alice", 30)
person2 = example.Person("Bob", 40)

example.display_people([person1, person2])

Important Points to Consider

  1. Constructor Binding: The Person class constructor must be exposed to Python for pybind11 to construct the object properly.
  2. Error Handling: Just like with built-in types, pybind11 will throw a TypeError if the Python list contains elements of the wrong type.

Conversion in the Reverse Direction: std::vector to Python List

Why You Might Need Reverse Conversion

While it’s important to know how to convert Python lists to C++ vectors, there are also cases where you’ll want to do the opposite: convert an std::vector to a Python list and return it.

Automatic Conversion by pybind11

Good news! pybind11 handles this conversion automatically. As long as the appropriate pybind11 headers are included, you can return an std::vector from a C++ function, and pybind11 will automatically convert it into a Python list.

Example 3: Returning a Vector of Integers as a Python List

Here is a simple example:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <vector>

namespace py = pybind11;

std::vector<int> get_sequence(int n) {
    std::vector<int> result(n);
    for (int i = 0; i < n; ++i) {
        result[i] = i;
    }
    return result;
}

PYBIND11_MODULE(example, m) {
    m.def("get_sequence", &get_sequence);
}

When you call this C++ function from Python, you’ll get a Python list:

import example

result = example.get_sequence(5)
print(result)  # Output: [0, 1, 2, 3, 4]

Important Notes

  1. Headers: Again, don’t forget to include <pybind11/stl.h>.
  2. Type Restrictions: The elements in the std::vector should be of a type that pybind11 knows how to convert. This includes basic types like int and float, as well as any custom types that you’ve exposed to Python using pybind11.

Handling Nested Containers: std::vector of std::vector

The Challenge of Nested Containers

Working with a single layer of containers is straightforward, but what about nested containers like a vector of vectors (std::vector<std::vector<T>>)? pybind11 can handle these cases, too!

Example 4: Converting a 2D Python List to std::vector of std::vector<int>

Let’s extend our previous examples to deal with a 2D list, corresponding to a std::vector<std::vector<int>>.

Here’s the C++ function:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <vector>

namespace py = pybind11;

int sum_of_2d_elements(const std::vector<std::vector<int>> &vec) {
    int sum = 0;
    for (const auto &subvec : vec) {
        for (const auto &elem : subvec) {
            sum += elem;
        }
    }
    return sum;
}

PYBIND11_MODULE(example, m) {
    m.def("sum_of_2d_elements", &sum_of_2d_elements);
}

You can use it in Python like this:

import example

result = example.sum_of_2d_elements([[1, 2], [3, 4], [5, 6]])
print(result)  # Output: 21

Important Points

  1. Type Matching: The type in the nested std::vector must match the type in the Python sub-lists. Type errors will be raised if there is a mismatch.
  2. Depth: pybind11 can handle containers with arbitrary nesting depth, but be cautious about the complexity this adds to your code. Debugging deeply nested structures can become challenging.

Error Handling and Best Practices

Being Cautious with Type Safety

pybind11 performs type checking, which is a valuable feature for catching errors early. But you should also be aware of the kinds of errors that can occur and how they manifest.

Example 5: Handling Type Errors

If a function expects a std::vector<int> and you pass a Python list that contains a non-integer, pybind11 will throw a TypeError.

int sum_of_elements(std::vector<int> const &vec) {
    // ...
}

PYBIND11_MODULE(example, m) {
    m.def("sum_of_elements", &sum_of_elements);
}

In Python:

import example

try:
    result = example.sum_of_elements([1, "two", 3])  # Raises TypeError
except TypeError as e:
    print(f"An error occurred: {e}")

Best Practices

  1. Explicit Type Documentation: Always document the expected types for your functions, both in C++ and in Python. This will make it easier for users of your library to understand the kind of data they should be working with.
  2. Unit Testing: Extensive unit tests can catch type errors and edge cases. Consider using Python’s unittest framework or C++ testing frameworks like Google Test for this purpose.
  3. Handling Optional Arguments: If your C++ function can handle an empty std::vector, make sure to specify a default value in your binding. For example: 
    m.def("sum_of_elements", &sum_of_elements, py::arg("vec") = std::vector());
  4. Custom Type Conversion: For more complicated types or specialized conversion logic, consider writing a custom type caster by specializing the py::cast template. This is an advanced topic and should be used cautiously.

Conclusion and Further Reading

Wrapping It Up

We’ve explored how pybind11 allows for smooth conversion between Python lists and C++ std::vector objects, covering both basic and advanced scenarios. We’ve also delved into nested containers and best practices for type safety and error handling.

By leveraging pybind11’s capabilities in this regard, you can develop more robust, flexible, and user-friendly C++ libraries that integrate seamlessly with Python.

Further Reading

If you wish to dig deeper into pybind11 and its features, here are some resources that can help:

  1. Official pybind11 Documentation: A comprehensive guide to all the features pybind11 offers.
  2. pybind11 GitHub Repository: For the latest updates and to contribute to the project.
  3. C++ and Python: Best Practices: For an overview of best practices when interfacing C++ with Python.
  4. Modern C++ Programming with Test-Driven Development: For good practices in C++ development, including unit testing.

Final Thoughts

The ability to convert Python lists to std::vector and vice versa is just one of the many features that make pybind11 a powerful tool for C++ and Python interoperability. Whether you are a library developer looking to expose your C++ code to Python, or a Python programmer in need of the speed and features of C++, understanding this conversion can greatly enhance your workflow.

Related Posts: