Fletcher
50-Minute Talk
Psycopg is the de facto standard PostgreSQL adapter for Python (or, depending on whom you ask, the Python adapter for PostgreSQL).
From its inception, Psycopg has been built on top of libpq, PostgreSQL's official C client library. This design choice has brought clear benefits: libpq handles the complexity of the wire protocol, query state management, and many low-level details, sparing users an entire class of bugs while delivering solid performance in a language not historically known for its raw speed.
At the same time, relying on an external C library comes with trade-offs. Build and deployment become more complex; choices must be made between tight isolation and system integration; and subtle dependency issues can arise. For these reasons, many alternative PostgreSQL drivers in the Python ecosystem have opted to bypass libpq entirely and work directly at the protocol level.
Even when libpq is used, its abstractions do not always align perfectly with the needs of higher-level clients. Psycopg 3 is likely among the most demanding libpq consumers, and over the years it has occasionally had to reimplement substantial pieces of functionality that one might reasonably expect libpq itself to provide.
We will discuss a few concrete examples:
-
A success story: pipeline mode, thanks to Simon's work—one of his many enduring gifts to the PostgreSQL ecosystem.
-
A less happy experience: asynchronous connections, and the surprising amount of functionality that must be rewritten once timeouts enter the picture.
-
A future challenge: logical replication support, which may require “drilling a hole” through libpq to access the underlying socket.
This talk reflects on two decades of largely successful collaboration between Psycopg and libpq. By highlighting the pain points we have encountered, we hope to inform future libpq evolution, set realistic expectations for client authors, and clarify when reimplementing functionality is not a failure—but simply an acknowledgment that libpq, powerful as it is, cannot solve every problem.
