Update Python 3.12: Is it 2 times faster? Key changes

Hi! My name is Oleksii. I am a first-year master's student at the Kyiv Polytechnic Institute, and I have been programming in Python for about five years. This is already the third article in the #mainfeatures series, highlighting the new features that come with the latest Python versions.

Let's talk about the changes, as Python firmly maintains its leading position in the TIOBE Index for January 2024 & PYPL Popularity of Programming Language, surpassing Java, JS, and C/C++. What's the secret?

I've divided the article into two parts: practical and theoretical.

The first one (which you are currently reading) is the main one. Here, you will learn about the key changes introduced by the Python 3.12 release. We'll explore examples of using these changes, the advantages they bring, and whether it's worth transitioning to the new version as quickly as possible.

In the second part, we'll focus more on the performance comparison of the latest versions: from installing Python versions over the last 5 years (versions 3.8-3.12) using pyenv to benchmarking speed measurements and comparing these results (link to the second part you'll find at the end of this article).

Changes in Python 3.12

Release date: October 2023

New syntax for type parameters — type

Let's recall that in Python 3.5 (PEP 484), the concept of type hints for variables was introduced. The development of this idea continued with PEP 612, which introduced parameter specifications, and PEP 646, which extended this concept to variable annotations for variational types.

While generic types and type parameters have become popular, the syntax for defining type parameters still looks somewhat "attached" to Python. This has led to confusion among Python developers.

Let's consider an example with a function responsible for sorting (using the QuickSort algorithm) a given list consisting of elements of possible types: int, float, or str:

Which is analogous for

So, here we declare a function that accepts as input a list of any elements. The result can also be returned as a list of any elements.

Let's provide an example of usage:

So far in the implementation, we haven't shown possible types like int, float, and str. Someone might think that it's sufficient to enumerate them in a Union set, like this:

Or, as I mentioned in the article about the new features in Python 3.10, using the new Union operator `|`:

But no, in this case, we allow passing lists with different types as input:

This will result in a type checker error:

We aim to demonstrate that the type of elements in the output list matches the type of elements passed to the input list parameter of the function. And for comparison, inside the algorithm, we will also use elements of the same type. To do this, we need to use a parameterized parameter, TypeVar.

It might seem great, definitely better than before, but here's the trick. In the group of developers using static typing in Python, there is a consensus that it's time to introduce a formal syntax that would better align with modern programming languages supporting generalized types.

An analysis of 25 popular typed Python libraries showed that typical variables, such as symbol typing.TypeVar, are used in 14% of the modules.

In Python 3.12, we got a new feature to introduce a similar function — only without declaring TypeVar:

*Warning: If you encounter a similar error:

Make sure you are running your code with Python 3.12 😉. At the time of testing, mypy doesn't yet support this syntax and shows the error:

However, after adding support for this syntax in all static type checkers, you'll be able to enjoy all the benefits of this concise notation. For the complete task, we still need to add bounds to T for types like int, float, and str:

You can repeat this for the new syntax using:

Now you will no longer need to import and manually initialize TypeVar. Additionally, you will be able to use this application method with multiple types as well as classes. Let's consider an example of using it with classes:

However, I haven't yet mentioned the new syntax for type parameters — type. Now we can create type aliases. The declaration is done using the type instruction:

This is particularly beneficial when describing nested structures with multiple possible variants using the union operator | or Union. It is advantageous to avoid the repetitive use of such verbose expressions in the codebase when there is an opportunity to use type aliases.

Syntactic formalization of f-strings

Python 3.12 introduces fantastic improvements to f-strings, providing greater flexibility and expressiveness in string formatting. These enhancements are formalized in PEP 701, which brings syntactic changes to f-strings, removing previous limitations and making f-strings even more powerful.

In Python 3.11, reusing the same curly braces within an embedded f-string caused a syntactic error, restricting the possibilities of nesting.

Example:

To overcome this problem, it was necessary to use different quotes, for example:

After all, the way to mark each nested quote character with a "\" character (as shown with Tolkien\'s) didn't work:

Because f-strings couldn't contain backslashes at all, such as the "\n" character for a newline or Unicode characters.

In Python 3.12 using backslashes to denote inner quotes and beyond is not allowed, but there is no longer a need for this. Restrictions on Unicode characters and using the same quotes that denote f-strings have been lifted. This allows for a more expressive use of f-strings.

Additionally, now f-strings can be nested arbitrarily, providing greater flexibility:

Also, the ability for multiline expressions and comments has been introduced.

Python 3.12 allows defining f-strings across multiple lines and includes inline comments to enhance readability:

These changes enable the creation of more expressive and versatile f-string expressions.

Along with this enhancement, we've received more precise error messages. In Python 3.12, error messages for f-strings are more accurate, thanks to the PEG parser. Unlike in Python 3.11:

Messages now include the exact location of the error in the string:

Furthermore, thanks to the changes in PEP 701, token generation using the tokenize module is accelerated by 64%.

GIL for each interpreter: a path to parallelism?

The global interpreter lock (GIL) in Python has traditionally limited the concurrent execution of threads, making it challenging to work with parallel code. PEP 684 introduces radical changes by allowing each sub-interpreter to have its own GIL. This enables better utilization of multi-core processors.

To create a sub-interpreter with its own GIL, developers can use the Py_NewInterpreterFromConfig() function, as shown in the example. The ability to create interpreters with separate GILs is currently available through the C-API, and starting from Python 3.13, it is expected to be available in the Python API as well.

With a separate GIL for each interpreter, Python developers gain flexibility in exploring new models of parallelism. Under the hood, there has been a refactoring of internal components in CPython. The initiative involved moving a significant portion of the global state store to a per-interpreter store. Despite the absence of significant changes, Python 3.12 does not alter existing approaches but serves as groundwork for significant improvements, laying the foundation for the future development of parallelism.

Improvements in NameError, ImportError, and SyntaxError messages

With each new version of Python, core developers improve the error output and error-finding process, making it easier to locate issues in your programs. Python 3.10 introduced more informative and precise error messages, particularly for syntax errors. Python 3.11 also expanded tracing capabilities, making it easier to find problematic code.

Python 3.12, the latest release, improves the developer experience by providing more informative error messages. Common error messages now come with helpful hints. The new and enhanced messages focus on the module import process.

A significant portion of these improvements relates to the module import process. Let's consider scenarios where we work with environment variables, fetching them from the os module, importing it before that.

3.11:

3.12:

During an attempt to use os without prior import, the traditional NameError occurs. However, in Python 3.12, a helpful reminder has been added, advising you to import the os module before attempting to access it.

Another improvement in error message clarity also relates to imports. Now, if you accidentally change the order of keywords in the from-import instruction:

Python 3.11:

Python 3.12 will gently nudge you towards the correct syntax:

In this example, attempting to import getenv from the os module results in a syntax error. I'm not sure if this is widespread among experienced developers, but Python 3.12 detects it and suggests using the correct syntax with from ... import ....

The third new improvement in the error message relates to importing specific names from the module.

Python 3.11:

Python 3.12:

In this case, attempting to import get_env from os results in an import error. Python 3.12 assumes that you probably intended to import getenv instead of get_env. This function, introduced in Python 3.10, now extends to import statements in Python 3.12.

In addition to import-related improvements, Python 3.12 introduces the latest enhancement regarding methods defined inside classes.

Consider, for example, the implementation of the Person class:

Python 3.11:

Python 3.12:

Instead of the usual NameError, Python 3.12 recognizes that name is an attribute accessible on self and suggests using the instance attribute self.name instead of the local name name.

Buffer protocol

Python 3.12 introduces PEP 688, making the buffer protocol directly available from Python code. This enhancement allows classes implementing the __buffer__() method to use buffer types. Additionally, an abstract class collections.abc.Buffer (ABC) has been added to standardize the representation of buffer objects, facilitating their use in type annotations.

PEP 688 is motivated by the desire to bridge the gap between the Python buffer protocol and C.

The buffer protocol, initially available only for C code, now has a Python-level API. This development enables static typing tools to assess whether objects implement the protocol, removing current limitations.

Currently, there is no mechanism in Python code to check whether an object supports the buffer protocol. Static type checkers also lack a specific type annotation to represent the protocol. This becomes problematic when writing type annotations for code that handles generic buffers. There are two approaches: using type aliases for known buffer types and using typing.ByteString, but they are considered inadequate for various reasons.

The C buffer protocol supports various capabilities related to strides, contiguity, and write support in the buffer. While typeshed provides type aliases for buffers available for both reading and writing, many of these options cannot be directly queried on a buffer protocol C-type object.

A practical example using collections.abc.Buffer is provided in the documentation:

It can also be used with isinstance and issubclass checks:

To demonstrate the use of the buffer protocol, consider the following Python class:

Productivity improvements for isinstance and asyncio

Among the minor enhancements, a notable performance boost has been achieved, which is discussed in more detail in the second article.

In Python 3.12, the isinstance() function for runtime protocol checks has significant improvements when checking protocols. Most isinstance() protocol checks should be at least twice as fast as in version 3.11, with some being up to 20 times faster or more. However, checking isinstance() protocols with fourteen or more members may, on the contrary, be slower than in Python 3.11.

Additionally, the asyncio package in Python 3.12 has undergone significant performance improvements, resulting in a substantial speed increase of 75%, which we will try to verify in the second part of this article by running various benchmarks and comparing the performance changes.

TypedDict for more precise **kwargs annotation

The first change related to annotation is discussed in PEP 692. Now we can describe **kwargs in more detail. **kwargs is a special syntax that allows passing a variable-length dictionary of keyword arguments to a function. It is short for keyword-arguments.

Previously, when passing **kwargs as a dictionary with keys of type str and any values, we could at most use the construction **kwargs: Any, or if lucky, **kwargs: dict[str, Any]. However, now we can declare a TypedDict and explicitly specify what is expected in the **kwargs using typing.Unpack.

Example:

This feature is already supported by type checkers, but when running on Python version 3.11, we would encounter the following error:

@override for explicitly override methods

The second new feature for type annotations in Python 3.12 is the @override decorator — a compelling way of type-checking explicitly designed to mark methods in subclasses that override their counterparts in the parent class. It's easy to guess that this enhancement is similar to mechanisms in languages like Java and C++.

Despite promoting safe coding and preventing errors during code refactoring, it's essential to understand that type annotations don't require using this decorator in every method of a subclass. So, whether this approach will become widespread among developers is uncertain. However, I'm sure that this decorator, like @abstractmethod for the parent class, strengthens the relationship between classes and helps developers create more reliable and error-resistant code through static type analysis.

Example:

Summary

In this latest update, we've gained:

1. A new way to specify types using the type parameter.
2. Clear syntax rules for f-strings.
3. GIL (global interpreter lock) for each interpreter.
4. Enhanced error messages for NameError, ImportError, and SyntaxError.
5. Introduction of the Buffer protocol.
6. Improved performance for isinstance and asyncio.
7. Usage TypedDict to accurate **kwargs annotation.
8. Introducing @override for explicitly overriding methods.

Some modules in the new version are already deprecated. And those that were deprecated in previous versions could have been cleaned up or even completely removed. If you are planning to update and are using any of them, pay attention to the list of modules that have changed: asynchat, asyncore, configparser, distutils, ensurepip, enum, ftplib, gzip, hashlib, importlib, imp, io, locale, smtpd, sqlite3, ssl, unittest, webbrowser, xml.etree.ElementTree, zipimport and others.

I cannot confidently say that you must update your projects to 3.12. The changes have significantly impacted some aspects of development, touching upon improvements in error detection and adding the ability for each interpreter to work with the GIL separately. This is important as it lays the foundation for parallelism in new versions.

Guido van Rossum (the author of the Python language) promised that Python 3.11 would be twice as fast. Did he manage to keep his promise? Look for the answer to this question in the second part of the article. There, we will concurrently install the latest versions of Python and demonstrate how to do it using pyenv. We will also run pyperformance and engage in a performance comparison of the latest versions of Python.

If you missed the previous releases or wish to revisit the details, here are links to a comprehensive article covering changes from Python 3.8 to 3.10 and another article highlighting Python 3.11 changes:

Thanks for reading. Glory to Ukraine 🇺🇦

Sources:

What's New In Python 3.12