This post was written by David Ziganto, Metis Sr. Data Scientist based in Chicago. It was originally posted on his blog here.
There is a plethora of information about how to speed up Python code. Some strategies revolve around leveraging libraries like Cython, whereas others propose a “do this, not that” coding approach. An example of the latter is using vectorized implementations instead of loops. While many “do this, not that” strategies exist, I decided to focus on just a few, which I discovered via the links featured at the bottom of this post.
My initial intent was just to test the assertions. However, as I collected experimental data on the various methods, I became curious about another claim I often hear: Python 3 is faster than Python 2. So I plied the exact same experimental methods in Python 2.7 (Py27) and Python 3.5 (Py35), affecting the necessary changes like using xrange for Py27 and range for Py35. All of my experiments, as well as timings and corresponding plots, can be found here: Py27 Notebook and Py35 Notebook.
The plots contained in this article depict the results of my experiments for Python 3 and are labeled as such.
For detailed information including methodology and quantitative values for timings as well as details about Python 2, please review the notebooks linked above.
This post is split into two parts. In Part 1, I will compare two approaches commensurate with the “do this, not that” logic to see if there is a substantial difference and, if so, which approach is better. In Part 2, I will compare Python 2 and Python 3 to see if there is credence to the claim that Python 3 is indeed faster.
Part 1: Comparing Methods
In this part, I will describe and compare approaches for the following:
- - Looping Over A Collection
- - Looping Over A Collection & Indices
- - Looping Over Two Collections
- - Sorting Lists In Reverse Order
- - Appending Strings
- - Standard Library
- - ListExp vs GenExp
- - Dots
Looping Over a Collection
Looping over a collection is like this: say you have a list of animals [‘aardvark’, ‘bee’, ‘cat’, ‘dog’] and you want to access each item one at a time. How can you do this? One way is to access the item by index. For example, you could use a listcomp like this:
Of course, this example is silly because we are generating a list from a list. However, the point is that we are iterating and doing something with the iterables. The thing we actually do can be anything.
Another way to approach the same problem is:
It is definitely easier to read and understand what is happening. But is it faster? Turns out that it is. The second approach cuts the processing time nearly in half. Below is a plot depicting the difference.
Please note that this plot and all subsequent plots are log-log plots.
Looping Over Two Collections
We know what looping over a collection looks like. What is this indices business that we’re now incorporating? Well, instead of just returning the list item, we also want to know its index value. The two approaches that were tested are as follows:
I suspect most everyone would agree that the second approach is more Pythonic. But is it faster? Here we see our first discrepancy between Py27 and Py35. The first approach is faster for Py27, while the second approach is faster for Py35. The reason for this peculiar behavior has to do with the zip function. In Py35, zip() is an iterator whereas it is not in Py27. One way to combat this in Py27 is by using izip() in the itertools library. As an aside, if you are not familiar with itertools, it is definitely worth looking up. Essentially it is a library of iterables that can dramatically speed up legacy code.
Sorting Lists in Reverse Order
This one is straightforward. Say I have my list of animals but I want to reverse the order so it looks like [‘dog’, ‘cat’, ‘bee’, ‘aardvark’]. Here are the two approaches:
The second approach is once again much more elegant. But is it faster? Yes, it is, and by quite a bit and for both versions of Python.
Say I want to loop through and append my strings like I would a list. If you are familiar with Python strings, you know that a new string is created, not actually appended to the original string. Good to know if you did not already. So let’s look at the two approaches:
The first is probably easier to wrap your head around unless you have already worked with the esoteric looking string.join() function before. So which is faster? It appears the first approach may be faster. However, I am calling this result inconclusive. Why? I will skip the implementation details, but the short version is that I was only able to time a single run for each approach using %time instead of the multi-run %timeit. Therefore, the results are questionable. I plan to circle back in the future to conduct a more thorough test.
Python’s standard library is replete with built-in functions. The claim here is that these built-in functions are highly optimized and, therefore, significantly faster than a pure programming approach. In case that is not clear, let us look at a concrete example. Say I want to generate a list that stores the cumulative sum of consecutive integers. So if my set of integers is [0, 1, 2, 3], my generated list would be [0, 1, 3, 6]. Here are the two approaches that were tested.
Which is faster? In this case, the second approach for both versions of Python is faster. An interesting result manifested, though. Here we see another discrepancy between Py27 and Py35. Py27 showed only slight improvement in the second approach, even as it scaled. However, Py35 not only ran much faster with approach two (nanoseconds instead of milliseconds or seconds) but scaled in constant time. That is a massive difference! There may be some credence to the claim that Python 3 is faster than Python 2 after all.
ListEXP vs. GenEXP
I will assume you are familiar with list expressions (ListExp) at this point. Generator expressions (GenExp) may or may not be new to you. For an in-depth description, see this doc. One key quote from that doc is:
Generator expressions are a high performance, memory efficient generalization of list comprehensions.
The doc also goes on to say:
As data volumes grow larger, generator expressions tend to perform better [than list expressions] because they do not exhaust cache memory and they allow Python to reuse objects between iterations.
Let’s now test those assertions by leveraging the cumulative sum example from the last section.
Generator expressions are certainly easy to use as they look almost identical to list expressions. So do they perform as touted? In both versions, the answer is unequivocally...YES! List expressions scaled with the data whereas generator expressions scaled in constant time for both versions. An interesting result is that Py27 generator expressions were slightly faster than Py35. I am unclear as to why that is the case – worth exploring in the future.
The final strategy that I tested had to do with this thing called dots. Dots is just another way to say methods. Lacking a clear way to state this strategy in words, I will instead get right to the approaches, which should be self-explanatory.
Was there a difference? Both Py27 and Py35 exhibited a marginal difference skewed towards the second approach as being more performant. The evidence seems to indicate that there may be a minute performance gain, but it hardly seems worth the effort, given some of the dramatic results we saw in other categories.
Part 2: Python 2 vs. Python 3
In the introduction, I presented a claim I often hear. Specifically, Python 3 is faster than Python 2.
First off, let me just say that while I gathered some empirical evidence by testing a few assertions, this small sampling of experiments is clearly insufficient to formally declare either way. That said, there was evidence that some of the inner workings of Python 3 are far more optimized and, therefore, Python 3 is faster in some regards. Take for example the standard library example using map(). We saw an enormous difference when comparing Py27 and Py35, where Py35 scaled in constant time and Py27 did not. In the Looping Over Two Collections test, Py35 also performed better because zip() is an iterator in Py35.
There are many ways to speed up Python code. Blazing fast libraries like Cython or Numba exist. However, I decided to focus on coding techniques in this article. As such, I identified a few techniques that looked promising and ran experiments to determine if the assertion that they lead to speedier code is in fact true. As an add-on, I wanted to explore the often asserted claim that Python 3 is faster than Python 2. Without further ado, here are the results in classic bullet point format:
- - Avoid using a list’s index to access its items when looping over a collection
- - Using enumerate() is more beautiful but not faster for looping over a collection and indices
- - Mind your Python version when looping over two collections - use itertools for Python 2
- - Use built-in functions whenever possible
- - How you append strings may matter at scale but my experiment was inconclusive
- - Forget list expressions at scale; rock generator expressions instead
- - Dots may give you marginal performance gains but may not be worth the effort
- - When Python 3 is better than Python 2, it is MUCH better
As a long time Python 2 user, this experiment got me much more excited about Python 3. Yes, I know Python 3 is the future, but now I have something tangible to grasp onto – proof that aspects are indeed speedier and can lead to faster code. If nothing else, having conducted my own experiments has provided me the catalyst to finally make the transition.
In sum, this was a fun project that got me thinking. There are so many avenues left unexplored and this project raised many more questions than it answered for me, yet now I feel much more confident about the prowess of Python 3.
I hope you found this enlightening, entertaining, or both. If you have questions, comments, or feedback, please let me know.
4 Performance Optimization Tips For Faster Python Code
PythonSpeed Performance Tips
6 Python Performance Tips