In this post, we will go over how to create thread-safe generators in Python. This post is heavily based on this excellent article.
Generator makes life so much easier when coding in Python, but there is a catch; raw generators are not thread-safe. Consider the example below:
We see that the generator does not produce correct output when multiple threads are accessing this at the same time.
One easy way to make it thread-safe is by creating a wrapper class that simply lets only one thread to execute the generator's next method at any given time with threading lock. This is shown below:
Note that the generator now is thread-safe but doesn't execute its next method in parallel. You can also use Python's decorator to make it look even easier, although it basically does the same thing.
This code runs in Python2, but not in Python3.
ReplyDeleteIn Python3, you get this error: TypeError: 'thread_safe_generator' object is not an iterator
I fixed it. Just replace thread_safe_generator.next with thread_safe_generator.__next__. Python3 needs the double underscore.
Delete