You’ve probably used range() or enumerate() a hundred times, but you probably don’t know what goes on “under the hood” with these functions.
Behind the scenes, these functions don’t build giant lists and make you iterate over them. Instead, they produce the values one at a time as you iterate over it.
This comes with 3 major benefits:
Memory efficiency: Generators don’t build entire data structures in memory. Instead, they give one item at a time (yield). In scenarios where you’re streaming logs or working with large CSV’s, storing all that data in memory would be wasteful or impossible.
Performance and Laziness: Since generators create values only when needed, it means that there’s an upfront computation cost that’s avoided.
Pre-iteration overhead: When iterating over a list/tuple, that data structure must be allocated in memory. Working with generators avoids this overhead, tying into point 1.
A generator’s design pattern is called lazy evaluation, and Python gives you the tools to build your own through generators.
Creating your own generator
Generators are simply functions that leverage the keyword yield instead of return. This keyword allows the code to “pause” and then pick right back up where it left off.
For instance, if you have a function that produces a list and you’re checking each value independently:
def check_values(value_to_check):
for val in [0, 1, 2, 3]:
if val == value_to_check:
return valThis snippet of code works, but what if we have 100 trillion values? Chances are, the computer won’t be able to store this list in memory, so using a generator will come in handy to avoid any memory issues:
# Generate 1 billion values without ever
# storing a list of 1 billion values
def generate_values(max_val = 500):
n_values_generated = 0
while True:
# return n_values_generated, but come back here
# when your logic is finished
yield n_values_generated
n_values_generated += 1
# If we reach our maximum, then we'll
# exit the loop and no longer generate
# values.
if n_values_generated == max_val:
break
def check_values(value_to_check):
# Generate 1 billion values
for val in generate_values(max_val = 1e11):
if val == value_to_check:
return valIf you’re working with really large files, you may use a generator to read a file line-by-line:
def read_large_file(fname):
with open(fname) as f:
for line in f:
yield line.strip()The drawbacks of generators
While generators are extremely useful, there’s 2 major drawbacks with using generators:
One iteration: Generators can only be iterated over once. After the value has been generated, it gets discarded.
Debugging: Sometimes, debugging generators can be tricky since all of the values aren’t stored in memory.
Final Thoughts
Overall, generators are a fantastic tool. It allows you to trade memory for speed, scale your code to handle massive data, and write cleaner iteration logic. It does come with drawbacks, but if there’s one “tool” to add to your development toolkit, it would be understanding lazy evaluation and knowing how to implement it using generators.
Happy coding!
📧 Join the Python Snacks Newsletter! 🐍
Want even more Python-related content that’s useful? Here’s 3 reasons why you should subscribe the Python Snacks newsletter:
Get Ahead in Python with bite-sized Python tips and tricks delivered straight to your inbox, like the one above.
Exclusive Subscriber Perks: Receive a curated selection of up to 6 high-impact Python resources, tips, and exclusive insights with each email.
Get Smarter with Python in under 5 minutes. Your next Python breakthrough could just an email away.
You can unsubscribe at any time.
Interested in starting a newsletter or a blog?
Do you have a wealth of knowledge and insights to share with the world? Starting your own newsletter or blog is an excellent way to establish yourself as an authority in your field, connect with a like-minded community, and open up new opportunities.
If TikTok, Twitter, Facebook, or other social media platforms were to get banned, you’d lose all your followers. This is why you should start a newsletter: you own your audience.
This article may contain affiliate links. Affiliate links come at no cost to you and support the costs of this blog. Should you purchase a product/service from an affiliate link, it will come at no additional cost to you.
