Why I Switched From Pandas to Polars | TDE Workshop
9,309
Published 2023-08-29
Led by: Ben Feifke
More Resources
TDS Article: medium.com/towards-data-science/the-3-reasons-why-…
Example Code: github.com/bfeif/personal-website/blob/main/code/n…
Udemy Course: www.udemy.com/course/data-analysis-with-polars/
Podcast Episode: www.inspiringcomputing.com/2107763/12718169
Connect with Ben
LinkedIn: www.linkedin.com/in/benjamin-feifke/
Medium: medium.com/@benfeifke
Twitter/X: twitter.com/benfeifke
Homepage: benfeifke.com/
Keep up with The Data Entrepreneurs!
🎥 YouTube: / @thedataentrepreneurs
👉 Discord: discord.gg/RSqZbF9ygh
📰 Medium: medium.com/the-data-entrepreneurs
📅 Events: lu.ma/tde
🗞️ Newsletter: the-data-entrepreneurs.ck.page/profile
Overview - 0:00
About Ben - 0:29
What's Polars - 3:19
Reason 1: .list namespace - 5:30
Reason 2: .scan_parquet() and .sink_parquet() - 11:48
Reason 3: Data-oriented Programming - 16:16
Demo - 20:58
Some Questions - 34:52
What’s Next? - 36:48
Q&A - 39:00
All Comments (7)
-
Great Job Ben! 👍
-
Although I see the benefits of Polars. I haven't had enough obstacle with Pandas for my workflows. I don't deal datasets that exceed memory and I think currently I can extend my memory limit using Dask ... but looking forward to the development of Polars and will likely adopt once it has more support!
-
It is a shame that the lazy API is so entangled into the API. Might be nice to write generic code which then has the option to switch on the lazy API with one single change. I don't like the idea of having to rewrite the whole codebase to switch between lazy and eager. I question if that was a good design decision from polars. - Side note please always upload videos in 1080p
-
Can Polars replace pyspark Or hadoop?
-
man, we can read. why read everything on the screen?
-
Pandas is a massive mess. It's very easy to write very poor code in Pandas but to write it well is not that intuitive, Matt Harrison has written some great stuff, but it's not that easy to learn. Polars is way better and improving all the time. It's much easier to write and way quicker. The lazy API is a thing of beauty.
