Split Large Parquet Files Automatically with Polars
Plus replace Docker with one Coiled decorator
Grab your coffee. Here are this week’s highlights.
📅 Today’s Picks
Split Large Parquet Files Automatically with Polars
Problem
When writing large datasets to Parquet, you end up with either one massive file that is slow to read or must manually split data into smaller files.
Solution
With Polars PartitionMaxSize, output is automatically broken into multiple Parquet files according to a defined size limit.
This enables:
Parallel reads across multiple cores
Faster, more reliable cloud storage transfers
📖 View the full article | 🧪 Run code | ⭐ View GitHub
Coiled: One Decorator Replaces Your Entire Docker Workflow (Sponsored)
Problem
Have you ever had code work locally but fail on cloud VMs because of missing dependencies or version mismatches?
Docker solves this by freezing dependencies, but introduces friction: Dockerfiles, slow builds, registry pushes, and full redeploys for minor package changes.
Solution
Coiled can remove Docker from the workflow entirely. With a single decorator, it automatically syncs your local environment to the cloud.
Key features:
Exact dependency replication from local to cloud
No need for container builds or registry management
Compatible with pandas, Polars, DuckDB, Dask, and more
Faster deployments through smart caching
☕️ Weekly Finds
crewAI [LLM] - Framework for orchestrating role-playing autonomous AI agents that work together to accomplish complex tasks
Ray [MLOps] - Unified framework for scaling AI and Python applications from laptop to cluster with distributed runtime and ML libraries
Metabase [Data Viz] - Open-source business intelligence tool that lets everyone visualize, analyze, and share data insights
Before You Go
🔍 Explore More on CodeCut
Tool Selector - Discover 70+ Python tools for AI and data science
Production Ready Data Science - A practical book for taking projects from prototype to production
⭐ Rate Your Experience
How would you rate your newsletter experience? Share your feedback →