Observe how data moves between executors during a shuffle, and what skew looks like.
Understanding the shuffle
Uniform: Each reduce partition gets ~equal work. All tasks finish at similar times.
Skewed: One reduce partition gets 60% of rows. That one task determines the stage duration.
Very skewed: One reduce partition dominates. 90% of the stage time is spent waiting for one task.
Salted: The hot key is appended with a random salt (0–3), distributing it across multiple reduce partitions. Reduces max partition size at the cost of a post-aggregation merge step.
The stragglers — the slowest tasks — determine total stage duration. Balancing data across partitions is the solution.