Last updated on 2026-07-10T21:48:45.840Z

Why Your Spark Job Gets Stuck at 199/200 Tasks (and How to Fix Data Skew)

Have you ever waited forever for a Spark stage that shows 199/200 tasks completed while a single last task grinds on? If yes, you have likely met data skew. This guide explains in plain language what is happening, how to confirm it, and the most effective ways to fix it. You will get actionable code snippets for both PySpark and Spark SQL, clear diagnosis steps using the Spark UI, and a checklist to keep skew at bay in future pipelines.

Primary keyword to remember: spark job stuck at 199 of 200 tasks fix. By the end of this guide, you will know why that last task drags and exactly what to change.

TL;DR

If your Spark stage shows 199/200 tasks completed and the last one runs far longer, you almost certainly have data skew—one partition got far more rows than others.
Quick fixes: enable Adaptive Query Execution (AQE), broadcast the small side of a join, or repartition by a better key. For extreme skew, use the salting technique.
Diagnose in Spark UI: look for one task with massive shuffle read size, long executor run time, high GC time, or repeated fetch retries.
Prefer map-side aggregations (reduceByKey/agg) over groupByKey, and use DataFrame/SQL over Python UDFs for heavy transformations.

What “199/200 Tasks” Really Means

The Default 200 Shuffle Partitions

When Spark performs wide transformations (like groupBy, join, distinct, or orderBy), it must shuffle data across the cluster. Spark SQL defaults to spark.sql.shuffle.partitions = 200, which means the shuffle output is cut into 200 partitions, creating 200 downstream tasks in the next stage. If your UI shows 199/200 tasks completed, it’s telling you that 199 partitions finished quickly, but one partition (one task) is taking way longer—often due to many more records landing in that partition than in the others.

Why the Last Task Drags

Data is partitioned by key. If a few keys are extremely frequent (for example, a “country = US” or “customerId = 12345” that dominates the dataset), the partition that holds those keys becomes gigantic. All the work for those records sits in that single partition and makes one task do far more work than all the others. That asymmetry is called data skew.

Common Symptoms and Where to Look

Spark UI > Stages: One stage shows 199/200 tasks complete; the last task’s duration is orders of magnitude longer.
Task metrics: The slow task has huge Shuffle Read or Input Size, high Executor Run Time, and possibly high GC Time.
Executors tab: One executor is extremely hot (CPU/memory), while others are mostly idle.
Logs: You may see shuffle fetch retries, spilled-to-disk messages, or out-of-memory errors on the dragging task.

Root Causes: Why Spark Jobs Get Stuck at 199/200 Tasks

1) Data Skew

Skew happens when the distribution of join keys or groupBy keys is imbalanced. A few keys have significantly more records than the rest, causing one or a handful of partitions to dominate the workload. This is the most common reason for a spark job stuck at 199 of 200 tasks fix scenario.

2) Wide Transformations and Heavy Shuffles

Operations that force a global rearrangement of data (joins, groupBy/agg, distinct, orderBy, window functions with wide frames) can magnify skew. Even light skew prior to a shuffle can become catastrophic after a join.

3) Inefficient APIs or UDFs

groupByKey on large datasets forces massive shuffles. Prefer reduceByKey or agg with combiners.
Python UDFs are slower than native SQL/DataFrame functions. If you must use UDFs, consider vectorized Pandas UDFs for better performance.

4) Resource Limits or Missing Speculation

Sometimes the last task is slow due to an ailing executor or uneven resource allocation. Speculative execution can help by running duplicates of slow tasks, though with data skew, speculation alone rarely solves the root cause.

How to Confirm Skew (Step-by-Step Diagnosis)

Step 1: Inspect the Stage in Spark UI

Open Spark UI > Stages and find the slow stage.
Click the stage to open Summary Metrics for Completed Tasks.
Sort by Shuffle Read Size or Duration. If one task has a dramatically higher shuffle read or duration than the rest, you have skew.
Note the task’s Partition ID. This identifies the outlier partition.

Step 2: Check the Keys Causing Skew

If you know the join or groupBy key, profile it to find heavy hitters.

# PySpark example: find top keys by frequency
from pyspark.sql import functions as F

df.groupBy("join_key").count().orderBy(F.desc("count")).show(20)

-- Spark SQL example
SELECT join_key, COUNT(*) AS cnt
FROM your_table
GROUP BY join_key
ORDER BY cnt DESC
LIMIT 20;

Look for keys with counts far above the mean/median. These are likely to land in the slow partition.

Step 3: Examine Shuffle Partitions

Check how many shuffle partitions you have and whether they make sense for your data size:

# PySpark
spark.conf.get("spark.sql.shuffle.partitions")

If your dataset is huge, 200 partitions might be too few, yielding large partitions and skew exaggeration. However, simply increasing partitions does not fix skew by itself—it spreads the load but the heaviest key can still dominate one partition. Combine partition count tuning with the fixes below.

Proven Fixes for Skew (From Fastest to Most Impactful)

Each fix includes a short explanation and a code example. Pick the least invasive method that addresses your case; you can stack multiple techniques if needed.

1) Broadcast the Smaller Side of the Join

If one side of a join is small enough to fit in executor memory, broadcast it to avoid a big shuffle. Broadcasting sends the small table to all executors and performs a map-side join, removing the skewed shuffle from the equation.

# PySpark: enable broadcasting via hint or config
small = spark.table("dim_small")
large = spark.table("fact_large")

# Hint approach
joined = large.hint("broadcast", "dim_small").join(small, "join_key", "inner")

# Or set threshold (bytes) so Spark auto-broadcasts small tables
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 50 * 1024 * 1024)  # 50 MB

-- Spark SQL: broadcast hint
SELECT /*+ BROADCAST(dim_small) */ f.*
FROM fact_large f
JOIN dim_small ON f.join_key = dim_small.join_key;

When broadcasting is possible, it is often the simplest and fastest spark job stuck at 199 of 200 tasks fix.

2) Turn On Adaptive Query Execution (AQE)

AQE lets Spark optimize shuffles and joins at runtime, based on actual statistics. It can coalesce small partitions, split skewed partitions, and pick better join strategies automatically.

# PySpark: enable AQE
spark.conf.set("spark.sql.adaptive.enabled", "true")
# Optional: also consider enabling these to improve partition sizing and skew handling
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

After enabling AQE, rerun the job and re-check the Spark UI. You should see more balanced partition sizes, and the notorious 199/200 should disappear or reduce dramatically.

3) Repartition by a Better Key (or Composite Key)

If your current partitioning key is skewed, repartition by a key with more uniform distribution. If no single column is even, consider a composite key (e.g., hash of multiple columns) to spread load.

# PySpark: repartition before the shuffle-heavy operation
from pyspark.sql import functions as F

# Example: composite evenly-distributed key
balanced = df.withColumn("part_key", F.xxhash64("colA", "colB"))
even = balanced.repartition(1000, "part_key")  # increase partitions if dataset is large

result = even.groupBy("part_key").agg(F.count("*").alias("cnt"))

-- Spark SQL: repartition by expression
WITH t AS (
  SELECT *, xxhash64(colA, colB) AS part_key FROM your_table
)
SELECT part_key, COUNT(*)
FROM t
DISTRIBUTE BY part_key;

Note: repartition(n, col) triggers a shuffle; use it strategically.

4) Salting (Key Splitting) for Hot Keys

When a few keys dominate (e.g., a single customer has 40% of records), salting spreads each hot key across multiple partitions by adding a random salt value to the join key. You add the same salt on both sides of the join (or only on the large side, then join with a cross-salt expansion on the small side if needed).

# PySpark: salting example for a skewed join on 'user_id'
from pyspark.sql import functions as F
import random

# Identify hot keys (approximate)
hot_keys = [row["user_id"] for row in df.groupBy("user_id").count().orderBy(F.desc("count")).limit(10).collect()]

salt_buckets = 20  # tune based on skew severity

# Add salt to large side
large_salted = (
  large
  .withColumn("salt", F.when(F.col("user_id").isin(hot_keys), F.rand(seed=42) * salt_buckets).cast("int").otherwise(F.lit(0)))
  .withColumn("user_id_salted", F.concat_ws("_", F.col("user_id"), F.col("salt")))
)

# Add matching salt to small side
small_expanded = (
  small
  .withColumn("salt", F.when(F.col("user_id").isin(hot_keys), F.explode(F.sequence(F.lit(0), F.lit(salt_buckets - 1)))).otherwise(F.lit(0)))
  .withColumn("user_id_salted", F.concat_ws("_", F.col("user_id"), F.col("salt")))
)

joined = large_salted.join(small_expanded, on=["user_id_salted"], how="inner")

Salting reduces per-partition load for hot keys and is a robust spark job stuck at 199 of 200 tasks fix when broadcasting isn’t possible.

5) Map-Side Pre-Aggregation (Avoid groupByKey)

Reduce the amount of data shuffled by aggregating early. For RDDs, use reduceByKey instead of groupByKey. For DataFrames, combine fields before wide aggregations and leverage built-in combiners.

# PySpark DataFrame: pre-aggregate at a finer grain
pre = df.groupBy("user_id", "date").agg(F.sum("value").alias("daily_sum"))
# Now join or further group on the aggregated data instead of raw records

# RDD example
rdd = sc.textFile("...")
pairs = rdd.map(lambda x: (x.split(",")[0], int(x.split(",")[1])))
# Good: reduceByKey combines values per key before shuffle
reduced = pairs.reduceByKey(lambda a, b: a + b)

6) Use DataFrame/SQL APIs Over Python UDFs

Python UDFs run slower and serialize/deserialize data between the JVM and Python. Prefer built-in SQL functions, which are optimized and can be pushed down, vectorized, and combined. If a UDF is necessary, try Pandas UDFs (vectorized) and keep them before major shuffles if possible.

# Prefer this
from pyspark.sql import functions as F
out = df.select(F.lower(F.col("name")).alias("name_lc"))

# Over this (slower)
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf
lower_udf = udf(lambda s: s.lower() if s else s, StringType())
out = df.select(lower_udf("name").alias("name_lc"))

7) Tune Partition Counts and Target Sizes

Ensure spark.sql.shuffle.partitions is not too low for large data. Pair this with AQE so Spark can coalesce or split partitions at runtime.

# PySpark
spark.conf.set("spark.sql.shuffle.partitions", 1000)  # example for very large datasets
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

Increasing partitions spreads the load but doesn’t remove fundamental skew. Combine with other fixes (broadcast, salting, repartition by composite keys).

8) Range Partitioning (Skew-Aware Distributions)

If your key is numeric or sortable and roughly uniform, range partitioning can produce more even partitions than hashing.

-- Spark SQL: repartition by range
SELECT * FROM t DISTRIBUTE BY RANGE(key) SORT BY key;

# PySpark
balanced = df.repartitionByRange(1000, "key").sortWithinPartitions("key")

This can reduce hotspots if your keys are amenable to ranges and your data is not heavily concentrated in one range.

9) Two-Phase Join for Extreme Skew

For highly skewed joins, isolate hot keys and process them with special logic (such as salting), while processing the rest normally. Finally, union the results.

# PySpark: split hot and cold keys
hot = df_keys.orderBy(F.desc("cnt")).limit(100).select("join_key")
hot_keys = [r[0] for r in hot.collect()]

large_hot = large.filter(F.col("join_key").isin(hot_keys))
large_cold = large.filter(~F.col("join_key").isin(hot_keys))

# Handle hot with salting or broadcast if small enough
treated_hot = salt_or_broadcast_join(large_hot, small)

# Handle cold with regular join
treated_cold = large_cold.join(small, "join_key", "inner")

result = treated_hot.unionByName(treated_cold)

10) Improve File and Table Layout

Partition your tables (e.g., by date) so each job reads less data.
Use columnar formats (Parquet/ORC) for predicate pushdown.
Avoid tiny files: compact small files so task startup overhead does not dominate. Use partition overwrite modes and compaction utilities.
Bucketing can help with consistent partitioning across tables used in frequent joins, though it adds maintenance overhead.

11) Resource and Speculative Execution Tuning

Right-size executors: too many cores per executor can cause GC pressure. Too little memory can cause spills.
Enable speculation to mitigate slow tasks due to node issues:

# PySpark: speculative execution
spark.conf.set("spark.speculation", "true")

Speculation helps with flaky nodes or transient slowness. For true skew, you still need data fixes.

Concrete, Beginner-Friendly Examples

Example A: GroupBy Aggregation Stuck at 199/200

Problem: You run a groupBy on country. One country (say, US) has 70% of rows. The groupBy triggers a shuffle to 200 partitions and the partition holding US becomes massive.

Fix options:

Enable AQE to split skewed partitions at runtime.
Repartition by a composite key (e.g., country + hashed user_id) if you need further aggregation later.
Pre-aggregate at a finer grain (e.g., user_id, day) to cut volume before the country-level groupBy.

# PySpark
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

# Pre-aggregate first
pre = df.groupBy("country", "user_id").agg(F.sum("amount").alias("amt"))
out = pre.groupBy("country").agg(F.sum("amt").alias("country_total"))

Example B: Skewed Join on customer_id

Problem: Joining a large fact table to a dimension on customer_id. A handful of customers dominate traffic.

Fix options:

Broadcast the dimension table if small enough.
Salt the large table’s hot keys and expand the small table to match.

# PySpark: broadcast if dim is small
dim = spark.table("dim_customer")
fact = spark.table("fact_events")

joined = fact.join(F.broadcast(dim), on="customer_id", how="inner")

Checklist: Diagnose and Resolve “199/200 Tasks”

Open Spark UI and identify the stage with 199/200 tasks.
Sort tasks by duration and shuffle read size; confirm skew (one huge task).
Identify the join/groupBy key; profile top keys by count.
Quick wins:
- Enable AQE: spark.sql.adaptive.enabled = true
- Broadcast small side: BROADCAST() hint or threshold
- Adjust spark.sql.shuffle.partitions to match scale
Structural fixes:
- Repartition by a better or composite key
- Pre-aggregate to reduce shuffled volume
- Use salting for hot keys
- Use DataFrame/SQL built-ins over Python UDFs
Re-run and validate improvement in Spark UI (balanced tasks, reduced GC/spill).

How to Size Partitions and Anticipate Skew

Partition count best practice depends on cluster size and data scale. A rough approach:

Start with spark.sql.shuffle.partitions = a few multiples of the total executor cores (e.g., 2–3x).
Monitor average vs. max task input size. If max is many times larger, investigate skew.
Use AQE so Spark can adapt partition sizes and split skewed partitions on the fly.

Anti-Patterns That Cause Skew Pain

groupByKey on huge datasets: switch to reduceByKey/agg with combiners.
Joining without statistics: not broadcasting the small side when you could.
Single-column partitioning on obviously skewed columns without mitigation.
Excessive Python UDFs in the middle of shuffle-heavy pipelines.
Too few partitions for large data, creating elephant partitions.

Monitoring Tips for Ongoing Health

Automate metrics collection from Spark UI or event logs (task duration percentiles, shuffle read/write distribution, GC time).
Alert on skew signals: a single task’s shuffle read size or duration exceeding the median by, say, 10x.
Track data distribution over time. Today’s balanced key can become tomorrow’s hotspot after product changes.

Frequently Asked Questions (FAQ)

Q1: Why does Spark always get stuck at 199/200 tasks?

Because 200 is the default number of shuffle partitions in Spark SQL. If a stage has 200 tasks and one is much slower, you will see 199/200 complete. This is usually due to data skew where one partition has much more data than the others.

Q2: Will increasing spark.sql.shuffle.partitions fix the problem?

It can help spread the load, but it won’t fix severe skew. The heaviest key will still dominate one or a few partitions. Pair partition tuning with techniques like AQE, broadcasting, salting, and smarter partitioning.

Q3: How do I know if broadcasting is safe?

Check the size of the small table. If it comfortably fits in executor memory (considering number of executors and concurrent tasks), broadcasting is usually safe. You can also set spark.sql.autoBroadcastJoinThreshold to a conservative value and let Spark decide.

Q4: What is the difference between coalesce and repartition?

coalesce(n) shrinks the number of partitions without a full shuffle (best for decreasing partitions). repartition(n) increases or changes partitioning with a full shuffle. For skew fixes, you often need repartition.

Q5: Is Adaptive Query Execution enough on its own?

Often, yes—AQE can split skewed partitions and coalesce small ones. But for extreme skew or very specific business keys, salting or two-phase joins may still be necessary for the best performance.

Q6: How do I pick the number of salt buckets?

Start small (e.g., 10–20) and monitor the result. You want each salted partition to be similar in size to non-skewed partitions. Over-salting increases overhead; under-salting leaves skew.

Q7: Does speculative execution fix skew?

Speculation helps when a task is slow due to a bad node or transient issues, not when the task has far more data. It is complementary but not a replacement for skew fixes.

Q8: My job is in Python. Are Python UDFs the cause?

Python UDFs can slow tasks, but they do not cause skew. They can amplify the pain because each record is slower to process. Prefer SQL/DataFrame functions or Pandas UDFs when possible, especially before shuffles.

Q9: What if both sides of the join are large?

Try pre-aggregating before the join, repartitioning by a better key, or applying salting. If there is a selective filter, push it down earlier to reduce data volumes.

Q10: How do I verify that the fix worked?

Re-run and check the Spark UI: task durations should be more even, max shuffle read size closer to the median, fewer spills, and no long tail with 199/200 tasks stuck.

Conclusion

The infamous “199/200 tasks” hang is almost always a sign of data skew—the final task is chewing through a disproportionate share of data. The good news is that you have battle-tested tools to fix it. Start with enabling Adaptive Query Execution and broadcasting the small side of joins. If the skew persists, repartition by better keys, pre-aggregate early, or apply salting for especially hot keys. Use the Spark UI to confirm the diagnosis and validate each change. With these techniques, your spark job stuck at 199 of 200 tasks fix won’t be a mystery—it’ll be a predictable, solvable engineering task.

Key takeaway: skew is a data problem first. Solve it at the data layout and transformation level, and your Spark jobs will run fast and predictably.