perhaps i gave spark too hard of a task. what i observed was compactions every 50s for 25 minutes with no code changes or visible progress. then it bricked itself by eating into compaction buffer. it also has a tiny context and compacts at around 110k.