Louis Tsang
№ 0069 min

Amdahl's law assumes the workers are interchangeable

A three-minute lyric video takes about three and a half minutes to render on one rented machine. The machine runs on Railway, a service that rents computers by the minute. It has 24 vCPU — 24 shared processor cores. The job never changes: 187.84 seconds of audio, 5,635 frames, 1080p at 30 frames per second.

On one machine, that job finishes in about 217.5 seconds. Slower than sitting through the video it draws. So the export is worth speeding up.

Remotion, the library that turns the frames into a video, documents a faster path. Split the frames into two halves. Send each half to its own machine. Render both at once, then stitch the halves into one file.

The math is not subtle. Two machines, half the work each, half the time. So the plan is simple: rent the second machine, split the job, wait half as long.

It came back slower

Two machines of twelve rendered the video in 337.5 seconds. One machine did it in 217.5. Same total compute, split in half, and 55.2% slower.

Wall clock per configuration against the realtime referenceHorizontal bars showing end-to-end wall clock for thirteen benchmark runs of a 187.84 second lyric video, grouped into four configurations: a concurrency sweep on the 32-core host (350.9, 323.0, 317.2 and 323.2 seconds), three runs at concurrency 16 on the 48-core host (222.4, 221.1 and 209.0 seconds), three distributed runs across two 24 vCPU services (197.0, 196.3 and 179.1 seconds), and three distributed runs across two 12 vCPU services at equal total compute (335.0, 338.0 and 339.5 seconds). A vertical hairline marks 187.84 seconds of realtime; only the 179.1 second run falls to its left. Two span annotations mark a 9.6 percent spread bought by concurrency tuning and a 32.1 percent gap between the mean of each host class at concurrency 16.REAL TIME187.84sCONCURRENCY −9.6%32-CORE HOST · CONCURRENCY SWEEPc8350.9sc12323.0sc16317.2sc16323.2sHOST CLASS −32.1%48-CORE HOST · C16run 1222.4srun 2221.1srun 3209.0s2 × 24 VCPU DISTRIBUTED · C16run 1197.0srun 2196.3srun 3179.1s1 of 13 runs finishes under realtime2 × 12 VCPU DISTRIBUTED · C8 · EQUAL COMPUTErun 1335.0srun 2338.0srun 3339.5s

Tuning had bought almost nothing next to this. Sweeping concurrency — how many frames a machine renders at once — from 8 to 16 bought 9.6%. Then the same job landed on a faster physical host and ran 32.1% quicker, with no change to any setting. Which machine you drew mattered 3.3 times more than every knob.

The obvious reading is that splitting made it slower, so splitting is a bad idea. The experiment killed distributed rendering on that number, and the kill was right. It was right for reasons the experiment did not yet have.

It wasn’t the protocol

The first suspect is the split itself. Maybe splitting a render is simply broken.

It is not. On two machines of 24 vCPU, the two halves finished in 115.7 seconds against 217.5 on one — a 1.880× speedup on double the compute. That is 94.0% parallel efficiency: how close to a perfect doubling the split got, near the ceiling arithmetic allows.

The output agrees. Every full render produced a file of exactly 36,218,049 bytes, across both host classes, cold and warm. Every distributed run produced 36,306,871 bytes — 0.245% larger, the price of stitching two halves. Concurrency moves speed by 10%, the host by 47%, and not one output byte moves.

The split is clean. Whatever cost the 55% was added after the frames were drawn.

It wasn’t the network

The next suspect is the network. Two machines have to move the chunks between them, and moving data takes time.

It takes 0.3 to 0.4 seconds. The network is not it either.

The real constant is one function. After both halves render, one machine runs combineChunks() — the step that stitches the two files into one. It ran 46.3 to 65.8 seconds every time: 31.9% of the whole job. Without it, two machines beat one by 44.0%. With it, by 12.3%. The seam ate 32 of those 44 points.

Where the distributed win goesThree bars compare one 24 vCPU machine at 217.5 seconds against two machines. The two-machine chunk phase finishes in 115.7 seconds, a 1.880 times speedup at 94.0 percent parallel efficiency. End to end, the same run takes 190.8 seconds, because a hatched seam block of combineChunks work runs after the chunks are done. The cross-machine chunk download costs 0.3 to 0.4 seconds and is under a pixel wide at this scale.ONE MACHINE, 24 VCPU217.5sTWO MACHINES, CHUNK PHASE115.7s1.880× on 2× compute — 94.0% parallel efficiencyTWO MACHINES, END TO ENDchunk download: 0.3–0.4s — under a pixel here190.8sCOMBINECHUNKS() 52–66SThe seam ate 32 of 44 points: 44.0% faster without it, 12.3% with it.

Remotion marks the spot itself. Its combineChunks page carries a note: “This is a hard-to-use API that most people should not use directly.” The warning sits on the exact step that cost the win — not on distribution, which worked. The library named the part that broke.

One result stays open. combineChunks always ran on the same machine. The smaller, slower 12 vCPU box stitched faster than the bigger 24 vCPU one: 46.32 to 46.38 seconds against 52.1 to 65.8. Same output to the byte, on half the cores, with a tighter tail. Nobody knows why.

But the seam is a constant, and it is not large. It cannot explain a 55% headline, nor one distributed run finishing in 179 seconds while another never came under 335. Something else was moving, machine to machine, and nothing in the plan had named it.

The field the logs were writing down

Every run wrote a log. One field in it, unread until now, had been recording the answer the whole time.

The field is cpus. It comes from os.cpus(), a call that asks the operating system how many cores it has. Inside a container — a fenced-off slice of a bigger machine — that call reads past the fence and returns the whole host’s cores, not the slice you rented.

One broken reading doing two jobsA container running under a cgroup quota of 24 vCPU sits inside a host with 48 physical cores. A call to os.cpus() inside the container reads straight through the container boundary and returns 48, the host’s core count, not the quota. A dashed leader marks the point on the container edge where the reading should have stopped. The same reading then feeds two things at once: it is logged as a cpus field, an accidental host fingerprint, and it sets the default concurrency at os.cpus() divided by 2, which is 24 — the setting that always failed.HOST — 48 PHYSICAL CORESCONTAINER — CGROUP QUOTA 24 VCPUos.cpus()reads through48cores on the hostwhere the reading should have stoppedlogged as cpus —an accidental host fingerprintdefault concurrency = os.cpus() / 2 = 24the setting that always failed

So the field fingerprints the physical host. Across the seven full renders, every slow run logged 32 cores and every fast run logged 48. There is no overlap. cpus=32 renders in 317 to 351 seconds; cpus=48 in 209 to 222. That is 1.472× on wall clock. It repeats at another size: two 12 vCPU machines splitting the same job run 278.0 seconds against 188.9, 1.472× again. Two sizes, one ratio.

The variance was never noise. It was a two-class draw, written into every record the service kept. Railway hands you one host or the other, and the slow class runs 1.472× behind the fast one. Which one you draw is settled before the first frame renders.

The same misread does a second thing. Remotion’s renderMedia() — the call that renders the frames — defaults concurrency to half the cores it sees. In a container, half of a wrong number.

A 48-core draw defaults to 24, well past the 16 that ran fastest. A 32-core draw defaults to 16 exactly. The default is a coin flip, and it lands wrong on the faster host. Nobody hit it, because the experiment always passed concurrency by hand.

The field explained the variance. What it did not do was add up the slowdown — and that took three more numbers.

Where the 120 seconds went

The gap is 120.0 seconds, and it comes apart cleanly.

The fast machine finished its half in 188.9 seconds, then sat idle for 89.1 seconds — 74.2% of the gap — waiting for the slow machine to finish. The seam added 53.6 seconds, 44.6%. Those over-count, because the smaller box gave back about 22.6 seconds of its own efficiency. So 89.1 + 53.6 − 22.6 = 120.1, against a measured 120.0. Three fields never fitted to each other close to a tenth of a second.

One 2×12 run, in timeA timeline of a single distributed render across two twelve-vCPU machines. Chunk 0 draws a 32-core host and runs 278.0 seconds; chunk 1 draws a 48-core host and finishes its work in 188.9 seconds, then sits idle for 89.1 seconds waiting for chunk 0. combineChunks() then runs for 53.6 seconds, ending at 337.5 seconds. A single 24-vCPU machine finishes the same render in 217.5 seconds. A dashed line marks 242.4 seconds, the arithmetic best case in which both machines draw the fast host — a figure calculated, not measured.050100150200250300350CHUNK 0 — CPUS=32combineChunks()278.0s53.6sCHUNK 1 — CPUS=48188.9sIDLE89.1sEND TO END337.5sONE MACHINE, 24 VCPU217.5s242.4sboth machines fast — the best caseARITHMETIC, NOT MEASURED

For 89 seconds of every equal-compute run, a machine billed by the minute sat doing nothing.

Give both machines the fast class and the run lands at 242.4 seconds: 11.5% behind one machine, not 55.2%. That figure is arithmetic, not a measurement — no such fair-draw run was ever recorded. A retry erases 44 of the 55 points. The number that killed distribution was mostly a coin.

The recovery has a stronger form, and it still loses. The split handed each machine half the frames, not half the speed. Match the split to the measured rates — 10.14 frames per second on the slow box, 14.91 on the fast — and the run still lands near 278.5 seconds, 28.1% behind. Even a perfect split loses. Equal compute is hostile by definition: half the frames on half the compute barely moves the stitch, so the seam is added for free.

The evidence has holes, and they run one way. Every logged row is a success, because the harness records only wins — so every failure survives as prose, not data. The published figure even put the loss at 52%, by comparing the distributed average against the single slowest fast run, not the average.

The concurrency sweep ran once per setting, so “16 is best” does not survive its own noise. The timing poll charged the distributed runs twice. Even the 24 vCPU is the dashboard’s number, not one the logs ever recorded. Every correction available makes distribution look better than the number that killed it.

The test asked distribution to reach 80% of single-machine time — to be at least a fifth faster. It measured 155%.

The margin the reasoning error had to crossA number line measured in percent of single-machine wall clock. A pre-committed kill line sits at 80 percent: distributed rendering had to be 20 percent faster to survive. The measured distributed result lands at 155 percent, shown as a solid mark. Correcting the reasoning error moves the answer to 111 percent for a fair host draw and 128 percent for a perfect speed-proportional split; both are arithmetic, not measured, and are shown as open marks. The whole correction is worth 44 points, against a 75-point margin to the kill line, so every corrected answer still fails.PERCENT OF SINGLE-MACHINE WALL CLOCKOPEN MARKS ARITHMETIC, NOT MEASUREDTHE PRE-COMMITTED KILL LINE20% FASTER OR DISTRIBUTED IS DEADfair host draw (arithmetic)measuredperfect speed-proportional split111%128%155%60%80%100%120%140%160%THE ENTIRE REASONING ERROR — 44 POINTSMARGIN — 75 POINTS

The whole reasoning error was worth 44 points against that 75-point margin. Correcting it moves the answer from no to no.

Amdahl’s law assumes the workers are interchangeable

There is a name for the ceiling all of this kept hitting. Amdahl’s law: divide the part you can parallelise by the number of machines, and the part you cannot — the seam — does not move. Spread 217.5 seconds of work across two machines and you still cannot land under the ~55-second stitch.

The law is right. The seam is the floor.

But the law assumes one thing this experiment broke. It assumes the workers are interchangeable — cores in a box you own, the same every run. A machine rented by the minute is not a worker. It is a draw from a pool with two classes, 1.472× apart.

Two host classes, measured, and the worst-of-N penalty, modelledOn the left, seven single-machine runs at 24 vCPU and six half-chunks at 12 vCPU each split into two clusters by the cpus field, with an empty corridor between them and a 1.47 times gap that reproduces at both machine sizes. On the right, a modelled curve of the expected penalty against the number of machines, rising from 1.134 at N equals 1 to 1.438 at N equals 8 and converging on a dashed asymptote at 1.47 times, labelled the slow machine. The curve assumes 2 slow draws in 7 and is a model, not measurement.MEASURED24 vCPU · one machine · 7 runscpus=48cpus=32209–222s1.47×317–351s12 vCPU · half-chunks of a 2×12 splitcpus=48cpus=32188–190s1.47×277–278sMODELLEDexpected slowdown of a split vs one machine1.47× — the slow machine1.001.251.501.1341.2301.3481.4381248N MACHINESMODEL ASSUMES 2 SLOW DRAWS IN 7. THE RIGHT-HAND CURVE IS A MODEL, NOT MEASUREMENT.

Rented machines add a term the law never had. One machine draws once; run it enough times and it averages out to the mean. N machines draw N times per run, and the run ends when the slowest of them finishes.

That is not the mean of N draws. It is the maximum. And the maximum only worsens as you add machines, because each new machine is another chance to hit the slow class.

The draw here came up slow twice in seven — an estimate with an enormous interval, one provider, one afternoon. The clamping holds for any chance above zero. The shape is the claim; the values are an illustration.

So the three terms pull apart. Split further and the division improves, the seam holds, the draw degrades. Only one of the three buys anything when you spend more. You can retry a draw; you cannot retry a maximum.

The rule carries with no video in it. A split pays when the work you can actually remove, T_single · (1 − 1/N), beats the seam. Here 217.5 × 0.5 = 108.7 seconds of removable work against a ~55-second seam, so it clears — barely. It buys 12.3% of wall clock at roughly double the cost: $0.037 to $0.039 on one machine against about $0.07 on two.

And distribution’s luckiest run beats a single machine’s average draw by 4.4 seconds — 1.8%. That is the entire prize for two services, a chunk contract, a file protocol, and a stitching step.

Run the rule backwards and it gets worse. Halve the cost per frame, from any cause, and one machine drops from 217.5 to about 110 seconds while the seam stays ~55. Then 110 × 0.5 = 55, and the split stops paying at all. Cheaper frames make splitting pointless — the seam never shrinks when the work does.

A split is a division. A seam is a constant. The machines are a draw. Only the first improves when you spend more. You can average a hundred draws. You cannot average a maximum.

References

  1. ”Distributed rendering”, Remotion documentation.
  2. ”combineChunks()”, Remotion documentation.
  3. ”renderMedia()”, Remotion documentation.
  4. ”Pricing”, Railway documentation.