In our last post, we moved liquidsoap’s scheduler onto OCaml 5 domains and measured the result. We found no strong evidence that liquidsoap’s own internals benefited: the gains were in FFmpeg and I/O, and every video sweep we ran moved the streaming ceiling by about one stream.

We have found one now. Clocks, the loops that drive every stream, ran on a thread each. Running them as duppy tasks on the domain pool instead takes the same machine from 27 audio streams to 51.

Video hid this. A video stream’s cost is dominated by encoding, so the compute is what sets the ceiling and scheduling is a rounding error. An audio stream’s compute is small, and what liquidsoap itself does around it, scheduling included, becomes the driving factor.

The last thing still on a thread

Since the last post, liquidsoap’s scheduler runs on a pool of domains. Requests, network I/O, timeouts and server commands all go through duppy.

Clocks did not. Each ran a loop on its own thread: produce a frame, hand it to the outputs, sleep until the next frame is due. Fifty clocks are fifty threads, each needing a core for a moment every 20 ms, coordinated by nothing but the operating system.

one clock: a tick, then rest until the next frame is due

Clocks are now scheduler tasks. A clock ahead of real time parks instead of sleeping on a thread, and duppy’s timer resumes it when the next frame is due, on whichever domain is free.

four clocks: a thread each, or tasks on the scheduler’s domains

This is the default on main. settings.clock.task := false restores a thread per clock.

The workload that showed it

The last post’s video sweeps were a few large streams, where at 4K one stream occupies most of a core.

This is the opposite shape. Each stream decodes one file, encodes it once into three variants, and serves those tracks as an HLS playlist and three icecast mounts through the shared encoder. Each stream is an independent graph with its own clock, so the stream count is the clock count.

def make_stream(i) =
  s = mksafe(single("music.mp3"))

  # One encode per variant, shared by every output below.
  aac_low = ffmpeg.encode.audio(%ffmpeg(%audio(codec = "aac", b = "48k")), s)
  aac_high = ffmpeg.encode.audio(%ffmpeg(%audio(codec = "aac", b = "128k")), s)
  lame = ffmpeg.encode.audio(%ffmpeg(%audio(codec = "libmp3lame", b = "128k")), s)

  let {audio = low, metadata = m, track_marks = tm} = source.tracks(aac_low)
  let {audio = high} = source.tracks(aac_high)
  let {audio = mp3} = source.tracks(lame)

  encoded =
    source(
      {audio_low = low, audio_high = high, audio_mp3 = mp3,
       metadata = m, track_marks = tm}
    )

  output.file.hls(
    "/dev/shm/hls/s#{i}",
    [
      ("low", %ffmpeg(format = "mpegts", %audio_low.copy, %audio_high.drop, %audio_mp3.drop)),
      ("high", %ffmpeg(format = "mpegts", %audio_low.drop, %audio_high.copy, %audio_mp3.drop))
    ],
    encoded
  )

  output.icecast(
    %ffmpeg(format = "adts", %audio_low.copy, %audio_high.drop, %audio_mp3.drop),
    mount="s#{i}_low", encoded
  )
  # ...and one mount each for the other two variants.
end

Every stream carries five outputs, four of them writing to a socket or a file inside the clock’s cycle: the shape of a busy radio server.

We raise the stream count until a clock reports a catchup of a second or more. A rung runs 45 seconds, the first 10 discarded as startup. The machine has 8 cores, files are local, icecast runs on the same machine. 2.4.6 is the control: no domains, no clock tasks, a thread per clock by construction, and the release users upgrade from. It is built with OCaml 4.14; the 2.5 builds use OCaml 5.5.

Results

streams held in real time, one clock each

build streams held against 2.4.6
2.4.6 27
2.5, clock.task := false 19 -30%
2.5, clock.task := true 51 +89%

At equal stream counts the task animator leads on every measure. At 19 streams, the last count both 2.5 modes hold: 243% CPU against 292%, 14% of the frame budget against 45%, 1,700 context switches per second against 48,200.

Two ways to fall over

The failure modes differ too.

process CPU up to the first failure

As tasks, the process exhausts the cores: CPU climbs to 673%, the cost of one more stream goes from 4.4% of a core to 74% over the last four rungs, and involuntary context switches triple. The machine is saturated.

With a thread per clock, the process gives up at 228% of eight cores, with CPU falling. Cores sit idle while clocks miss frames: fifty threads each needing a slice every 20 ms cannot all be scheduled in time.

The difference is wakeups. A clock on a thread sleeps and wakes every frame, and the kernel arbitrates between all of them:

voluntary context switches up to the first failure

build voluntary context switches per added stream
2.4.6 2,565
2.5, clock.task := false 4,136
2.5, clock.task := true 69

A parked clock is a continuation in a queue, resumed on a domain that is already running. No thread wakes, so an added stream adds almost no scheduling cost.

Thread clocks on OCaml 5

2.5 with clock.task := false holds 19 streams against 27 for 2.4.6, burns more CPU at every stream count, and costs 1.6 times as many context switches per added stream. The two builds run on different runtimes: OCaml 5 reimplements how threads share the runtime, and a thread per clock is the workload that exercises it most. Running 2.5 with domains disabled entirely recovers 2 of the 8 streams. We have not yet run 2.4.6 on OCaml 5 to separate the runtime from liquidsoap’s own code.

The thread animator still matters: it runs whenever settings.clock.task is off, and JACK forces it. Against 2.4.6 the task animator’s gain is +89%; against 2.5’s own thread mode it is 2.7x.