Livecoding the Spectral Canvas ✨

SpectralOli is a browser-based live coding tool that bypasses synthesis entirely by treating audio spectrograms as an editable visual canvas. Integrated directly with the Freesound, SpectralOli allows performers to retrieve and dynamically reshape audio loops in real time using a custom domain-specific language (DSL).

By shifting the focus from synthesizing sound to visually sculpting existing material, it significantly lowers the barrier to entry to advanced audio effects, and enables rapid musical experimentation. Check out this short demo/ performance:

SpectralOli's DSL is structured around four primary operational layers:

Global directives that establish parameters used by multiple operations.
Time sequencing operators that reorder and pattern the rhythmic grid.
Frequency masks that isolate specific spectral bands.
Time-frequency transformations that warp the matrix spatially.

To compose the DSL, statements for these layers must be separated by newlines. Within a layer, operations (like time sequencing operators or time-frequency transformations) can be chained together using dot syntax to compose complex effects. Crucially, the DSL enforces a strict order of execution regardless of the order in which the code is written. Global directives and slice boundaries are always resolved first to define the available audio material. Time sequencing runs next to determine which slice plays. Then, frequency masks isolate the spectral bands. Finally, time-frequency transformations operate on that isolated, sequenced spectrum.

Because the DSL compiles directly to JavaScript, all numeric arguments accept JavaScript expressions. So, Math functions, bitwise operators and conditional expressions are valid wherever a number is expected. A built-in time variable also exposes the current playback position as a floating-point scalar, allowing any parameter to evolve continuously as the loop plays. The following sections detail these DSL layers and how they combine to create sonic textures.

0. Global Directives

Global directives establish the fundamental processing environment for the track before any manipulation occurs. Because SpectralOli operates in the time-frequency domain, the foundation of the audio processing relies on how the sound is converted into a time-frequency matrix.

Unlike operations in other layers, global directives cannot be chained together; instead, they are written one by one and separated by a space or a newline.

Directive Reference

Directive	Description
`fft <size>`	Configures the Short-Time Fourier Transform (STFT) window size. Smaller sizes (e.g. `fft 512`) provide excellent time resolution. Larger sizes (e.g. `fft 4096`) provide high frequency resolution.
`clock <multiplier>`	Adjusts the global playback speed for the track relative to the master tempo. Positive values speed up playback (e.g. `clock 2` plays at double speed). Negative values (e.g. `clock -1`) play the audio buffer in reverse.
`gain <expr>`	Scales the overall output amplitude. Its value can be a dynamic expression that evolves over time (e.g. `gain 1.5` for a fixed gain, or `gain Math.sin(time) * 0.5 + 1` for dynamic modulation).

// Optimize the matrix for melodic frequency resolution
fft 2048
// Play the track at half-speed
clock 0.5
// Double the gain (Yeahh!)
gain 2

1. Loop Slicing & Sequencing

Chop audio in the time domain using automatic onset detection or uniform grid slicing, then reorder, stutter, and sculpt slices rhythmically using the seq() sequencer.

Global Slicing Directives

These prefix commands chop the buffer and populate the slice editor automatically.

Command	Behavior & Analysis
`slicep n`	Percussive onset detection based on high-frequency energy transients. `n` must be a power of 2 between `256` and `8192`.
`slicem n`	Melodic onset detection using spectral flux to detect pitch and harmonic transitions.
`slicee n`	Equal-width slicing. Divides the audio buffer into `n` chunks of uniform length.

Sequencing & Slice Specs

Once the loop is discretized into an array of slices, the seq function allows users to construct rhythmic patterns with slice syntax using colons, similar to Python list indexing:

Spec Example	Selection Result
`seq()`	Play all existing slices sequentially (`0..N-1`).
`seq("0, 2, 4")`	Play slice 0, then 2, then 4 in order.
`seq("0:4")`	Play slices 0, 1, 2, and 3 (stop index is exclusive).
`seq("-4:")`	Play the last 4 slices of the buffer.

Pattern Transformations

Target specific steps using .at(spec, op, prob) where `spec` selects the target slices, `op` is the operation and `prob` is the probability of that operation being applied at every loop iteration, We can also chain multiple transformations directly with a dot.

Operation	Description
`stutter(n)`	Repeat each slice `n` times.
`fast(multiplier)`	Speeds up playback by a positive factor, So `2` plays at double tempo.
`slow(divisor)`	Slows down playback by a positive factor, So `2` plays at half tempo.
`reverse()`	Play the slices backwards without altering sequence order.
`repeat(n)`	Repeat the entire block `n` times.
`shuffle()`	Randomly permute/ shuffle targeted slices.
`silence()`	Mute the slices.
`euclid(hits, steps, offset)`	Distribute `hits` evenly as possible `steps` (Euclidean rhythm).
`mirror()`	Append a reversed copy of the sequence.

// Some chops and transformations on a 512-sample percussive loop
slicep 512
seq()
  .at("4:6", reverse())
  .at("6:10", fast(1.2))
  .at("14", stutter(2))
  .at("30:33", shuffle())
  .at("33:", slow(1.5))

Periodic Operations (every)

You can apply transformations periodically using the every(n, operation) method. It runs the provided pattern transformation only once every n loop cycles.

// Every 4 cycles, reverse the entire pattern
slicep 512
seq(":8").every(4, reverse())

2. Frequency Operations

Filter sound in the frequency domain by carving frequency passbands, and dynamically combining regions using boolean algebra and mathematical formulas.

Frequency global directive: fft n sets the STFT resolution i.e frame size with the default 1024

Filtering Regions

Method	Behavior
`low(hz)` / `high(hz)`	Pass frequencies below (low-pass) or above (high-pass) `hz`.
`band(min, max)`	Pass frequencies between `min` and `max` Hz.
`harmonic(f0, count, width)`	Pass fundamental frequency `f0` and its `count` integer harmonics.

Combining Regions with Mask Algebra

A track has at most one frequency mask, written as a single infix expression combining regions with real arithmetic operators and not method chaining:

Operator	Behavior
`a + b`	Combine two regions, e.g. `band(200, 4000) + high(8000)`.
`a - b`	Carve `b` out of `a`, e.g. `band(200, 4000) - band(800, 1000)`.
`!a`	Flip the region,Use parens to negate a whole sub-expression: `!(a + b)`.

JS arithmetic & Playback Time Modulation

Arguments support JavaScript arithmetic (+, -, *, /, %), Math functions (Math.sin, Math.cos), and the dynamic variable time (current playback time in seconds):

// LFO-modulated bandpass filter
band(400 + Math.sin(time * 2) * 200, 2000) + high(8000) - band(800, 1000)

// Sweeping low-pass filter inverted into a sweeping notch
!low(time * 1000 % 10000)

3. Spectral Operations

The final layer consists of geometric transformations applied directly to the time-frequency matrix.

Spectral Blur & Granulation

Method	Description
`.blur(time_amt, freq_amt)`	2D spectral smoothing. `time_amt` controls frame decay across time; `freq_amt` smears energy across adjacent frequency bins.
`sgranulate(scatter, mix)`	Spectral granulation scatters and recombines spectral frames from past rolling buffer history. Mix is a dry/wet blend (0 = dry, 1 = fully granular).

Matrix Transformations

Method	Sonic Equivalent & Matrix Behavior
`.scale(scaleX, scaleY, mix)`	`scaleX` stretches time; `scaleY` stretches bin spacing.
`.rotate(degrees, mix)`	Rotates the matrix around its center point, shearing time into frequency and frequency into time.
`.skew(skewX, skewY, mix)`	Independent X/Y rotation of time-frequency matrix.
`.transpose(mix)`	Flip time and frequency axes across the diagonal.

// Heavy time decay with subtle frequency smear on a bandpass filter (mask, then pipeline)
band(100, 5000)
blur(0.85, 0.2)

// Modulated spectral granulator where scatter density shifts over time
sgranulate(Math.sin(time * 0.5) * 0.5 + 0.5, 0.9)

// Inharmonic frequency stretch with 2x time zoom
band(100, 6000)
scale(2, 1.5, 1)

// Rhythmic matrix rotation modulated by playback time
rotate(time * 45 % 360, 0.8)

// Spectral delay dispersion (X skew) and continuous pitch riser (Y skew)
band(200, 8000)
skew(0.5, 0.2, 1)

// Transpose: reflect matrix across diagonal (swap time and frequency dimensions)
transpose(1)