How Gumdrop Visualizer Works | Oliver Cheng (Oli)

I built Gumdrop Visualizer because I wanted a music product that felt more like atmosphere than utility.

Not a DAW. Not a dashboard. Not a plugin that only works for one narrow setup.

I wanted something you could open quickly, point at a room, a speaker, a record player, or a laptop, and turn sound into a live visual environment.

That product goal shaped almost every technical decision that came after.

The product shape

Gumdrop has a few surfaces, but the web app is the center of gravity.

the browser app is the main product,
the Spicetify version is a companion for Spotify power users,
and there is a parked Mac prototype for future system-audio support.

That was intentional.

The browser product has the cleanest distribution story. Open a link, allow microphone access, and it works. No install friction. No platform lock-in. No need to explain audio routing to a casual user.

The Spicetify build is still useful, but I think of it more as a credibility surface. It proves the visual system can live close to the music timeline and makes the project legible to a certain kind of enthusiast.

Why I made it browser-first

A lot of music tools get trapped by their own setup complexity.

They assume the user is willing to:

install software,
configure drivers,
route audio internally,
or learn a specialized creative tool just to get started.

I wanted the opposite.

The core use case for Gumdrop is simple:

open the product,
choose Begin listening,
play music in the room,
let the visual world respond.

That is why the flagship web app listens through the device microphone instead of depending on direct player integrations.

It makes the product work with:

speakers,
instruments,
vinyl,
ambient sound,
and live rooms.

That choice is less technically “pure” than a fully routed audio input, but it is much better product design for the first experience.

The main architecture

At a high level, Gumdrop does three things:

capture sound,
turn that sound into a compact music-state frame,
feed that frame into GPU-rendered scenes.

The web app itself is a fairly lean React + TypeScript app, but the interesting part is the boundary between audio analysis and rendering.

The browser microphone stream flows into a small analysis pipeline built around:

getUserMedia
AudioContext
AnalyserNode
optional AudioWorkletNode beat detectors

From there, the app produces an AudioFrame object that looks roughly like this:

type AudioFrame = {
  beat: number;
  beatPhase: number;
  subdivision: number;
  syncConfidence: number;
  bpm: number;
  bar: number;
  section: number;
  onset: number;
  energy: number;
  brightness: number;
};

That frame is the bridge between music and visuals.

The renderer does not need raw audio buffers. It just needs a stable, expressive snapshot of the current musical state.

How the audio side works

The audio analysis tries to balance two things:

enough musical sensitivity to feel alive,
enough simplicity to stay robust in a browser.

There are a few layers to it.

1. Spectrum and waveform analysis

The app continuously samples both the frequency spectrum and time-domain waveform from an analyser node.

That is used to derive broad signals like:

low-end energy,
midrange presence,
highs and brightness,
onset intensity,
and general room energy.

Those values help the scenes react even when beat-lock is weak.

2. Multi-band beat detection

On top of the analyser, I added band-specific onset detectors for low, mid, and high frequency ranges.

That matters because musical attacks do not all live in the same place.

kicks often sit low,
snares and claps live more in the mids,
hats and sharper transient detail live higher up.

Each band can surface timing information, and the app fuses nearby events together so one chord or hit does not get double-counted as multiple beats.

3. Tempo estimation and clock confidence

The app keeps a rolling set of recent onset events and uses that to estimate tempo and beat interval.

It does not immediately pretend it knows the BPM. It builds confidence over time.

That is why Gumdrop exposes concepts like:

syncConfidence
beatPhase
phaseErrorMs

This is important product behavior.

When the app is not yet confident, it should still look good. When the pulse locks in, it should feel tighter and more intentional.

So the scenes are designed to degrade gracefully between “reactive” and “clocked.”

How the visuals work

The visuals are all GPU-rendered with WebGL 2 fragment shaders.

Each scene is essentially a small visual instrument.

The shared shader library currently contains scenes like:

Bloom
Tunnel
Aurora
Orbit
Prairie
Bubbles

Every scene receives the same core uniform set:

time,
resolution,
beat,
beat phase,
subdivision,
sync confidence,
bar,
section,
onset,
energy,
brightness,
and two color channels.

That means I can keep the rendering contract stable while changing the artistic logic per scene.

One shader might use u_beat to expand a ring. Another might use u_onset to flash an internal layer. Another might use u_bar and u_section to create slower, more structural movement.

I like this model because it keeps the art layer flexible without turning the whole app into a mess of special-case logic.

Why the shared scene library matters

One of the best decisions in the codebase was separating the scene library into a shared layer.

That lets the browser product, the Spicetify version, and any future surface all draw from the same visual core.

So the system is not:

one app with one renderer,
then another app with a separate renderer,
then a third experiment starting from scratch.

Instead, it is one visual language with multiple delivery surfaces.

That is a much better product architecture.

It gives me room to experiment with distribution without fragmenting the identity of the product.

Another product decision I like here is that scenes and palettes can be shared directly through URL parameters.

That is a small detail, but it changes the feel of the product.

Instead of making people create an account just to pass around a visual setup, I can let them share a specific look with a link.

That keeps the interaction lightweight:

pick a scene,
tune colors,
copy the look,
send it to someone else.

I think more creative tools should try this kind of sharing model before they reach for account systems.

Privacy and local processing

Gumdrop does not need to upload audio to be useful.

That was important to me.

The browser product analyzes sound locally and keeps preferences on the device.

For a music-reactive product, that is both a trust decision and a latency decision.

Shipping audio off-box would make the experience worse and the trust story weaker.

Sometimes the right architecture is not the most elaborate one. It is the one that keeps the loop tight.

Hosting and distribution

The web build is static, which keeps deployment simple.

That means it can live happily on Cloudflare Pages and similar static hosts while still feeling like a polished product.

That setup matches the broader product philosophy well:

fast to open,
easy to share,
easy to ship,
and not overbuilt for the job it needs to do.

What I think is interesting about this project

Gumdrop is not just a shader toy.

What interests me most is the product boundary it sits in:

part creative tool,
part ambient interface,
part visual instrument,
part shareable browser object.

It is a good example of the kind of thing I like building:

something expressive, technically opinionated, and simple enough that people can actually use it without a tutorial.

That usually matters more than adding one more clever subsystem.

Related notes:

Best,
Oli
August 18, 2026