I’m the founding father of StreamGen, an AI clipping platform for Twitch. Whereas constructing it, I observed that fashionable clipping tech continues to be caught within the speech to textual content period. We now have insanely highly effective multimodal fashions that may natively watch and hear, but virtually each clipping instrument on the market nonetheless doesn’t leverage these capabilities.
Whereas video-native understanding continues to be too costly at scale, I feel it’s time for audio-native clipping to grow to be the brand new commonplace. However why is it higher?
Conventional STT pipelines throw away lots of info: laughs, ambient sound, overlapping audio system, recreation audio cues (gunshots, win/lose stingers), quantity and tone, shouting vs. whispering… Loads of helpful alerts get misplaced.
With an audio-native strategy, the mannequin makes use of the complete audio options, considerably enhancing clipping high quality. It’s additionally a lot easier: as a substitute of a posh multi-step pipeline, audio is remodeled instantly into clips in a single go.
Why aren’t we already utilizing audio-native fashions?
They’re too costly. Audio-native processing is 2–3× costlier than STT + textual content LLMs. The costs within the chart above had been calculated utilizing Gemini’s batched enter/output pricing. In follow, it’s best to ship clips to your customers in far lower than 24 hours, so a extra sensible worth can be roughly double.
The issue is even worse for merchandise like StreamGen as a result of we analyze livestreams. A streamer might go stay 20 instances a month for a median of 4 hours per stream. With Gemini’s native audio understanding, that will price about $7/month with Gemini 3.1 Flash Lite and $25/month with Gemini 3.5 Flash.
You might already decrease prices by utilizing smaller fashions (Gemini Flash Lite) or by decreasing/eradicating reasoning. However that will instantly damage clipping efficiency, and we shouldn’t enhance margins by decreasing product high quality. This isn’t a superb route.
On this put up, I’ll present you the way I educated a tiny audio LLM to unravel this downside. The standard is almost on par with Gemini 3.5 Flash (on my evaluations!), however at the very least 22× cheaper.
Let’s begin.
Constructing the coaching dataset
I constructed a dataset from ~250 hours of Italian livestreams, pairing quarter-hour of uncooked audio with extracted “greatest moments” from Gemini 3.5 Flash. I reused components of the StreamGen infrastructure to construct this knowledge pipeline. It may scale to much more hours of audio, however for this primary experiment I selected to begin small.
Why solely Italian livestreams?
- My finish aim is to construct a multilingual clipper, so why begin with English?
- I do know the Italian Twitch/Livestream group properly sufficient to confidently and manually choose what’s clippable and what isn’t
- I’m Italian 🍕
Try 1: simply SFT it
I began with the best strategy that might work: supervised fine-tuning Voxtral instantly on Gemini’s output. Listed below are the outcomes, measured with an IoU threshold of 0.5 (a clip is matched if its IoU with the ground-truth clip is bigger than 0.5):
- Precision@0.5 = 20%
- Recall@0.5 = 18%
The mannequin discovered solely 18% of the ground-truth clips. Even after coaching on the complete 250-hour dataset, clipping efficiency was nonetheless fairly weak.
My preliminary thought was: let’s construct an even bigger dataset! However earlier than throwing extra compute on the downside, it was price asking whether or not there was a extra data-efficient strategy.
Try 2: make it easy
Within the first try, the mannequin needed to predict a number of clips inside a 15-minute audio, and for every one, a title, virality rating, description, and exact begin/finish instances. That’s so much to ask.
So I stripped it down:
- 15-minute audio → 5-minute audio
- Predict a number of clips → predict simply the only greatest one
- Title + description + virality rating → simply begin and finish time
Listed below are the outcomes after one other full SFT run:
The outcomes are even worse than the primary run. Why? My greatest guess is that in Try 1, the mannequin had a number of tries per chunk, so the chances of hitting at the very least one right clip had been increased. Right here, it will get precisely one shot, and it’s proper solely 11% of the time. Not nice.
Per week into the undertaking, I used to be beginning to really feel discouraged, particularly after burning many {dollars} on compute with little to indicate for it. However I needed to make this work, so I saved grinding.
Try 3: a brand new route
After brainstorming with GPT5.6 Sol, I landed on a totally totally different strategy. As a substitute of treating this as an autoregressive technology downside, I deal with it as a classification downside.
As a substitute of asking the mannequin to generate a JSON describing a clip in an audio chunk, I ask a a lot easier query: is that this particular second price clipping? The mannequin simply outputs a single likelihood.
That is a better activity to unravel, however it prices us the flexibility to instantly predict clip boundaries. To work round that, every 5-minute chunk will get break up into overlapping, strided home windows. In my case, 13 home windows of 1 minute every, with a 20-second stride.
The very best-scoring window turns into our clippable second. The boundaries are now not correct right down to the second, however we’ve lowered the issue from “decide a begin and finish day out of ~40k prospects” to “decide the most effective of 13 home windows.”
The loss perform
All home windows from a given 5-minute chunk are batched collectively and compete towards one another. The mannequin predicts a “clip” likelihood for every window, and the extra a window overlaps with an actual ground-truth clip, the extra it’s rewarded.
This can be a cross-entropy loss between the mannequin’s predictions and every window’s overlap price with the ground-truth clip.
Outcomes
After one epoch over the complete dataset, I anticipated one other underwhelming end result. However I used to be unsuitable. The mannequin did study, and it was superb.
Right here’s the base mannequin’s efficiency at recognizing clippable moments:
This 15-minute audio is split into 1-minute home windows with a 20-second stride. Every bar is the anticipated clip likelihood, shaded in purple. Principally a clip heatmap. The bottom fact clips (in blue) had been produced by Gemini 3.5 Flash, however manually verified.
As you possibly can see, there isn’t a lot sign right here, virtually each window will get a excessive likelihood. Now let’s evaluate that to the fine-tuned mannequin:
Sure! After weeks of failed makes an attempt and lifeless ends, the mannequin clearly realized to clip. Seeing this picture for the primary time was unimaginable. Going from an concept to “it truly works” feels superb, and the truth that it is a tiny mannequin educated on just some hundred hours of knowledge makes it even higher.
For an additional take a look at, I took an actual user-made Twitch clip, and recorded about 8 minutes of the encircling audio:
The purple clip is the one made by an actual person on Twitch. Discover there are additionally excessive possibilities simply earlier than it. This was a Name of Obligation match, and people “sizzling” minutes from 1:40 to five:20 correspond to gamers flaming one another. The true “last flame” second, although, was truly contained in the Twitch clip.
I additionally ran the identical audio section by way of different AI clipping instruments and picked the highest clips based mostly on every platform’s personal “virality” rating.
Whereas VEED.io discovered the moments earlier than and after the “flame” (which I nonetheless assume are good), OpusClip additionally caught the precise Twitch clip.
This fast comparability highlights one other necessary side of this undertaking: good moments are simple to identify towards unhealthy ones, however they’re a lot tougher to rank towards actually nice ones. It is because the bounce from “good” to “superb” is usually subjective and taste-driven.
In different phrases, given two clips that each have a excessive likelihood of going viral, it’s very tough to foretell which one will go extra viral than the opposite. Even for a human.
What’s price noting, as a substitute, is that sgclipper-3b-v1 understood which half was in all probability clippable, however at a fraction of the fee, and with out doing any transcription step.
This mannequin is thrilling!
Depraved quick
My present inference script could be very unoptimized: it masses the BF16 mannequin, splits the audio into home windows, and processes them separately in sequence. Regardless of this, analyzing 10 minutes of audio on an RTX 5090 takes about 12 seconds.
Small and low price
sgclipper-3b-v1 is a small mannequin competing with Gemini 3.5 Flash’s native audio understanding.
With my present inference setup, it’s already able to processing roughly 50 hours of audio per hour of compute. At a median of $0.7/hr for an RTX 5090, that works out to $0.014 per hour of audio processed, already cheaper than the most cost effective STT APIs available on the market.
And that is with a BF16 mannequin, no batching, unoptimized audio tokenization, a reasonably small stride, and working on HuggingFace Transformers. I’m assured this might get 5–10x quicker with some optimization work, earlier than even contemplating mannequin pruning or utilizing different smaller/quicker pre-trained encoders/decoders. There are additionally cheaper GPUs that might run this, just like the NVIDIA RTX 4000 SFF Ada Technology, obtainable on Hetzner for $0.4391/hr.
Correctly optimized, this pipeline might discover clips at a value that’s too low cost to meter.
Linear processing time
As a result of we course of audio in home windows, we don’t have any context-length limitations and don’t pay any quadratic tax for longer durations. Processing time scales linearly as audio size will increase.
Subsequent steps
sgclipper-3b-v1 is succesful, quick, and low cost. It may establish clippable moments from uncooked audio alone, however it’s nonetheless removed from good. Subsequent, I’m specializing in scaling what works (extra knowledge and languages), making analysis extra rigorous, and making a production-ready inference stack.
I’m not open-sourcing the mannequin or the coaching dataset presently, because the analysis continues to be within the early phases. Nonetheless, I hope you continue to discovered this weblog put up priceless!
Conclusion
In 2026 AI clipping ought to be audio-native, however at this time’s options are nonetheless too costly. I needed to unravel this by distilling the clipping capabilities of Gemini 3.5 Flash right into a small 3B mannequin.
So I constructed a coaching dataset of 250 audio hours and took a number of failed makes an attempt. I began with an SFT generation-oriented aim and ended up with a classification strategy. And after weeks of iteration, I lastly educated sgclipper-3b-v1: a tiny, audio-native clipper, with linear-time processing and no transcription steps.
I don’t assume I might have ever finished this with out the superb work of the HuggingFace crew and contributors. I additionally wish to thank Mistral for offering a tremendous base mannequin to work on!
I’m desirous to know your ideas and concepts on this. So let’s chat! You’ll be able to attain me at cimolaimarco01@gmail.com or dm me on X @CimolaiMarco.
Thanks for studying 🤗
