API reference

MCP tools

19 tools an MCP client can call. Connect to https://api.scarleta.ai/v1/mcp/rpc with your API key, call tools/list to get this list, then tools/call to run a tool. The request format is on the MCP reference. A tool that runs long answers with a job_id; read the result with get_analysis_result.

Connect your agent

Point your MCP client at the endpoint and pass your API key. Once connected, the tools below appear to your agent next to its other tools.

{
  "mcpServers": {
    "scarleta": {
      "url": "https://api.scarleta.ai/v1/mcp/rpc",
      "headers": {
        "Authorization": "Bearer $SCARLETA_API_KEY"
      }
    }
  }
}

alignment_analysis

2 parameters

Compare a performance to a reference recording bar-by-bar. Returns a warping path mapping target time → reference time, an alignment-quality score, an overall tempo ratio, and per-warping-step timing deviations in milliseconds. Use when the user asks to compare timing, sync, line up, align measure-by-measure, or check whether their take 'plays in time with' a reference. Requires both audio_url and reference_url. It always returns an alignment path, even for unrelated recordings — check alignment_confidence first; a low score means the path is unreliable, not a small timing offset.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
reference_urlstringYesURL of the reference recording the target should be aligned against.

audio_description_analysis

2 parameters

ZERO-SHOT audio<->text matcher. You supply your OWN free-text labels (text_queries) and it ranks how well the audio matches each, returning per-label similarity scores + the best match. Use for open-vocabulary questions where you define the concept: 'does this match description X?', 'which of these fits best?', mood/genre/scene/vibe/recording-quality, or matching audio to a brief. Works for ANY audio — music, speech, podcasts, field recordings, sound design (not music-only). PREFER over audio_tagging_analysis when the label is custom/abstract/user-worded; use audio_tagging_analysis instead when detecting a KNOWN sound event or instrument from a fixed set. CAVEAT: scores are COMPARATIVE — the best match is only the best of the phrases you give it, so supply a broad, contrasting candidate set (include an 'other/unrelated' option) and treat close or low scores as uncertain. Whole-clip only (no timestamps — crop first to ask about a section). Cannot do pitch/tempo/key/chords/melody/transcription/stems. With no text_queries it returns a 512-dim semantic embedding for similarity search/clustering. Clips over 10 s use a random 10 s window per call, so results vary run-to-run — crop to the relevant section first for stable results.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
text_queriesarray of stringNoText descriptions to compare against audio (e.g., ['calm piano music', 'upbeat electronic'])

audio_effect_analysis

3 parameters

Renders a NEW processed audio file by applying an ORDERED CHAIN of DSP effects — it does NOT analyze or measure (ignore the '..._analysis' name). Use for processing/mastering/effects on ANY audio: music, stems, voice, podcasts, field recordings, sound design. 18 effects: filters (highpass, lowpass, eq, ladder_filter), dynamics (compressor, limiter, gain, clipping), reverb + convolution (cathedral-hall space), modulation (chorus, phaser, delay), pitch_shift (±24 semitones), and lo-fi/codec character (distortion, bitcrush, resample, mp3_compressor, gsm 'telephone/radio' voice). Use distortion and bitcrush whenever the request calls for their character (gritty, overdriven, fuzzy, lo-fi, vintage, broken, aggressive), scaled to how strongly it is asked: distortion (drive_db 0-100, default 25) adds harmonic overdrive, so about 5-15 is gentle warmth, 25 is crunch and 50+ is heavy fuzz, and it makes the sound harsher and brighter, so a lowpass or eq after it tames that; bitcrush (bit_depth 1-32, default 8) adds digital grit by cutting resolution, so 12-16 is subtle, 8 is retro lo-fi and 4-6 is harsh, and it pairs with 'resample' for a lo-fi sound. Both ADD grit and noise; they never remove noise or make a sound cleaner. Chain up to 20 stages in order (typical: filters -> modulation -> time-based -> dynamics). ALWAYS-ON at the end of every job: (a) LUFS loudness-normalization to output.normalize_lufs (default -16; set it explicitly — e.g. -14 streaming, -16 podcast, -23 broadcast) and (b) a -1 dB safety limiter — so NEVER add your own final limiter, and know a lone gain stage will NOT set final loudness (normalize_lufs governs it). Content-agnostic: no need to isolate vocals first. Handles mono/stereo and can process just a clip (start-offset + duration). NOTE: 'pitch' in cannot_do means no pitch DETECTION — pitch_shift for CHANGING pitch IS supported; do not refuse it. Output sample-rate in the output block is a no-op — to reduce rate/go lo-fi, add a 'resample' effect stage. Out-of-range params fail with a clear error. Route 'measure/detect key/tempo/loudness' to an analysis engine, not here. On near-silent input, loudness normalization is silently skipped and won't hit the target — check processed_lufs in the response. To approximate a reference recording's sound, you may have an analysis engine measure the reference so the customer sees its numbers, but the chain is set from the request wording and the step reason, never automatically from those measured values: call the result an approximation and never claim to have copied a measured curve; never claim a match to a reference that was not measured.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
effectsarray of objectYesOrdered chain of effects to apply, in order (typical order: filters -> modulation -> time-based -> dynamics). 1 to 20 stages. Each stage is { type, params, enabled }. A loudness normalization stage and a -1 dB safety limiter are ALWAYS applied at the end automatically, so never add your own final limiter.
outputobjectNoOptional output settings for the rendered file.

audio_tagging_analysis

4 parameters

Primary engine for open-ended 'what sounds are in this audio?' across ANY content — podcasts, voice, field recordings, sound design, mixed/unknown uploads, and music. Returns the most likely labels from a fixed 527-class sound vocabulary (speech, music, instruments, laughter, animals, vehicles, water, wind, machinery, alarms, etc.), each with an independent 0-1 confidence. These are tag PROBABILITIES, not facts — say 'likely/detected', never 'this IS'. Run on the FULL original mix — do NOT isolate vocals or use a stem (that erases the non-music sounds this engine exists to find). Clip-wise, not timestamped: one prediction over the whole file; for 'when does X happen', segment the audio and tag chunks. Multi-label; de-duplicate nested labels. Faint/brief sounds may fall below the default 0.1 threshold — lower confidence_threshold or raise top_k for subtle background sounds. Mono/32kHz internally: no stereo/spatial answers. Not for pitch/key/tempo/melody/chords or precise transcription; not speech transcription (use speech_analysis) and not free-text caption matching (use audio_description_analysis when the user supplies their own descriptions). include_embeddings returns a 2048-dim fingerprint for similarity/dedup/clustering.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
top_kintegerNoNumber of top predictions to return (default: 10, max: 527)
include_embeddingsbooleanNoWhether to include 2048-dim audio embeddings (default: false)
confidence_thresholdnumberNoMinimum confidence threshold for tag inclusion (default: 0.1)

create_video

2 parameters

Final reassembly step for VIDEO inputs: muxes a processed audio track (full mix OR a single stem) back into the ORIGINAL video's frames and returns processed_video_url. Picture is byte-identical (frames never re-encoded — no visual quality loss, fast); audio is re-encoded to AAC (lossy). Use as the LAST step whenever the user uploaded a video and wants a VIDEO back. Pass video_url = the ORIGINAL video (the one extract_video_audio echoed back), audio_url = the processed audio. Bookend to extract_video_audio; full flow: extract_video_audio -> (analysis/effect engines) -> create_video. NO mono/stereo or sample-rate prep needed. CAVEAT — length must match: the mux ends at whichever input is shorter and SILENTLY truncates the rest. If a length-changing effect ran (crop/trim, time-stretch, tempo change) so the audio no longer matches the video's duration, the output is clipped to the shorter one — warn the user the video may be shortened, or avoid length-changing effects when the full video must be preserved. Also: only the first audio and first video stream survive — secondary audio tracks, subtitles, and chapters on the original are dropped. Whole-track REPLACEMENT only — cannot mix/duck the new audio under the original or place it at an offset. Output is MP4 (MOV for ProRes-type sources; input must be MP4/MOV). The returned video's audio is always re-encoded to AAC 128 kbps — lossless/high-quality input is degraded, with no original-codec passthrough.

ParameterTypeRequiredDescription
video_urlstringYesURL of the original video
audio_urlstringYesURL of the processed audio to place into the video

crop_audio

4 parameters

Free, fast server-side TIME-TRIM: cuts one contiguous section from an audio file and returns a new, shorter clip. Use ONLY when the user names a specific time range or section (e.g. 'the chorus 1:00-1:30', 'the first 30s', 'around 2:15') — NEVER crop for whole-file requests. Put crop FIRST in the recipe: every later step then operates on the CROP automatically (it's linked as a child of the original, per the Cropping CRITICAL RULE), so don't repeat the range downstream. Lossless for compressed formats (mp3/flac/ogg/m4a/aac: stream-copy), BUT cuts snap to the nearest ~26ms frame boundary — not sample-exact, so avoid for gapless/tight edits. WAV/AIFF (and other PCM) get re-encoded to 16-bit — warn the user if a high-bit-depth source needs archival precision. One range only: cannot extract multiple segments, stitch, or drop a middle; cannot run in parallel mode. TIME trim only — does NOT remove vocals, isolate an instrument, resample, or clean audio; for 'just the vocals/drums' use element-isolation, and for 'audio from a video' use the video-audio extractor. A start_time at or past end-of-file errors out (no empty clip) — confirm the file's duration before cropping near the end.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
start_timenumberYesstart of the crop, in seconds
end_timenumberNoend of the crop, in seconds (use this OR duration)
durationnumberNolength of the crop, in seconds (use this OR end_time)

embedding_analysis

3 parameters

General-purpose deep audio 'sounds-like' engine — works on any audio (music, voice/podcasts, SFX, ambience, field recordings). Turns a clip into a semantic embedding capturing overall sonic CHARACTER (timbre/texture/room), not notes/words/tempo/key. Uses: (a) single-track embedding for clustering / similarity search / near-duplicate detection; (b) comparison — pass reference_url for a cosine similarity_score (0..1). Comparison collapses each entire track to one average vector, so it answers 'how similar do these sound overall', order-blind: it does NOT tell you WHERE they match and is not exact-match fingerprinting. For bar-by-bar timing/sync alignment use alignment_analysis instead. Present the score as relative similarity, not a hard match/no-match verdict. Set content_type=env for SFX/ambience/non-music (default is music) or results degrade silently. Audio is downmixed to mono, so this can't judge stereo width; very short/near-silent clips give low-confidence scores. For matching audio to a TEXT description, use audio_description_analysis instead.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
content_typestringNomusic (default) or env for non-music audio.
reference_urlstringNoOptional URL of a second audio file to compare against.

extract_video_audio

1 parameter

FIRST step for ANY video input — every other engine reads audio, not video, so extract the audio track first. Input: video_url (MP4/MOV/QuickTime containers only; WebM/MKV/AVI video is rejected). Returns: audio_url (feed into any analysis, transcription, or effect engine), video_url (echoed back so create_video can later re-attach the picture to processed audio), media_probe, and duration_seconds. IMPORTANT — the extracted audio is downmixed to MONO at 22050 Hz: fine for key/chord/tempo, transcription, speech, and general analysis, but it discards stereo imaging and audio above ~11 kHz. Do NOT treat the result as a faithful copy of the soundtrack, and warn the user before using it for stereo-field, mastering/loudness, spatial, or high-fidelity/deliverable work. Limits: hard 60-minute cap (longer videos rejected); if the video has no audio track (e.g. a muted screen recording) extraction fails — tell the user plainly, it is not a system error. CPU-only, fast. Works for any video-sourced audio: music clips, podcasts/interviews/lectures, voiceover, sound design, field recordings. Extraction always takes the first audio track — multi-track videos (dual language, commentary) silently use stream 0 with no way to pick another.

ParameterTypeRequiredDescription
video_urlstringYesURL of the uploaded video (MP4/MOV)

melody_analysis

6 parameters

Polyphonic NOTE transcription: turns pitched audio into discrete musical notes (MIDI pitch + onset/offset + amplitude), a downloadable MIDI file, and a plain-language melody summary (note names, pitch span, notes/sec). Works on any clearly-pitched audio — solo instruments, a hummed/whistled/sung tune, a synth melody. Not continuous vocal-pitch: it snaps to chromatic notes, so for intonation/pitch-bends/'is the singer flat or sharp' use tempo_analysis include_pitch. Detection range ~32-2000 Hz. For best results transcribe a clean, exposed line — on a full mix, isolate the target part first (see stem_analysis / remove_audio_element); dense mixes, percussion, and noise produce muddy or empty output. Fails soft (note_count 0 / 'No notes detected'), so check note_count before asserting a melody. Slow and heavyweight. Not the default for key/chord questions — use spectral_analysis chroma for those. Use only for genuine transcription / MIDI / score work.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
include_melodybooleanNoInclude melody-pitch SUMMARY statistics (mean/std/min/max Hz, confidence, share of confident frames) - whole-file numbers only, NOT a per-frame contour and not chartable over time; for pitch over time use tempo_analysis with include_pitch (default: true)
include_transcriptionbooleanNoInclude note transcription (default: true)
include_midibooleanNoInclude MIDI file generation (default: true)
max_notesintegerNoMaximum number of notes to return in transcription (default: 500)
melody_thresholdnumberNoFrame-confidence cutoff for melody stats, 0-1 (default: 0.5); lower for quiet lines

musical_intelligence_analysis

4 parameters

Two modes for symbolic (MIDI-only) music: harmony (Krumhansl key + chord progression from MIDI — a SECONDARY cross-check; for musical key/chord/harmony on AUDIO use the audio path chroma.cqt instead), midi_diff (per-note diff between two passes — added/removed/timing/pitch deviations; this is the primary use). Both non-audio — inputs come from prior jobs (melody_analysis MIDI/notes). Mode is required. Do not use for audio key/chord/harmony questions — those are read directly from the audio's pitch content. Use only for genuine transcription / MIDI / score work.

ParameterTypeRequiredDescription
modestringYesharmony=Krumhansl key + chord progression from MIDI/note input. midi_diff=per-note diff between two passes.
midi_datastringNoRequired for mode=harmony. Either a base64-encoded MIDI string (e.g. melody_analysis midi_base64) or a JSON-serialized list of {start, end, pitch, velocity} note dicts.
notes_aarrayNoRequired for mode=midi_diff. Reference-pass note list ({start, end, pitch, velocity}).
notes_barrayNoRequired for mode=midi_diff. Comparison-pass note list, same shape as notes_a.

noise_reduction

2 parameters

Remove noise, enhance speech, restore audio. Returns processed audio URL. voice-hifi and voice-enhance are speech-only — on music/non-voice they strip tonal and instrumental content as noise; use voice-lite instead.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
modestringYesvoice-hifi=48kHz best quality (speech only: podcasts, voice; use voice-lite for music). voice-telephony=16kHz for phone/VoIP (loses highs). voice-lite=fast CPU cleanup. voice-enhance=restore degraded/muffled audio.

remove_audio_element

4 parameters

Remove one element from a song (return the whole track minus that element) or isolate it (return only that element) as a SINGLE downloadable file. Use for 'remove the vocals', 'karaoke', 'only remove the bass', 'isolate the drums'. Beyond vocals/drums/bass/other, specific instrument classes — guitar, piano, saxophone, strings, brass, woodwinds, organ, harp, and more — are reachable on this same tool: name the instrument as the target and it routes automatically to a higher-quality separation path. One target per job; to split a song into ALL its stems use stem_analysis. Slower/costlier than a plain split. If a call reports the instrument model is unavailable, that instrument isn't supported on this endpoint right now — it's not a user error and not retryable; fall back to a plain vocals-vs-instrumental split or tell the user that instrument isn't supported yet.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
operationstringYesREQUIRED — always emit operation on the same step as target; there is NO default and a missing operation is rejected, never treated as remove. The user's VERB selects this, the user's NAMED noun selects target — decide them together. isolate/separate/extract/solo/'just the'/'only the' <X> = operation:isolate, target:<X> (return only X). remove/without/karaoke/minus <X> = operation:remove, target:<X> (song minus X). Example: 'separate the bass' = {isolate, bass}; 'remove the vocals' = {remove, vocals}. Never substitute vocals for a different named element.
targetstringYesThe element the user NAMED (vocals, drums, bass, other, or a specific instrument class such as guitar/piano/saxophone/strings/brass, which routes to the higher-quality separation path). Use vocals ONLY when the user asked for an instrumental/karaoke and named no other element — never substitute vocals for a named instrument.
profilestringNoOptional quality/speed profile (default balanced)

rhythm_tracking_analysis

6 parameters

MUSIC-ONLY rhythm + structure engine. One pass returns beat times, downbeats/bar-lines, time signature, BPM, tempo stability + drift curve, and song-form sections (intro/verse/chorus/bridge/outro). Use for beat grids, bar lines, time signature, tempo steadiness/drift, or structure/form on MUSIC. It runs stem separation INTERNALLY, so do NOT pre-remove vocals or pre-separate stems — feed the raw track. Mono/stereo and sample rate are handled automatically. IMPORTANT: this model assumes music with a steady beat; on speech, podcasts, voice notes, field recordings, ambient/drone, or sound-design it will silently INVENT a fake beat grid, meter, and verse/chorus labels — do not run it there, and if unsure the file is music, ask or say so rather than reporting form/meter. HEAVIEST analysis path (stem separation + transformer, GPU): for a plain BPM/tempo-only question prefer tempo_analysis. BPM is tempo_stability.bpm_mean, so if this already ran, reuse it instead of calling tempo_analysis. Caveats to pass the user: time signature is a best-guess limited to common meters (falls back to 'unknown'); a low tempo-stability score can mean an intentional tempo change, not an error; structure and beats are unreliable on very short, quiet, or free-time clips. MP3 input shifts beat/downbeat timestamps ~20–40 ms — use WAV when timing precision matters (sync, beat-grid export). MAX LENGTH: about 6 minutes of audio (about 5.5 at 48 kHz); longer fails with success:false, error_kind=over_duration_limit. If the audio is longer, crop to the section needed first (or warn the user).

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
include_beatsbooleanNoInclude beat times and count (default: true)
include_downbeatsbooleanNoInclude downbeat times and meter estimation (default: true)
include_tempo_stabilitybooleanNoInclude BPM mean/std, stability score, BPM-over-time curve (default: true)
include_structurebooleanNoInclude song structure segments (intro, verse, chorus, bridge, outro). Default: true.
is_excerptbooleanNoSet true when the uploaded audio is an excerpt cut from the middle of a longer track. The opening of a crop is not the track's intro, so the intro refinement is skipped. Default: false.

signal_analysis

6 parameters

Extract mathematical and perceptual features: zero-crossing rate, tonal centroid (tonnetz), normalized chroma statistics, LUFS loudness (integrated, short-term, momentary, LRA, sample/true peak), and optional psychoacoustic quality metrics (roughness, sharpness, time-varying loudness). Use for technical signal analysis and rehearsal dynamics comparison.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
include_zcrbooleanNoInclude zero-crossing rate analysis (default: true)
include_tonnetzbooleanNoInclude tonal centroid features (default: true)
include_chroma_censbooleanNoInclude chroma energy normalized statistics (default: true)
include_loudnessbooleanNoInclude LUFS loudness (BS.1770 + EBU R128): integrated, short-term, momentary, LRA, sample peak, true peak (default: true)
include_psychoacousticbooleanNoInclude psychoacoustic quality metrics — roughness, sharpness, time-varying loudness. Default: false (adds ~60-70 s of cold-start compute). Only enable when the user explicitly asks for perceptual-quality / roughness / sharpness metrics.

spectral_analysis

4 parameters

Broad timbre and spectral feature extractor for music, speech, and mixed audio. Strong for brightness/centroid, rolloff, spectral flux, flatness, RMS/energy, MFCC-family timbre, harmonic/percussive ratio, onset/attack timing, and CQT chroma. Also the audio key/chord path: request the CQT-chroma feature for a musical key or chord/harmony estimate. Caveats: every value is averaged over the WHOLE file (no time-localization); audio is analyzed in mono at 22.05 kHz (no stereo/spatial info, nothing above ~11 kHz); missing features return 0/None (treat as unknown). Not for tempo/beat (use tempo_analysis) or note transcription (use melody_analysis). Request only the categories you need — the full sweep and deep variants are expensive.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
analysis_levelstringNoone of: basic, phase2, phase3, phase4, phase5, phase6, comprehensive. transform_types and scale_types only apply at phase2+.
transform_typesarrayNospectral transforms to run (phase2+ only). Valid values: "st" (STFT), "cqt", "cwt", "vqt", "nsgt", "pwt", "fst", "bft". ⚠️ "mel" is NOT a valid transform_type — use "st" (STFT) or "cqt" instead.
scale_typesarrayNofrequency scales for the transforms (phase2+ only), e.g. "Linear", "Mel", "Bark".

speech_analysis

6 parameters

PRIMARY engine for SPEECH: transcription, subtitles/captions (word-level timestamps), translation to English, and speaker diarization (who-said-what, talk-time). Use for podcasts, interviews, meetings, voicemails, lectures, calls, field/oral-history recordings, and dialogue extracted from video. SPEECH ONLY — it cannot do pitch/tempo/melody/key/chords/MIDI/spectral, and on music/singing/non-speech it will HALLUCINATE words; never send a full song mix here; for sung lyrics isolate the vocals first (see below); key/chord/musical-content questions do not belong here (key & chords -> chroma.cqt; 'what sounds are in this?' -> the sound/event classifier). MONO — stereo/spatial cues are discarded. Modes: transcribe (default) for words + timestamps; transcribe-diarize when who-is-speaking matters; translate / translate-diarize for any language -> English (output always English); diarize-only for a fast 'how many speakers?' with no transcript. Diarization and translation are the expensive paths (translate-diarize heaviest); cost scales with length and model size; transcribe / transcribe-diarize default to model 'turbo' (balances speed/accuracy; large-v3 for max accuracy, tiny/base for drafts) — translate / translate-diarize default to 'medium' instead, because turbo cannot translate (it silently returns the source language, not English); requesting model='turbo' on a translate mode is automatically substituted to 'medium' and flagged in the response (extraction_info.model_substituted=true, requested_model='turbo') rather than silently returning the wrong language. Pass num_speakers when the exact count is known (more accurate than min/max). Caveats to tell the user: diarization can be unavailable and then returns a transcript with NO speaker labels (check speaker_diarization.enabled) — don't promise counts blindly; overlapping speakers, heavy accents, far-field/noisy audio, and very short clips lower accuracy — surface no_speech_prob / avg_logprob rather than presenting an uncertain transcript as fact. include_word_timestamps=false saves ~10-15s when only plain segment text is needed. For a song where the user wants sung words, plan it: isolate the vocals first, then transcribe (never decline it as unsupported). Accuracy on melodic singing is lower than on speech and not measured. Words with numbers, currency, or punctuation (e.g. "5pm", "$13.60") silently lose word-level timestamps — transcribed but unsynced, breaking subtitles and speaker labels there.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
modestringYestranscribe=standard speech-to-text. transcribe-diarize=identify who said what. translate=foreign language to English. translate-diarize=translate + identify speakers. diarize-only=speaker count and timeline only (no transcription, fast).
min_speakersintegerNoMinimum expected speakers (only for *-diarize modes, improves accuracy)
max_speakersintegerNoMaximum expected speakers (only for *-diarize modes, improves accuracy)
include_word_timestampsbooleanNoInclude word-level timestamps via alignment (default: true). Set false to skip alignment and save ~10-15 seconds.
include_full_textbooleanNoCombine segments into full transcript text (default: true). Set false if only segments are needed.

stem_analysis

5 parameters

Multi-stem separation. Splits audio into a fixed set of sources and returns EACH as a separate downloadable file: drums, bass, vocals, other (extended mode adds guitar + piano, lower quality). Use when the user wants ALL or several parts at once (full stem split, remix stems, both acapella and instrumental). If they want just ONE result file — 'remove the vocals', karaoke as a single track, 'isolate the drums' — use remove_audio_element instead, which returns full 44.1 kHz stereo; stem_analysis files are decoded at 22.05 kHz mono and band-limited (nothing above ~11 kHz), good for analysis/preview but not a hi-fi deliverable. Cannot separate arbitrary instruments ('other' is a catch-all); for a specific instrument use remove_audio_element. Music-oriented: not a voice/podcast separator and not a noise cleaner. Expensive and slow — only run when separation is explicitly asked for. Fails soft: absent stems come back near-silent (check presence_ratio). Set stems_to_return to keep only the stems you need.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
modelstringNoSeparation model. Default standard. extended adds guitar + piano stems.
stems_to_returnarray of stringNoOptional subset of stems to keep (e.g., ["vocals"]). Default: all stems. Reduces R2 upload bytes and per-stem FFT CPU. Separation compute is unchanged — the model always produces all stems internally.
include_stem_analysisbooleanNoInclude per-stem FFT analysis (RMS, peak, spectral centroid, frequency-band energy, presence ratio). Default: true. Set false to skip FFT on retained stems and save CPU — useful when you only need the separated audio files.
two_stemsstringNoLegacy 2-stem combine mode: returns the named stem plus an "accompaniment" stem (everything else combined). Mutually exclusive with stems_to_return — if both are set, stems_to_return wins.

tempo_analysis

6 parameters

Fast engine (~3-4s) for three things: (1) global BPM + beat times, (2) tempo steadiness over time, and (3) monophonic fundamental-frequency (F0) pitch tracking. Primary tool for F0/pitch — set include_pitch=true (off by default). F0 is MONOPHONIC: it tracks one pitch at a time, so run it on a single isolated source, never a full mix or chord (for a sung line, isolate the vocal first). F0 works for both speech/voice and single-source music. Default F0 band C2-C7 (65-2093 Hz); narrow pitch_fmin/fmax to the expected range (e.g. ~80-300 Hz for speech) to avoid octave errors. Beat/tempo here is a quick global BPM only — no downbeats, meter, per-section curve, or structure; for those use rhythm_tracking_analysis. BPM is meaningful for music only; on rubato/sparse material it can report half/double time. tempo_stability is 0-1 (higher = steadier). beat_times capped at 2000.

ParameterTypeRequiredDescription
audio_urlstringYesURL of the audio file to process.
include_tempobooleanNoInclude tempo and beat analysis (BPM, beat_count, beat_times) (default: true)
include_tempogrambooleanNoInclude tempogram analysis: dominant_tempo_bpm, tempo_stability, tempo_variance — 2-3 static summary numbers for the whole file, NOT a time-series/curve (default: true). With fewer than 2 usable per-frame tempo estimates, tempo_stability and tempo_variance are omitted and tempo_undetermined=true is returned instead, because one sample has no variance.
include_pitchbooleanNoInclude pYIN F0 pitch estimation - reliable for speech (default: false). Stores the real per-frame pitch contour (time + Hz per frame, unvoiced frames empty) plus summary stats; that contour is what create_chart draws for data_ref category "pitch_contour" (pitch/intonation over time).
pitch_fminintegerNoMinimum pitch frequency in Hz (default: 65 = C2)
pitch_fmaxintegerNoMaximum pitch frequency in Hz (default: 2093 = C7)

get_analysis_result

1 parameter

Retrieve the result (or current status) of a previously submitted analysis job by its job_id. Poll this until status is COMPLETED. Returns the finished analysis payload when done, or a terminal error if the job failed.

ParameterTypeRequiredDescription
job_idstringYesThe job_id returned by a prior tool call that is still running.