API reference
MCP tools
19 tools an MCP client can call. Connect to https://api.scarleta.ai/v1/mcp/rpc with your API key, call tools/list to get this list, then tools/call to run a tool. The request format is on the MCP reference. A tool that runs long answers with a job_id; read the result with get_analysis_result.
Connect your agent
Point your MCP client at the endpoint and pass your API key. Once connected, the tools below appear to your agent next to its other tools.
{
"mcpServers": {
"scarleta": {
"url": "https://api.scarleta.ai/v1/mcp/rpc",
"headers": {
"Authorization": "Bearer $SCARLETA_API_KEY"
}
}
}
}alignment_analysis
2 parameters
alignment_analysisCompare a performance to a reference recording bar-by-bar. Returns a warping path mapping target time → reference time, an alignment-quality score, an overall tempo ratio, and per-warping-step timing deviations in milliseconds. Use when the user asks to compare timing, sync, line up, align measure-by-measure, or check whether their take 'plays in time with' a reference. Requires both audio_url and reference_url. It always returns an alignment path, even for unrelated recordings — check alignment_confidence first; a low score means the path is unreliable, not a small timing offset.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| reference_url | string | Yes | URL of the reference recording the target should be aligned against. |
audio_description_analysis
2 parameters
audio_description_analysisZERO-SHOT audio<->text matcher. You supply your OWN free-text labels (text_queries) and it ranks how well the audio matches each, returning per-label similarity scores + the best match. Use for open-vocabulary questions where you define the concept: 'does this match description X?', 'which of these fits best?', mood/genre/scene/vibe/recording-quality, or matching audio to a brief. Works for ANY audio — music, speech, podcasts, field recordings, sound design (not music-only). PREFER over audio_tagging_analysis when the label is custom/abstract/user-worded; use audio_tagging_analysis instead when detecting a KNOWN sound event or instrument from a fixed set. CAVEAT: scores are COMPARATIVE — the best match is only the best of the phrases you give it, so supply a broad, contrasting candidate set (include an 'other/unrelated' option) and treat close or low scores as uncertain. Whole-clip only (no timestamps — crop first to ask about a section). Cannot do pitch/tempo/key/chords/melody/transcription/stems. With no text_queries it returns a 512-dim semantic embedding for similarity search/clustering. Clips over 10 s use a random 10 s window per call, so results vary run-to-run — crop to the relevant section first for stable results.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| text_queries | array of string | No | Text descriptions to compare against audio (e.g., ['calm piano music', 'upbeat electronic']) |
audio_effect_analysis
3 parameters
audio_effect_analysisRenders a NEW processed audio file by applying an ORDERED CHAIN of DSP effects — it does NOT analyze or measure (ignore the '..._analysis' name). Use for processing/mastering/effects on ANY audio: music, stems, voice, podcasts, field recordings, sound design. 18 effects: filters (highpass, lowpass, eq, ladder_filter), dynamics (compressor, limiter, gain, clipping), reverb + convolution (cathedral-hall space), modulation (chorus, phaser, delay), pitch_shift (±24 semitones), and lo-fi/codec character (distortion, bitcrush, resample, mp3_compressor, gsm 'telephone/radio' voice). Use distortion and bitcrush whenever the request calls for their character (gritty, overdriven, fuzzy, lo-fi, vintage, broken, aggressive), scaled to how strongly it is asked: distortion (drive_db 0-100, default 25) adds harmonic overdrive, so about 5-15 is gentle warmth, 25 is crunch and 50+ is heavy fuzz, and it makes the sound harsher and brighter, so a lowpass or eq after it tames that; bitcrush (bit_depth 1-32, default 8) adds digital grit by cutting resolution, so 12-16 is subtle, 8 is retro lo-fi and 4-6 is harsh, and it pairs with 'resample' for a lo-fi sound. Both ADD grit and noise; they never remove noise or make a sound cleaner. Chain up to 20 stages in order (typical: filters -> modulation -> time-based -> dynamics). ALWAYS-ON at the end of every job: (a) LUFS loudness-normalization to output.normalize_lufs (default -16; set it explicitly — e.g. -14 streaming, -16 podcast, -23 broadcast) and (b) a -1 dB safety limiter — so NEVER add your own final limiter, and know a lone gain stage will NOT set final loudness (normalize_lufs governs it). Content-agnostic: no need to isolate vocals first. Handles mono/stereo and can process just a clip (start-offset + duration). NOTE: 'pitch' in cannot_do means no pitch DETECTION — pitch_shift for CHANGING pitch IS supported; do not refuse it. Output sample-rate in the output block is a no-op — to reduce rate/go lo-fi, add a 'resample' effect stage. Out-of-range params fail with a clear error. Route 'measure/detect key/tempo/loudness' to an analysis engine, not here. On near-silent input, loudness normalization is silently skipped and won't hit the target — check processed_lufs in the response. To approximate a reference recording's sound, you may have an analysis engine measure the reference so the customer sees its numbers, but the chain is set from the request wording and the step reason, never automatically from those measured values: call the result an approximation and never claim to have copied a measured curve; never claim a match to a reference that was not measured.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| effects | array of object | Yes | Ordered chain of effects to apply, in order (typical order: filters -> modulation -> time-based -> dynamics). 1 to 20 stages. Each stage is { type, params, enabled }. A loudness normalization stage and a -1 dB safety limiter are ALWAYS applied at the end automatically, so never add your own final limiter. |
| output | object | No | Optional output settings for the rendered file. |
audio_tagging_analysis
4 parameters
audio_tagging_analysisPrimary engine for open-ended 'what sounds are in this audio?' across ANY content — podcasts, voice, field recordings, sound design, mixed/unknown uploads, and music. Returns the most likely labels from a fixed 527-class sound vocabulary (speech, music, instruments, laughter, animals, vehicles, water, wind, machinery, alarms, etc.), each with an independent 0-1 confidence. These are tag PROBABILITIES, not facts — say 'likely/detected', never 'this IS'. Run on the FULL original mix — do NOT isolate vocals or use a stem (that erases the non-music sounds this engine exists to find). Clip-wise, not timestamped: one prediction over the whole file; for 'when does X happen', segment the audio and tag chunks. Multi-label; de-duplicate nested labels. Faint/brief sounds may fall below the default 0.1 threshold — lower confidence_threshold or raise top_k for subtle background sounds. Mono/32kHz internally: no stereo/spatial answers. Not for pitch/key/tempo/melody/chords or precise transcription; not speech transcription (use speech_analysis) and not free-text caption matching (use audio_description_analysis when the user supplies their own descriptions). include_embeddings returns a 2048-dim fingerprint for similarity/dedup/clustering.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| top_k | integer | No | Number of top predictions to return (default: 10, max: 527) |
| include_embeddings | boolean | No | Whether to include 2048-dim audio embeddings (default: false) |
| confidence_threshold | number | No | Minimum confidence threshold for tag inclusion (default: 0.1) |
create_video
2 parameters
create_videoFinal reassembly step for VIDEO inputs: muxes a processed audio track (full mix OR a single stem) back into the ORIGINAL video's frames and returns processed_video_url. Picture is byte-identical (frames never re-encoded — no visual quality loss, fast); audio is re-encoded to AAC (lossy). Use as the LAST step whenever the user uploaded a video and wants a VIDEO back. Pass video_url = the ORIGINAL video (the one extract_video_audio echoed back), audio_url = the processed audio. Bookend to extract_video_audio; full flow: extract_video_audio -> (analysis/effect engines) -> create_video. NO mono/stereo or sample-rate prep needed. CAVEAT — length must match: the mux ends at whichever input is shorter and SILENTLY truncates the rest. If a length-changing effect ran (crop/trim, time-stretch, tempo change) so the audio no longer matches the video's duration, the output is clipped to the shorter one — warn the user the video may be shortened, or avoid length-changing effects when the full video must be preserved. Also: only the first audio and first video stream survive — secondary audio tracks, subtitles, and chapters on the original are dropped. Whole-track REPLACEMENT only — cannot mix/duck the new audio under the original or place it at an offset. Output is MP4 (MOV for ProRes-type sources; input must be MP4/MOV). The returned video's audio is always re-encoded to AAC 128 kbps — lossless/high-quality input is degraded, with no original-codec passthrough.
| Parameter | Type | Required | Description |
|---|---|---|---|
| video_url | string | Yes | URL of the original video |
| audio_url | string | Yes | URL of the processed audio to place into the video |
crop_audio
4 parameters
crop_audioFree, fast server-side TIME-TRIM: cuts one contiguous section from an audio file and returns a new, shorter clip. Use ONLY when the user names a specific time range or section (e.g. 'the chorus 1:00-1:30', 'the first 30s', 'around 2:15') — NEVER crop for whole-file requests. Put crop FIRST in the recipe: every later step then operates on the CROP automatically (it's linked as a child of the original, per the Cropping CRITICAL RULE), so don't repeat the range downstream. Lossless for compressed formats (mp3/flac/ogg/m4a/aac: stream-copy), BUT cuts snap to the nearest ~26ms frame boundary — not sample-exact, so avoid for gapless/tight edits. WAV/AIFF (and other PCM) get re-encoded to 16-bit — warn the user if a high-bit-depth source needs archival precision. One range only: cannot extract multiple segments, stitch, or drop a middle; cannot run in parallel mode. TIME trim only — does NOT remove vocals, isolate an instrument, resample, or clean audio; for 'just the vocals/drums' use element-isolation, and for 'audio from a video' use the video-audio extractor. A start_time at or past end-of-file errors out (no empty clip) — confirm the file's duration before cropping near the end.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| start_time | number | Yes | start of the crop, in seconds |
| end_time | number | No | end of the crop, in seconds (use this OR duration) |
| duration | number | No | length of the crop, in seconds (use this OR end_time) |
embedding_analysis
3 parameters
embedding_analysisGeneral-purpose deep audio 'sounds-like' engine — works on any audio (music, voice/podcasts, SFX, ambience, field recordings). Turns a clip into a semantic embedding capturing overall sonic CHARACTER (timbre/texture/room), not notes/words/tempo/key. Uses: (a) single-track embedding for clustering / similarity search / near-duplicate detection; (b) comparison — pass reference_url for a cosine similarity_score (0..1). Comparison collapses each entire track to one average vector, so it answers 'how similar do these sound overall', order-blind: it does NOT tell you WHERE they match and is not exact-match fingerprinting. For bar-by-bar timing/sync alignment use alignment_analysis instead. Present the score as relative similarity, not a hard match/no-match verdict. Set content_type=env for SFX/ambience/non-music (default is music) or results degrade silently. Audio is downmixed to mono, so this can't judge stereo width; very short/near-silent clips give low-confidence scores. For matching audio to a TEXT description, use audio_description_analysis instead.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| content_type | string | No | music (default) or env for non-music audio. |
| reference_url | string | No | Optional URL of a second audio file to compare against. |
extract_video_audio
1 parameter
extract_video_audioFIRST step for ANY video input — every other engine reads audio, not video, so extract the audio track first. Input: video_url (MP4/MOV/QuickTime containers only; WebM/MKV/AVI video is rejected). Returns: audio_url (feed into any analysis, transcription, or effect engine), video_url (echoed back so create_video can later re-attach the picture to processed audio), media_probe, and duration_seconds. IMPORTANT — the extracted audio is downmixed to MONO at 22050 Hz: fine for key/chord/tempo, transcription, speech, and general analysis, but it discards stereo imaging and audio above ~11 kHz. Do NOT treat the result as a faithful copy of the soundtrack, and warn the user before using it for stereo-field, mastering/loudness, spatial, or high-fidelity/deliverable work. Limits: hard 60-minute cap (longer videos rejected); if the video has no audio track (e.g. a muted screen recording) extraction fails — tell the user plainly, it is not a system error. CPU-only, fast. Works for any video-sourced audio: music clips, podcasts/interviews/lectures, voiceover, sound design, field recordings. Extraction always takes the first audio track — multi-track videos (dual language, commentary) silently use stream 0 with no way to pick another.
| Parameter | Type | Required | Description |
|---|---|---|---|
| video_url | string | Yes | URL of the uploaded video (MP4/MOV) |
melody_analysis
6 parameters
melody_analysisPolyphonic NOTE transcription: turns pitched audio into discrete musical notes (MIDI pitch + onset/offset + amplitude), a downloadable MIDI file, and a plain-language melody summary (note names, pitch span, notes/sec). Works on any clearly-pitched audio — solo instruments, a hummed/whistled/sung tune, a synth melody. Not continuous vocal-pitch: it snaps to chromatic notes, so for intonation/pitch-bends/'is the singer flat or sharp' use tempo_analysis include_pitch. Detection range ~32-2000 Hz. For best results transcribe a clean, exposed line — on a full mix, isolate the target part first (see stem_analysis / remove_audio_element); dense mixes, percussion, and noise produce muddy or empty output. Fails soft (note_count 0 / 'No notes detected'), so check note_count before asserting a melody. Slow and heavyweight. Not the default for key/chord questions — use spectral_analysis chroma for those. Use only for genuine transcription / MIDI / score work.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| include_melody | boolean | No | Include melody-pitch SUMMARY statistics (mean/std/min/max Hz, confidence, share of confident frames) - whole-file numbers only, NOT a per-frame contour and not chartable over time; for pitch over time use tempo_analysis with include_pitch (default: true) |
| include_transcription | boolean | No | Include note transcription (default: true) |
| include_midi | boolean | No | Include MIDI file generation (default: true) |
| max_notes | integer | No | Maximum number of notes to return in transcription (default: 500) |
| melody_threshold | number | No | Frame-confidence cutoff for melody stats, 0-1 (default: 0.5); lower for quiet lines |
musical_intelligence_analysis
4 parameters
musical_intelligence_analysisTwo modes for symbolic (MIDI-only) music: harmony (Krumhansl key + chord progression from MIDI — a SECONDARY cross-check; for musical key/chord/harmony on AUDIO use the audio path chroma.cqt instead), midi_diff (per-note diff between two passes — added/removed/timing/pitch deviations; this is the primary use). Both non-audio — inputs come from prior jobs (melody_analysis MIDI/notes). Mode is required. Do not use for audio key/chord/harmony questions — those are read directly from the audio's pitch content. Use only for genuine transcription / MIDI / score work.
| Parameter | Type | Required | Description |
|---|---|---|---|
| mode | string | Yes | harmony=Krumhansl key + chord progression from MIDI/note input. midi_diff=per-note diff between two passes. |
| midi_data | string | No | Required for mode=harmony. Either a base64-encoded MIDI string (e.g. melody_analysis midi_base64) or a JSON-serialized list of {start, end, pitch, velocity} note dicts. |
| notes_a | array | No | Required for mode=midi_diff. Reference-pass note list ({start, end, pitch, velocity}). |
| notes_b | array | No | Required for mode=midi_diff. Comparison-pass note list, same shape as notes_a. |
noise_reduction
2 parameters
noise_reductionRemove noise, enhance speech, restore audio. Returns processed audio URL. voice-hifi and voice-enhance are speech-only — on music/non-voice they strip tonal and instrumental content as noise; use voice-lite instead.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| mode | string | Yes | voice-hifi=48kHz best quality (speech only: podcasts, voice; use voice-lite for music). voice-telephony=16kHz for phone/VoIP (loses highs). voice-lite=fast CPU cleanup. voice-enhance=restore degraded/muffled audio. |
remove_audio_element
4 parameters
remove_audio_elementRemove one element from a song (return the whole track minus that element) or isolate it (return only that element) as a SINGLE downloadable file. Use for 'remove the vocals', 'karaoke', 'only remove the bass', 'isolate the drums'. Beyond vocals/drums/bass/other, specific instrument classes — guitar, piano, saxophone, strings, brass, woodwinds, organ, harp, and more — are reachable on this same tool: name the instrument as the target and it routes automatically to a higher-quality separation path. One target per job; to split a song into ALL its stems use stem_analysis. Slower/costlier than a plain split. If a call reports the instrument model is unavailable, that instrument isn't supported on this endpoint right now — it's not a user error and not retryable; fall back to a plain vocals-vs-instrumental split or tell the user that instrument isn't supported yet.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| operation | string | Yes | REQUIRED — always emit operation on the same step as target; there is NO default and a missing operation is rejected, never treated as remove. The user's VERB selects this, the user's NAMED noun selects target — decide them together. isolate/separate/extract/solo/'just the'/'only the' <X> = operation:isolate, target:<X> (return only X). remove/without/karaoke/minus <X> = operation:remove, target:<X> (song minus X). Example: 'separate the bass' = {isolate, bass}; 'remove the vocals' = {remove, vocals}. Never substitute vocals for a different named element. |
| target | string | Yes | The element the user NAMED (vocals, drums, bass, other, or a specific instrument class such as guitar/piano/saxophone/strings/brass, which routes to the higher-quality separation path). Use vocals ONLY when the user asked for an instrumental/karaoke and named no other element — never substitute vocals for a named instrument. |
| profile | string | No | Optional quality/speed profile (default balanced) |
rhythm_tracking_analysis
6 parameters
rhythm_tracking_analysisMUSIC-ONLY rhythm + structure engine. One pass returns beat times, downbeats/bar-lines, time signature, BPM, tempo stability + drift curve, and song-form sections (intro/verse/chorus/bridge/outro). Use for beat grids, bar lines, time signature, tempo steadiness/drift, or structure/form on MUSIC. It runs stem separation INTERNALLY, so do NOT pre-remove vocals or pre-separate stems — feed the raw track. Mono/stereo and sample rate are handled automatically. IMPORTANT: this model assumes music with a steady beat; on speech, podcasts, voice notes, field recordings, ambient/drone, or sound-design it will silently INVENT a fake beat grid, meter, and verse/chorus labels — do not run it there, and if unsure the file is music, ask or say so rather than reporting form/meter. HEAVIEST analysis path (stem separation + transformer, GPU): for a plain BPM/tempo-only question prefer tempo_analysis. BPM is tempo_stability.bpm_mean, so if this already ran, reuse it instead of calling tempo_analysis. Caveats to pass the user: time signature is a best-guess limited to common meters (falls back to 'unknown'); a low tempo-stability score can mean an intentional tempo change, not an error; structure and beats are unreliable on very short, quiet, or free-time clips. MP3 input shifts beat/downbeat timestamps ~20–40 ms — use WAV when timing precision matters (sync, beat-grid export). MAX LENGTH: about 6 minutes of audio (about 5.5 at 48 kHz); longer fails with success:false, error_kind=over_duration_limit. If the audio is longer, crop to the section needed first (or warn the user).
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| include_beats | boolean | No | Include beat times and count (default: true) |
| include_downbeats | boolean | No | Include downbeat times and meter estimation (default: true) |
| include_tempo_stability | boolean | No | Include BPM mean/std, stability score, BPM-over-time curve (default: true) |
| include_structure | boolean | No | Include song structure segments (intro, verse, chorus, bridge, outro). Default: true. |
| is_excerpt | boolean | No | Set true when the uploaded audio is an excerpt cut from the middle of a longer track. The opening of a crop is not the track's intro, so the intro refinement is skipped. Default: false. |
signal_analysis
6 parameters
signal_analysisExtract mathematical and perceptual features: zero-crossing rate, tonal centroid (tonnetz), normalized chroma statistics, LUFS loudness (integrated, short-term, momentary, LRA, sample/true peak), and optional psychoacoustic quality metrics (roughness, sharpness, time-varying loudness). Use for technical signal analysis and rehearsal dynamics comparison.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| include_zcr | boolean | No | Include zero-crossing rate analysis (default: true) |
| include_tonnetz | boolean | No | Include tonal centroid features (default: true) |
| include_chroma_cens | boolean | No | Include chroma energy normalized statistics (default: true) |
| include_loudness | boolean | No | Include LUFS loudness (BS.1770 + EBU R128): integrated, short-term, momentary, LRA, sample peak, true peak (default: true) |
| include_psychoacoustic | boolean | No | Include psychoacoustic quality metrics — roughness, sharpness, time-varying loudness. Default: false (adds ~60-70 s of cold-start compute). Only enable when the user explicitly asks for perceptual-quality / roughness / sharpness metrics. |
spectral_analysis
4 parameters
spectral_analysisBroad timbre and spectral feature extractor for music, speech, and mixed audio. Strong for brightness/centroid, rolloff, spectral flux, flatness, RMS/energy, MFCC-family timbre, harmonic/percussive ratio, onset/attack timing, and CQT chroma. Also the audio key/chord path: request the CQT-chroma feature for a musical key or chord/harmony estimate. Caveats: every value is averaged over the WHOLE file (no time-localization); audio is analyzed in mono at 22.05 kHz (no stereo/spatial info, nothing above ~11 kHz); missing features return 0/None (treat as unknown). Not for tempo/beat (use tempo_analysis) or note transcription (use melody_analysis). Request only the categories you need — the full sweep and deep variants are expensive.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| analysis_level | string | No | one of: basic, phase2, phase3, phase4, phase5, phase6, comprehensive. transform_types and scale_types only apply at phase2+. |
| transform_types | array | No | spectral transforms to run (phase2+ only). Valid values: "st" (STFT), "cqt", "cwt", "vqt", "nsgt", "pwt", "fst", "bft". ⚠️ "mel" is NOT a valid transform_type — use "st" (STFT) or "cqt" instead. |
| scale_types | array | No | frequency scales for the transforms (phase2+ only), e.g. "Linear", "Mel", "Bark". |
speech_analysis
6 parameters
speech_analysisPRIMARY engine for SPEECH: transcription, subtitles/captions (word-level timestamps), translation to English, and speaker diarization (who-said-what, talk-time). Use for podcasts, interviews, meetings, voicemails, lectures, calls, field/oral-history recordings, and dialogue extracted from video. SPEECH ONLY — it cannot do pitch/tempo/melody/key/chords/MIDI/spectral, and on music/singing/non-speech it will HALLUCINATE words; never send a full song mix here; for sung lyrics isolate the vocals first (see below); key/chord/musical-content questions do not belong here (key & chords -> chroma.cqt; 'what sounds are in this?' -> the sound/event classifier). MONO — stereo/spatial cues are discarded. Modes: transcribe (default) for words + timestamps; transcribe-diarize when who-is-speaking matters; translate / translate-diarize for any language -> English (output always English); diarize-only for a fast 'how many speakers?' with no transcript. Diarization and translation are the expensive paths (translate-diarize heaviest); cost scales with length and model size; transcribe / transcribe-diarize default to model 'turbo' (balances speed/accuracy; large-v3 for max accuracy, tiny/base for drafts) — translate / translate-diarize default to 'medium' instead, because turbo cannot translate (it silently returns the source language, not English); requesting model='turbo' on a translate mode is automatically substituted to 'medium' and flagged in the response (extraction_info.model_substituted=true, requested_model='turbo') rather than silently returning the wrong language. Pass num_speakers when the exact count is known (more accurate than min/max). Caveats to tell the user: diarization can be unavailable and then returns a transcript with NO speaker labels (check speaker_diarization.enabled) — don't promise counts blindly; overlapping speakers, heavy accents, far-field/noisy audio, and very short clips lower accuracy — surface no_speech_prob / avg_logprob rather than presenting an uncertain transcript as fact. include_word_timestamps=false saves ~10-15s when only plain segment text is needed. For a song where the user wants sung words, plan it: isolate the vocals first, then transcribe (never decline it as unsupported). Accuracy on melodic singing is lower than on speech and not measured. Words with numbers, currency, or punctuation (e.g. "5pm", "$13.60") silently lose word-level timestamps — transcribed but unsynced, breaking subtitles and speaker labels there.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| mode | string | Yes | transcribe=standard speech-to-text. transcribe-diarize=identify who said what. translate=foreign language to English. translate-diarize=translate + identify speakers. diarize-only=speaker count and timeline only (no transcription, fast). |
| min_speakers | integer | No | Minimum expected speakers (only for *-diarize modes, improves accuracy) |
| max_speakers | integer | No | Maximum expected speakers (only for *-diarize modes, improves accuracy) |
| include_word_timestamps | boolean | No | Include word-level timestamps via alignment (default: true). Set false to skip alignment and save ~10-15 seconds. |
| include_full_text | boolean | No | Combine segments into full transcript text (default: true). Set false if only segments are needed. |
stem_analysis
5 parameters
stem_analysisMulti-stem separation. Splits audio into a fixed set of sources and returns EACH as a separate downloadable file: drums, bass, vocals, other (extended mode adds guitar + piano, lower quality). Use when the user wants ALL or several parts at once (full stem split, remix stems, both acapella and instrumental). If they want just ONE result file — 'remove the vocals', karaoke as a single track, 'isolate the drums' — use remove_audio_element instead, which returns full 44.1 kHz stereo; stem_analysis files are decoded at 22.05 kHz mono and band-limited (nothing above ~11 kHz), good for analysis/preview but not a hi-fi deliverable. Cannot separate arbitrary instruments ('other' is a catch-all); for a specific instrument use remove_audio_element. Music-oriented: not a voice/podcast separator and not a noise cleaner. Expensive and slow — only run when separation is explicitly asked for. Fails soft: absent stems come back near-silent (check presence_ratio). Set stems_to_return to keep only the stems you need.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| model | string | No | Separation model. Default standard. extended adds guitar + piano stems. |
| stems_to_return | array of string | No | Optional subset of stems to keep (e.g., ["vocals"]). Default: all stems. Reduces R2 upload bytes and per-stem FFT CPU. Separation compute is unchanged — the model always produces all stems internally. |
| include_stem_analysis | boolean | No | Include per-stem FFT analysis (RMS, peak, spectral centroid, frequency-band energy, presence ratio). Default: true. Set false to skip FFT on retained stems and save CPU — useful when you only need the separated audio files. |
| two_stems | string | No | Legacy 2-stem combine mode: returns the named stem plus an "accompaniment" stem (everything else combined). Mutually exclusive with stems_to_return — if both are set, stems_to_return wins. |
tempo_analysis
6 parameters
tempo_analysisFast engine (~3-4s) for three things: (1) global BPM + beat times, (2) tempo steadiness over time, and (3) monophonic fundamental-frequency (F0) pitch tracking. Primary tool for F0/pitch — set include_pitch=true (off by default). F0 is MONOPHONIC: it tracks one pitch at a time, so run it on a single isolated source, never a full mix or chord (for a sung line, isolate the vocal first). F0 works for both speech/voice and single-source music. Default F0 band C2-C7 (65-2093 Hz); narrow pitch_fmin/fmax to the expected range (e.g. ~80-300 Hz for speech) to avoid octave errors. Beat/tempo here is a quick global BPM only — no downbeats, meter, per-section curve, or structure; for those use rhythm_tracking_analysis. BPM is meaningful for music only; on rubato/sparse material it can report half/double time. tempo_stability is 0-1 (higher = steadier). beat_times capped at 2000.
| Parameter | Type | Required | Description |
|---|---|---|---|
| audio_url | string | Yes | URL of the audio file to process. |
| include_tempo | boolean | No | Include tempo and beat analysis (BPM, beat_count, beat_times) (default: true) |
| include_tempogram | boolean | No | Include tempogram analysis: dominant_tempo_bpm, tempo_stability, tempo_variance — 2-3 static summary numbers for the whole file, NOT a time-series/curve (default: true). With fewer than 2 usable per-frame tempo estimates, tempo_stability and tempo_variance are omitted and tempo_undetermined=true is returned instead, because one sample has no variance. |
| include_pitch | boolean | No | Include pYIN F0 pitch estimation - reliable for speech (default: false). Stores the real per-frame pitch contour (time + Hz per frame, unvoiced frames empty) plus summary stats; that contour is what create_chart draws for data_ref category "pitch_contour" (pitch/intonation over time). |
| pitch_fmin | integer | No | Minimum pitch frequency in Hz (default: 65 = C2) |
| pitch_fmax | integer | No | Maximum pitch frequency in Hz (default: 2093 = C7) |
get_analysis_result
1 parameter
get_analysis_resultRetrieve the result (or current status) of a previously submitted analysis job by its job_id. Poll this until status is COMPLETED. Returns the finished analysis payload when done, or a terminal error if the job failed.
| Parameter | Type | Required | Description |
|---|---|---|---|
| job_id | string | Yes | The job_id returned by a prior tool call that is still running. |