The result

GET /v1/analyses/:id/result returns one JSON document: the breakdown of the video. This page says what every field holds. The Example page shows a whole one.

Conventions

Times are seconds from the start of the video, as numbers. s is a start and e an end; t is an instant.
Boxes are { "x", "y", "w", "h" } as fractions of the frame from its top-left corner: x and y are the box's top-left corner, w and h its size. They hold at any resolution.
Numbers n and id count from 1 in time order.
Empty means found nothing. An empty list was analyzed and has nothing in it. transcript is null only when the video has no audio track.
Nothing is an opinion. The result says what is there and when, never whether it is good or what to do with it.

The envelope

Field	Type	What it holds
`format`	string	Always `video-analysis`.
`version`	string	The version of this document, `1`. It changes when a field changes meaning or goes away; new fields can appear without a new version.
`generator`	string	What wrote it.
`made_at`	string	When, ISO 8601 in UTC.
`methods`	object	How each part was made, in words: the tool or the model, and its settings. Keys are `transcript`, `cuts`, `boundaries`, `elements`, `characters`.

video

What the file is, from the container.

Field	Type	What it holds
`name`	string	The file name you gave.
`bytes`	integer	Its size.
`container`	string	The container, as ffprobe names it.
`duration`	number	Seconds.
`width`, `height`	integer	Pixels, as the video plays (rotation applied).
`fps`	number	Frames per second.
`codec`	string	The video codec.
`audio`	boolean	Whether there is an audio track.

transcript

What is said, timed by a speech model. null when the video has no audio track; present with empty lists when there is sound but no speech.

Field	Type	What it holds
`language`	string	The detected language, as a BCP 47 tag. Absent when there are no words.
`seconds`	number	The length of the audio that was transcribed.
`text`	string	Everything said, as one string.
`words`	list	Every word in order: `w` the word with its punctuation, `s` and `e` its start and end, `c` the model's confidence from 0 to 1, `spk` the speaker's number from 0.
`utterances`	list	The sentences the words make: `s`, `e`, `spk`, `text`.

cuts

The boundaries that count, in time order: each one ends a shot and starts the next.

Field	Type	What it holds
`t`	number	When it happens.
`frame`	integer	The frame it lands on.
`kind`	string	What it is. See the table below.
`end`	number	For a `zoom`: when the move ends.
`factor`	number	For a `punch-in`, a `punch-out` or a `zoom`: the change of scale. Over 1 is closer.
`firings`	integer	For a `zoom`: how many times the detector fired during it.

`kind`	Meaning
`cut`	A different picture: almost nothing on one side matches the other. Starts a new scene.
`jump`	The same setup, and the content changed: a jump cut. Stays in the scene.
`punch-in`	The same setup at a larger scale: the picture was pushed in. Stays in the scene.
`punch-out`	The same at a smaller scale. Stays in the scene.
`zoom`	An animated zoom across several frames. Stays in the scene.

boundaries

Every firing of the cut detector, including the ones that turned out to be nothing, each with the measurements that decided its kind. Use cuts unless you want to judge the boundaries yourself.

A boundary has the fields of a cut, plus match: inliers (how many points of the picture matched across the boundary), spread (how much of the frame those points cover), scale (the change of scale between the two sides), and diff (how much the two sides differ once aligned). kind can also be none: the detector fired on a flash or a fast move, and nothing was cut. Those are left out of cuts.

shots and scenes

The video divided at the cuts. A shot runs from one cut to the next; a scene runs from one hard cut to the next and holds one or more shots.

Field	Type	What it holds
`shots[].n`	integer	The shot's number.
`shots[].s`, `shots[].e`	number	Its start and end. Shots are contiguous: each starts where the last ended.
`shots[].scene`	integer	The scene it belongs to.
`shots[].kind`	string	How it begins: `start` for the first shot, otherwise the kind of the cut before it.
`scenes[].n`	integer	The scene's number.
`scenes[].s`, `scenes[].e`	number	Its start and end. Scenes are contiguous.

elements

Everything that was put on the picture in the edit, as opposed to filmed. A vision model reads the video four times a second and names each element; runs of the same element across frames are joined into one; then the pixels inside its box put its start and end on the exact frame.

Field	Type	What it holds
`id`	integer	Its number, in order of appearance.
`kind`	string	What it is. See the table below.
`text`	string	For text: exactly what is written, with its case and line breaks. For a picture or a graphic: what it shows, in a few words. For a caption revealed word by word: the whole line.
`description`	string	What it looks like and what it shows, in a sentence or two, and how it came on if that was visible. For a clip, a new sentence each time what it shows changes, prefixed with the time.
`s`, `e`	number	When it is on the picture.
`box`	box	Where it sits. For text, the whole line; for a picture, its edges.
`refined`	object	`s` and `e` as booleans: `true` when that edge was put on the frame from the pixels, `false` when it stands at the sampled time (accurate to a quarter of a second).
`sampled`	object	`s` and `e` as the model saw them, before the pixels refined them.
`frames`	integer	On how many of the sampled frames it was seen.

`kind`	Meaning
`caption`	The words being spoken, shown as text.
`title`	Added text that is not the spoken words: a headline, a claim, a label.
`lower third`	A name or role strip.
`image`	A photo or picture placed over the footage.
`clip`	A video placed over the footage, or a picture-in-picture.
`logo`, `sticker`, `emoji`, `arrow`, `price`, `blur`	As named.
`highlight`	A circle, box or underline drawn to point at something.
`progress bar`	A bar or ring that fills as time passes.
`button`	A call to action drawn as a button.
`scene text`	Text that is part of the filmed world: a sign, a label, a screen.
`other`	Anything else added.

characters

Who the video is about, found from two or three frames of every scene.

characters.cast lists the individuals filmed as subjects. A member stays in the cast only when it appears in two or more scenes or is on screen for a quarter of the video; hands, body parts and anyone seen once are extras. characters.rule records those thresholds.

Field	Type	What it holds
`cast[].label`	string	`A`, `B`, `C`… The same letter is the same individual in every scene, whatever they wear and however they are framed.
`cast[].name`	string	A short name to call them by, from how they look.
`cast[].kind`	string	`person` or `animal`.
`cast[].role`	string	`presenter` (speaks to camera or carries the video), `pet` (the animal the video is about), or `companion` (recurs beside them).
`cast[].description`	string	Enough to know them again.
`cast[].scenes`	list	The scenes they were seen in.
`cast[].crops`	list	One crop per appearance: `scene`, `t`, `box` (where they are in that frame), `sure` (`false` when the model was not certain it is them), `file` and `url`. Sorted with the sure and the largest first.
`cast[].portrait`, `cast[].portrait_url`	string	The best crop: the first of `crops`.
`extras[]`	list	Who was left out of the cast and `why`, with `label`, `name`, `kind`, `role` and the `scenes` they were seen in.
`scenes[]`	list	Per scene: `n`, `s`, `e`, `cast` (the labels present), `extras` (how many extras), and `frames`, the frames that were looked at, each with its `t` and who is `present` with their `box`, `sure`, `crop` and `url`.

The crops

Every crop is a JPEG cut from the video at its own resolution, with a margin around the box. file and crop are the crop's name inside the result; url and portrait_url are signed links to the picture. The links last an hour: fetch the pictures you want to keep, or ask for the result again to get fresh links. The pictures themselves are kept as long as the result.

The text format

?format=text returns the same facts as a document: the video's facts, the cast, then scene by scene the cast present, the shots, what is said and what is on the picture. It carries no links and none of the measurements in boundaries. Use it to hand the breakdown to a model as reading; use the JSON for anything a program does.