PERCEPT

The perception runtime for Audio. Video. Agent

Program whatdeserves attention.

Percept turns continuous multimodal streams into sparse, actionable events. Describe what matters. Percept operates the watch and tells your application when it happens.

Why now Multimodal judgment is finally good enough to trust on a live stream, and far too costly to run on every frame. The runtime closes that gap.

Try the Playground Install the SDK

LiveStream+Watch (Goal + Policy)stream<PerceptEvent>

Percept runtime · porch camera · 1280×720 · 24 fps00:10.0

Standing watch

"Tell me when a package has been left at the door."

FIRE

One inspectable event is emitted.

The application receives evidence and provenance, then decides what action to take.

{
  "gate": "FIRE",
  "incident": "inc_pkg_001",
  "closed_at": "00:08.2"
}

240frames

142admitted

1judge call

1event

01 · The product and the problem

A frame can answer a question. A product has to operate a watch.

Models can perceive. Applications can act. Percept runs the continuous loop between them: deciding when evidence is useful, what moments belong together, when a judgment is needed, and whether one inspectable event should be emitted.

Judge only after evidence becomes useful.

Cheap attention signals and a bounded episode turn a burst of frame changes into one model call and one de-duplicated event.

1judge call

1logical event

boundedpending work

02 · The continuous model

From standing intent to one useful interruption.

The animation is illustrative: it explains the runtime through causality rather than decoration. A continuous stream is filtered, assembled into an event capsule, judged once, then routed to FIRE, ASK, or SILENT.

Plate 01

Continuous stream, one selected event.

Frames arrive without pause. Most pass through quietly. Attention selects a useful moment and seals it into one bounded episode.

01 · frames translate continuously02 · attention closes on one moment03 · evidence lifts into an episode04 · only one capsule reaches judgment

Runtime boundary

Percept operates the standing watch.

Streams and models remain replaceable inputs. Percept owns attention, episode construction, bounded judgment, context, decision semantics, replay, and event delivery. Your application still owns the action.

03 · Who it is for

A live stream. A sparse event. A real cost of missing it, or firing falsely.

The best early workloads are continuously observable, operationally meaningful, and already connected to an application that can use a reliable event.

Screen procedures

Know when a task succeeds, stalls, or diverges.

Developer assistance, guided setup, compliance, and workflow completion.

"Tell me when installation succeeds or the user becomes blocked."

Physical operations

Observe meaningful process change over time.

Inspections, manufacturing, field work, facilities, and security operations.

"Emit when the inspection step completes or deviates from plan."

Ambient assistants

Give agents a reason to wake up.

Watch voice, vision, screen, and environmental streams without reacting continuously.

"Nudge me if I leave without my purse."

Continuous safety

Represent absence, persistence, and escalation.

Express concerns that are not reducible to one object detector or isolated frame.

"Alert when the person has not moved for the configured interval."

Live interviews

Start the workflow at the meaningful moment.

Interviews, walkthroughs, support sessions, and real-time collaboration.

"Emit when the customer has completed the intake explanation."

Product infrastructure

Standardize perception across applications.

One event contract, one replay surface, one evidence reference, and one runtime boundary.

watch(stream, goal) → WatchEvent

04 · OSS · SDK

The smallest stable surface from a stream to an event.

Install the runtime, describe a watch in plain language, connect a source, and subscribe to typed events. The core is pure standard library, and it runs offline on deterministic fakes with no API keys. The source is open.

pip install percept-vision

Plain-language watches

Author a WatchPlan in Python, or pass a bare string. The condition travels verbatim to the judge; it is never compiled to a rigid predicate.

Fail-closed judges

A requested judge is authorized, bound to a hash-pinned context, then dispatched, or the arm refuses. Never a silent fallback.

Vendor-neutral backends

Anthropic, Gemini, OpenRouter, and Deepgram sit behind one seam. Deterministic fakes run offline with no API keys.

Optimizer

Pre-registered evaluation plus a mechanical search over operator-declared safe knobs. The winner ships as an OptimizationLock.

import percept

# stream: any async iterable of frames: a camera, a file, an RTSP feed
# one WatchEvent per gate FIRE or ASK; silent frames yield nothing
async for event in percept.watch(stream, "the kettle boils"):
    handle(event)

# or the batteries-included front door: watch a source, receive Incidents
async for incident in percept.run(
    source="kitchen.mp4",               # camera index, file, or RTSP
    goals=["the milk is boiling over"],
    fps=4.0,
):
    notify(incident)                    # a durable Incident, deduped on redelivery

Quickstart →API reference →Hosted API status →

What you can configure

goal + policy
evidencevision · audio · fused
source profilephysical scene · screen
executionbalanced · burst · low-cost · privacy-edge
audio window

OSS · the runtimeRuns offline. No keys.

The runtime itself. `pip install percept-vision` and it runs offline against deterministic fakes. No API key, no account, no network required.

HOSTED · managed runsManaged runs, live traces.

Managed runs over the pinned, provenance-stamped 0.8.3 wheel: provisioned API keys, media uploads, live trace streaming, and the Playground.

05 · Playground

Try a simple watch. Then inspect the entire system.

The preview below is an illustration of the flow. The real Playground runs the actual OSS wheel against your media or a hosted sample. A quiet run is an answer, not a failure.

Percept Playground · interactive preview

Natural-language watch target

Ready to arm

Plain language → WatchPlan → armed watch

from percept.contracts.watchplan import WatchPlan

plan = WatchPlan(
    # plain language, carried verbatim to the judge
    condition=(
        "The user successfully completes installation and starts Claude Code without an unresolved error."
    ),
)

# percept.watch(stream, plan) arms it and yields WatchEvents

SILENT · subscriber waiting

Run the watch to produce an event.

The preview will expose arming, admission, episode assembly, one judge call, and one delivered event.

Preview: sample screen and video watches · Full: real runs on the provenance-stamped OSS wheel.Open full Playground →

06 · Benchmarks

Benchmark the runtime, not just the model.

A model can classify a clip and still fail in a continuous product. Percept's evaluation rail is pre-registered and offline: bars are declared before numbers exist, and silence is never a pass.

Illustrative curves only. No published numbers exist yet. These panels explain the measurement contract; published results will remain tied to locked configurations and reproducible artifacts.

Does the runtime remain bounded under overload?

Drive input faster than the judge. Pass only when pending depth and oldest age remain bounded, supersession is explicit, memory is flat, and stale completions cannot mutate the gate.

oldest pending agememory slopestale completionsdelivery isolation

Design partners

Bring us a stream and the event that matters.

We are working with teams that already have a continuous perception problem and a real cost of missing the event, or firing falsely.

Become a design partner Join the Discord