Building on-device Video Notes in a macOS app

Short summary

A macOS video player implements on-device Video Notes by chaining libmpv (audio extraction), SpeechAnalyzer (transcription), Translation framework (optional EN↔JA), and Foundation Models (summarization)—all locally, no cloud, no API keys. The post details production shipping challenges: guardrail false positives on normal speech, context-window overflow on long videos, and SwiftUI's constraint on translation API placement.

• •On-device transcription + translation + summarization pipeline for local video files using Apple's macOS APIs
• •Production workarounds for Foundation Models guardrails false-positives and context-window overflow via chunking
• •Architectural asymmetry: Translation framework only works through SwiftUI, unlike other service-layer APIs

Generated with AI, which can make mistakes.