Multi-Modal Audio & Video Synthesis: Timestamped Chunking & LaTeX Math Extraction
How NorAI built the Course Note-Taker ingestion engine to parse 2-hour university lectures into timestamped summaries, definition glossaries, and clean LaTeX mathematical formula cards.
Sonu Singh
Head of Developer Tooling
The Challenge of Long-Form Academic Audio
University lectures and technical webinars contain dense spoken formulas, fast transitions between whiteboard proofs, and colloquial professor explanations. Standard speech-to-text models produce continuous walls of unformatted text without logical chapter breaks.
Extracting usable study notes requires three synchronized processing stages: acoustic silence-aware segmentation, phonetic speech-to-text with specialized mathematical vocabulary prompting, and post-processing mathematical normalization into LaTeX syntax.
LaTeX Formula Extraction & KaTeX Rendering
When a speaker says "the integral from zero to infinity of e to the minus x squared dx equals square root pi over two," standard transcription outputs broken text. Our pipeline converts spoken mathematical phrasing into standard LaTeX notation:
$$\int_{0}^{\infty} e^{-x^2} \, dx = \frac{\sqrt{\pi}}{2}$$
We parse the output directly with KaTeX at compile time for zero client-side layout shift, formatting theorems into structured study cards with clickable timestamps.
export interface LectureStudyCard {
timestamp: string; // "14:22"
timestampSeconds: number; // 862
sectionTitle: string;
summary: string;
latexFormulas: string[];
keyDefinitions: Array<{ term: string; explanation: string }>;
quizQuestion: {
question: string;
options: string[];
correctIndex: number;
explanation: string;
};
}Course Note-Taker
Turn hours of video lectures and audio into structured study notes and flashcards.