[0.1.8] Video capture phase 5: sound, from the computer and the microphone, in sync
- modules/Screen_Record/audio_source.jai: WASAPI, in loopback from the default output device (what the computer plays) and from the default input device (the microphone), asked for as 48 kHz stereo float, which Windows converts any device to; a device that will not convert is taken as it is when it gives 48 kHz float. Packets carry the time they were heard. A device lost (unplugged, default changed) is opened again from the new default once a second. - An audio thread mixes them onto the video's own timeline: the same time 0, less the same pauses (record_pause now keeps one account of time paused for both threads). A packet within 30 ms of where its source left off follows on exactly; a source that sends nothing is silence, passed 200 ms late. Each 1024 samples every source has covered become an AAC frame. - encoder.jai: an AAC stream (FFmpeg's aac, 160 kb/s) beside the video, written from the audio thread; both threads' packets meet in the file under a lock. The remux keeps it. - [video] system_audio (true) and microphone (false). capture.json: Audio, AudioSources. A toast when a source asked for could not be opened. - record-probe: -audio system|mic|both, the sound read back (length, level), and -sync-test, which flashes a white square and beeps together every 2 s, then finds both in the file. - Measured: 20 s, beeps 0-21 ms from their flashes, no drift; both sources mixed, 5-18 ms; a 6 s pause cuts the same 6 s from sound and video; 3 min, 85 of 90 beeps within 3-24 ms and -14 ms from first to last, with 5 outliers (53-147 ms) where two beeps went missing. Tests: an MP4 with sound keeps its AAC stream through the remux into the library (30 checks).
This commit is contained in:
parent
e0eaf5642b
commit
72f2754226
12
README.md
12
README.md
@ -187,7 +187,9 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S
|
||||
go of it starts nothing; drag again.
|
||||
- Escape or a right click cancels.
|
||||
3. **Recording starts** at once, at up to 30 frames a second, cursor included. A frame is written
|
||||
only when the screen changes, so a still screen costs almost nothing.
|
||||
only when the screen changes, so a still screen costs almost nothing. **Sound:** what the
|
||||
computer plays is recorded with it, and the microphone too if `microphone` is set in `[video]`
|
||||
(off by default).
|
||||
- **A red border** is drawn just outside the region.
|
||||
- **A toolbar** beside it shows the time, the size and any frames dropped, with **Pause** (the time
|
||||
paused is left out of the video), **Stop**, and **Discard**, which asks first.
|
||||
@ -206,8 +208,12 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S
|
||||
recording carries on once it is back.
|
||||
- A recording a crash cut short is offered at the next start: into the library, or deleted. It
|
||||
keeps all but its last two seconds or so.
|
||||
- **Settings:** `[video]` in the settings file: `fps` (30), `cursor` (true), and `capture`: `auto`
|
||||
(Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`.
|
||||
- **Settings:** `[video]` in the settings file:
|
||||
- `fps` (30) and `cursor` (true).
|
||||
- `capture`: `auto` (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`.
|
||||
- `system_audio` (true): what the computer plays, from the default output device.
|
||||
- `microphone` (false): the default input device.
|
||||
- Sound is AAC at 160 kb/s. A device unplugged mid-recording is picked up again from the new default.
|
||||
- **PrintScreen while recording** takes the whole screen under the cursor at once, without freezing
|
||||
the screens, which would show in the recording.
|
||||
- **The key** is `record_region` in `[global_hotkeys]`.
|
||||
|
||||
@ -29,7 +29,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
GIT_BRANCH :: "check";
|
||||
GIT_HASH :: "-";
|
||||
VERSION :: "0.1.7";
|
||||
VERSION :: "0.1.8";
|
||||
BUILD_DATE :: "-";
|
||||
DEBUG :: true;
|
||||
COMPILE_TESTS :: true;
|
||||
|
||||
@ -8,7 +8,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
|
||||
|
||||
VERSION :: "0.1.7";
|
||||
VERSION :: "0.1.8";
|
||||
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
|
||||
// builds never write over the ones you run by hand.
|
||||
build_directory := "bin";
|
||||
|
||||
@ -136,6 +136,9 @@ Capture: "DXGI DuplicateOutput1 on NVIDIA GeForce RTX 5080 Laptop GPU"
|
||||
or "Windows.Graphics.Capture on ..., border off"
|
||||
DroppedFrames: 0 frames the encoder fell too far behind for
|
||||
Paused: 10.2 seconds paused, left out of the video; only when it was paused
|
||||
Audio: true whether it has sound (AAC)
|
||||
AudioSources: "SAMSUNG (NVIDIA High Definition Audio); Microphone Array (Realtek(R) Audio)"
|
||||
the devices heard; only when it has sound
|
||||
```
|
||||
|
||||
A recording a crash cut short, put in the library at the next start, has no window, Encoder or
|
||||
|
||||
263
modules/Screen_Record/audio_source.jai
Normal file
263
modules/Screen_Record/audio_source.jai
Normal file
@ -0,0 +1,263 @@
|
||||
// Sound for a recording, with WASAPI: what the computer plays (the default output device, in loopback)
|
||||
// and the microphone (the default input device). Each is asked for in shared mode as 48 kHz stereo
|
||||
// float, which Windows converts any device's own format to (AUTOCONVERTPCM), so nothing here
|
||||
// resamples. Where a device will not convert, its own format is taken if it is 48 kHz float, and its
|
||||
// channels are made stereo here.
|
||||
//
|
||||
// Each packet comes with the QueryPerformanceCounter time its first sample was heard, which places it
|
||||
// on the recording's timeline (module.jai's audio thread mixes them). Loopback sends nothing while
|
||||
// nothing plays: those gaps are silence. A device unplugged mid-recording (a headset) is opened
|
||||
// again, from the new default, once a second.
|
||||
|
||||
AUDIO_RATE :: 48000;
|
||||
AUDIO_CHANNELS :: 2;
|
||||
|
||||
Audio_Source_Kind :: enum u8 { SYSTEM; MICROPHONE; }
|
||||
|
||||
Audio_Source :: struct {
|
||||
kind: Audio_Source_Kind;
|
||||
name: string; // "Speakers (Realtek(R) Audio)"; heap
|
||||
client: *Audio_Client;
|
||||
capture: *Audio_Capture_Client;
|
||||
channels: s32; // as the device gives them: 2, unless it would not convert
|
||||
open: bool;
|
||||
lost_at: s64; // when it was lost (ticks), to open it again a second on; 0 while open
|
||||
error: string; // why it could not be opened; heap
|
||||
}
|
||||
|
||||
Audio_Packet :: struct {
|
||||
samples: [] float; // interleaved stereo, AUDIO_CHANNELS a frame; empty for silence
|
||||
frames: s64;
|
||||
ticks: s64; // when its first frame was heard, in QueryPerformanceCounter ticks
|
||||
silent: bool;
|
||||
}
|
||||
|
||||
// The default device of its kind, started. False with why: none, or it will not open.
|
||||
audio_open :: (s: *Audio_Source, kind: Audio_Source_Kind) -> ok: bool, error: string {
|
||||
s.kind = kind;
|
||||
enumerator: *Device_Enumerator;
|
||||
hr := audio_CoCreateInstance(*CLSID_MM_DEVICE_ENUMERATOR, null, CLSCTX_ALL, *IID_MM_DEVICE_ENUMERATOR, xx *enumerator);
|
||||
if FAILED(hr) return false, hresult_text("the audio device list", hr);
|
||||
defer com_release(*enumerator);
|
||||
device: *Audio_Device;
|
||||
flow: s32 = ifx kind == .SYSTEM then E_RENDER else E_CAPTURE;
|
||||
hr = enumerator.vtable.GetDefaultAudioEndpoint(enumerator, flow, E_CONSOLE, *device);
|
||||
if FAILED(hr) return false, ifx kind == .SYSTEM then "no sound output device" else "no microphone";
|
||||
defer com_release(*device);
|
||||
free(s.name);
|
||||
s.name = copy_string(audio_device_name(device));
|
||||
|
||||
hr = device.vtable.Activate(device, *IID_AUDIO_CLIENT, CLSCTX_ALL, null, xx *s.client);
|
||||
if FAILED(hr) return false, hresult_text(tprint("% (%)", s.name, "IAudioClient"), hr);
|
||||
|
||||
flags: u32 = AUTOCONVERTPCM | SRC_DEFAULT_QUALITY;
|
||||
if kind == .SYSTEM flags |= STREAMFLAGS_LOOPBACK;
|
||||
wanted := WAVEFORMATEX.{wFormatTag = WAVE_FORMAT_IEEE_FLOAT, nChannels = AUDIO_CHANNELS, nSamplesPerSec = AUDIO_RATE,
|
||||
nAvgBytesPerSec = AUDIO_RATE * AUDIO_CHANNELS * 4, nBlockAlign = AUDIO_CHANNELS * 4, wBitsPerSample = 32};
|
||||
BUFFER :: 2_000_000; // 200 ms, in 100 ns: room for the audio thread being late
|
||||
hr = s.client.vtable.Initialize(s.client, SHAREMODE_SHARED, flags, BUFFER, 0, *wanted, null);
|
||||
s.channels = AUDIO_CHANNELS;
|
||||
if FAILED(hr) {
|
||||
// No conversion here: the device's own format, if it is 48 kHz float, its channels made stereo after.
|
||||
mix: *WAVEFORMATEX;
|
||||
if FAILED(s.client.vtable.GetMixFormat(s.client, *mix)) return false, hresult_text(tprint("% will not give 48 kHz stereo", s.name), hr);
|
||||
defer audio_CoTaskMemFree(mix);
|
||||
is_float := mix.wFormatTag == WAVE_FORMAT_IEEE_FLOAT || (mix.wFormatTag == WAVE_FORMAT_EXTENSIBLE && mix.wBitsPerSample == 32);
|
||||
if !is_float || mix.nSamplesPerSec != AUDIO_RATE
|
||||
return false, tprint("% gives % Hz %, not 48 kHz float, and will not convert", s.name, mix.nSamplesPerSec, ifx is_float then "float" else "PCM");
|
||||
com_release(*s.client);
|
||||
hr = device.vtable.Activate(device, *IID_AUDIO_CLIENT, CLSCTX_ALL, null, xx *s.client);
|
||||
if FAILED(hr) return false, hresult_text(s.name, hr);
|
||||
hr = s.client.vtable.Initialize(s.client, SHAREMODE_SHARED, flags & ~(AUTOCONVERTPCM | SRC_DEFAULT_QUALITY), BUFFER, 0, mix, null);
|
||||
if FAILED(hr) return false, hresult_text(tprint("% (Initialize)", s.name), hr);
|
||||
s.channels = mix.nChannels;
|
||||
}
|
||||
hr = s.client.vtable.GetService(s.client, *IID_AUDIO_CAPTURE_CLIENT, xx *s.capture);
|
||||
if FAILED(hr) return false, hresult_text(tprint("% (IAudioCaptureClient)", s.name), hr);
|
||||
hr = s.client.vtable.Start(s.client);
|
||||
if FAILED(hr) return false, hresult_text(tprint("% (Start)", s.name), hr);
|
||||
s.open = true;
|
||||
s.lost_at = 0;
|
||||
return true, "";
|
||||
}
|
||||
|
||||
// The packets waiting, each as stereo float (temp), oldest first. lost: the device went away
|
||||
// (unplugged, or the default changed): audio_close, and open it again later.
|
||||
audio_take :: (s: *Audio_Source, frequency: s64) -> [] Audio_Packet, lost: bool {
|
||||
packets: [..] Audio_Packet;
|
||||
packets.allocator = temp;
|
||||
while true {
|
||||
next: u32;
|
||||
hr := s.capture.vtable.GetNextPacketSize(s.capture, *next);
|
||||
if hresult_is(hr, AUDCLNT_E_DEVICE_INVALIDATED) return packets, true;
|
||||
if FAILED(hr) || !next break;
|
||||
data: *u8;
|
||||
frames, flags: u32;
|
||||
device_position, qpc_position: u64; // qpc_position: QueryPerformanceCounter time, in 100 ns
|
||||
hr = s.capture.vtable.GetBuffer(s.capture, *data, *frames, *flags, *device_position, *qpc_position);
|
||||
if hresult_is(hr, AUDCLNT_E_DEVICE_INVALIDATED) return packets, true;
|
||||
if FAILED(hr) || hresult_is(hr, AUDCLNT_S_BUFFER_EMPTY) break;
|
||||
|
||||
packet: Audio_Packet;
|
||||
packet.frames = frames;
|
||||
packet.ticks = cast(s64) (cast(float64) qpc_position * frequency / 10_000_000.0);
|
||||
packet.silent = (flags & BUFFERFLAGS_SILENT) != 0;
|
||||
if !packet.silent && frames {
|
||||
packet.samples = NewArray(cast(s64) frames * AUDIO_CHANNELS, float, initialized = false,, temp);
|
||||
input := cast(*float) data;
|
||||
if s.channels == AUDIO_CHANNELS {
|
||||
memcpy(packet.samples.data, input, cast(s64) frames * AUDIO_CHANNELS * size_of(float));
|
||||
} else {
|
||||
// Mono to both sides; more channels than two: the first two (front left and right).
|
||||
channels := cast(s64) s.channels;
|
||||
for f: 0..cast(s64) frames - 1 {
|
||||
left := input[f * channels];
|
||||
right := ifx channels > 1 then input[f * channels + 1] else left;
|
||||
packet.samples[f * 2] = left;
|
||||
packet.samples[f * 2 + 1] = right;
|
||||
}
|
||||
}
|
||||
}
|
||||
s.capture.vtable.ReleaseBuffer(s.capture, frames);
|
||||
array_add(*packets, packet);
|
||||
}
|
||||
return packets, false;
|
||||
}
|
||||
|
||||
audio_close :: (s: *Audio_Source) {
|
||||
if s.client && s.open s.client.vtable.Stop(s.client);
|
||||
com_release(*s.capture);
|
||||
com_release(*s.client);
|
||||
s.open = false;
|
||||
}
|
||||
|
||||
audio_free :: (s: *Audio_Source) {
|
||||
audio_close(s);
|
||||
free(s.name);
|
||||
free(s.error);
|
||||
s.name, s.error = "", "";
|
||||
}
|
||||
|
||||
// COM on this thread, in the multithreaded apartment, as WASAPI's objects are used from it only.
|
||||
audio_thread_start :: () {
|
||||
audio_CoInitializeEx(null, COINIT_MULTITHREADED);
|
||||
}
|
||||
|
||||
audio_thread_end :: () {
|
||||
audio_CoUninitialize();
|
||||
}
|
||||
|
||||
#scope_file
|
||||
|
||||
audio_device_name :: (device: *Audio_Device) -> string {
|
||||
store: *Property_Store;
|
||||
if FAILED(device.vtable.OpenPropertyStore(device, STGM_READ, *store)) return "";
|
||||
defer com_release(*store);
|
||||
value: PROPVARIANT;
|
||||
if FAILED(store.vtable.GetValue(store, *PKEY_DEVICE_FRIENDLY_NAME, *value)) return "";
|
||||
defer audio_PropVariantClear(*value);
|
||||
VT_LPWSTR :: 31;
|
||||
if value.vt != VT_LPWSTR || !value.pointer return "";
|
||||
name, ok := wide_to_utf8(cast(*u16) value.pointer,, temp);
|
||||
return ifx ok then name else "";
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- bindings
|
||||
|
||||
ole32_dll :: #library,system "ole32";
|
||||
|
||||
audio_CoInitializeEx :: (reserved: *void, flags: u32) -> HRESULT #foreign ole32_dll "CoInitializeEx";
|
||||
audio_CoUninitialize :: () #foreign ole32_dll "CoUninitialize";
|
||||
audio_CoCreateInstance :: (clsid: *GUID, outer: *void, class_context: u32, iid: *GUID, out: **void) -> HRESULT #foreign ole32_dll "CoCreateInstance";
|
||||
audio_CoTaskMemFree :: (memory: *void) #foreign ole32_dll "CoTaskMemFree";
|
||||
audio_PropVariantClear :: (value: *PROPVARIANT) -> HRESULT #foreign ole32_dll "PropVariantClear";
|
||||
|
||||
COINIT_MULTITHREADED : u32 : 0;
|
||||
CLSCTX_ALL : u32 : 0x17;
|
||||
E_RENDER : s32 : 0;
|
||||
E_CAPTURE : s32 : 1;
|
||||
E_CONSOLE : s32 : 0;
|
||||
STGM_READ : u32 : 0;
|
||||
SHAREMODE_SHARED : s32 : 0;
|
||||
STREAMFLAGS_LOOPBACK : u32 : 0x00020000;
|
||||
AUTOCONVERTPCM : u32 : 0x80000000;
|
||||
SRC_DEFAULT_QUALITY : u32 : 0x08000000;
|
||||
BUFFERFLAGS_SILENT : u32 : 0x2;
|
||||
WAVE_FORMAT_IEEE_FLOAT : u16 : 3;
|
||||
WAVE_FORMAT_EXTENSIBLE : u16 : 0xFFFE;
|
||||
AUDCLNT_S_BUFFER_EMPTY :: 0x08890001;
|
||||
AUDCLNT_E_DEVICE_INVALIDATED :: 0x88890004;
|
||||
|
||||
CLSID_MM_DEVICE_ENUMERATOR := #run guid("BCDE0395-E52F-467C-8E3D-C4579291692E");
|
||||
IID_MM_DEVICE_ENUMERATOR := #run guid("A95664D2-9614-4F35-A746-DE8DB63617E6");
|
||||
IID_AUDIO_CLIENT := #run guid("1CB9AD4C-DBFA-4C32-B178-C2F568A703B2");
|
||||
IID_AUDIO_CAPTURE_CLIENT := #run guid("C8ADBD64-E71E-48A0-A4DE-185C395CD317");
|
||||
|
||||
PROPERTYKEY :: struct { fmtid: GUID; pid: u32; }
|
||||
PKEY_DEVICE_FRIENDLY_NAME := PROPERTYKEY.{#run guid("A45C254E-DF1C-4EFD-8020-67D146A850E0"), 14};
|
||||
|
||||
// A PROPVARIANT, as far as a string in it: its type, then (after 6 reserved bytes) the pointer.
|
||||
PROPVARIANT :: struct {
|
||||
vt: u16;
|
||||
reserved: [3] u16;
|
||||
pointer: *void;
|
||||
padding: *void;
|
||||
}
|
||||
|
||||
WAVEFORMATEX :: struct {
|
||||
wFormatTag: u16;
|
||||
nChannels: u16;
|
||||
nSamplesPerSec: u32;
|
||||
nAvgBytesPerSec: u32;
|
||||
nBlockAlign: u16;
|
||||
wBitsPerSample: u16;
|
||||
cbSize: u16;
|
||||
}
|
||||
|
||||
Unknown_Methods :: struct {
|
||||
QueryInterface: (this: *void, iid: *GUID, out: **void) -> HRESULT #c_call;
|
||||
AddRef: (this: *void) -> u32 #c_call;
|
||||
Release: (this: *void) -> u32 #c_call;
|
||||
}
|
||||
|
||||
Device_Enumerator :: struct { vtable: *struct {
|
||||
using base: Unknown_Methods;
|
||||
EnumAudioEndpoints: *void;
|
||||
GetDefaultAudioEndpoint: (this: *Device_Enumerator, flow: s32, role: s32, out: **Audio_Device) -> HRESULT #c_call;
|
||||
}; }
|
||||
|
||||
Audio_Device :: struct { vtable: *struct {
|
||||
using base: Unknown_Methods;
|
||||
Activate: (this: *Audio_Device, iid: *GUID, class_context: u32, parameters: *void, out: **void) -> HRESULT #c_call;
|
||||
OpenPropertyStore: (this: *Audio_Device, access: u32, out: **Property_Store) -> HRESULT #c_call;
|
||||
}; }
|
||||
|
||||
Property_Store :: struct { vtable: *struct {
|
||||
using base: Unknown_Methods;
|
||||
GetCount: *void;
|
||||
GetAt: *void;
|
||||
GetValue: (this: *Property_Store, key: *PROPERTYKEY, out: *PROPVARIANT) -> HRESULT #c_call;
|
||||
}; }
|
||||
|
||||
Audio_Client :: struct { vtable: *struct {
|
||||
using base: Unknown_Methods;
|
||||
Initialize: (this: *Audio_Client, share_mode: s32, flags: u32, buffer_duration: s64, periodicity: s64, format: *WAVEFORMATEX, session: *GUID) -> HRESULT #c_call;
|
||||
GetBufferSize: *void;
|
||||
GetStreamLatency: *void;
|
||||
GetCurrentPadding: *void;
|
||||
IsFormatSupported: *void;
|
||||
GetMixFormat: (this: *Audio_Client, out: **WAVEFORMATEX) -> HRESULT #c_call;
|
||||
GetDevicePeriod: *void;
|
||||
Start: (this: *Audio_Client) -> HRESULT #c_call;
|
||||
Stop: (this: *Audio_Client) -> HRESULT #c_call;
|
||||
Reset: *void;
|
||||
SetEventHandle: *void;
|
||||
GetService: (this: *Audio_Client, iid: *GUID, out: **void) -> HRESULT #c_call;
|
||||
}; }
|
||||
|
||||
Audio_Capture_Client :: struct { vtable: *struct {
|
||||
using base: Unknown_Methods;
|
||||
GetBuffer: (this: *Audio_Capture_Client, data: **u8, frames: *u32, flags: *u32, device_position: *u64, qpc_position: *u64) -> HRESULT #c_call;
|
||||
ReleaseBuffer: (this: *Audio_Capture_Client, frames: u32) -> HRESULT #c_call;
|
||||
GetNextPacketSize: (this: *Audio_Capture_Client, out: *u32) -> HRESULT #c_call;
|
||||
}; }
|
||||
@ -78,6 +78,26 @@ gpu_thread_dpi_aware :: () {
|
||||
user32_dll :: #library,system "user32";
|
||||
SetThreadDpiAwarenessContext :: (value: HANDLE) -> HANDLE #foreign user32_dll;
|
||||
|
||||
// A GUID from its text, "A95664D2-9614-4F35-A746-DE8DB63617E6": for #run.
|
||||
guid :: (text: string) -> GUID {
|
||||
hex :: (s: string) -> u64 {
|
||||
value: u64;
|
||||
for 0..s.count - 1 {
|
||||
c := s[it];
|
||||
digit: u64 = ifx c >= #char "a" then cast(u64) (c - #char "a" + 10) else ifx c >= #char "A" then cast(u64) (c - #char "A" + 10) else cast(u64) (c - #char "0");
|
||||
value = value * 16 + digit;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
g: GUID;
|
||||
g.Data1 = cast(u32) hex(slice(text, 0, 8));
|
||||
g.Data2 = cast(u16) hex(slice(text, 9, 4));
|
||||
g.Data3 = cast(u16) hex(slice(text, 14, 4));
|
||||
for 0..1 g.Data4[it] = cast(u8) hex(slice(text, 19 + it * 2, 2));
|
||||
for 0..5 g.Data4[it + 2] = cast(u8) hex(slice(text, 24 + it * 2, 2));
|
||||
return g;
|
||||
}
|
||||
|
||||
com_release :: (p: *$T) {
|
||||
if p.* { IUnknown_Release(cast(*IUnknown) p.*); p.* = null; }
|
||||
}
|
||||
|
||||
@ -4,6 +4,9 @@
|
||||
// BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with
|
||||
// swscale (BT.709, limited range, as tagged).
|
||||
//
|
||||
// With sound, an AAC stream (FFmpeg's own encoder, 160 kb/s, 48 kHz stereo) beside it, written from the
|
||||
// audio thread: the two streams' packets meet in the file under a lock (mux).
|
||||
//
|
||||
// The file is written fragmented, a fragment at each keyframe, so one cut short by a crash still plays
|
||||
// up to its last fragment. Frames are written only when the screen changes, so a count of frames can
|
||||
// span any time: a keyframe is made every RECORD_KEYFRAME_SECONDS of time instead, which bounds what a
|
||||
@ -11,6 +14,7 @@
|
||||
|
||||
RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"];
|
||||
RECORD_KEYFRAME_SECONDS :: 2;
|
||||
RECORD_AUDIO_BIT_RATE :: 160_000;
|
||||
|
||||
Encoder :: struct {
|
||||
format: *AVFormatContext;
|
||||
@ -26,11 +30,20 @@ Encoder :: struct {
|
||||
last_key_pts: s64 = -1; // the last keyframe asked for
|
||||
header_written: bool;
|
||||
bytes: s64; // of packets written
|
||||
|
||||
// Sound, when asked for: AAC, written from the audio thread.
|
||||
audio: *AVCodecContext;
|
||||
audio_stream: *AVStream;
|
||||
audio_frame: *AVFrame;
|
||||
audio_packet: *AVPacket;
|
||||
audio_frame_size: s32; // samples a frame: AAC's 1024
|
||||
mux: Mutex; // the video and audio threads' packets into the file
|
||||
mux_ready: bool;
|
||||
}
|
||||
|
||||
// Opens the file and the first encoder that will take width x height (wanted alone, when given).
|
||||
// Failures to open an encoder are listed in tried (temp), for the log.
|
||||
encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string) -> ok: bool, error: string, tried: string {
|
||||
encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string, audio := false) -> ok: bool, error: string, tried: string {
|
||||
av_log_set_level(AV_LOG_ERROR);
|
||||
e.width, e.height = width, height;
|
||||
result := avformat_alloc_output_context2(*e.format, null, "mp4", temp_c_string(path));
|
||||
@ -55,6 +68,12 @@ encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, w
|
||||
if !e.stream { encoder_free(e); return false, "FFmpeg cannot add a stream", tried_text; }
|
||||
avcodec_parameters_from_context(e.stream.codecpar, e.codec);
|
||||
e.stream.time_base = e.codec.time_base;
|
||||
if audio {
|
||||
audio_ok, audio_error := encoder_open_audio(e);
|
||||
if !audio_ok { encoder_free(e); return false, audio_error, tried_text; }
|
||||
}
|
||||
init(*e.mux, "record mux");
|
||||
e.mux_ready = true;
|
||||
|
||||
result = avio_open(*e.format.pb, temp_c_string(path), AVIO_FLAG_WRITE);
|
||||
if result < 0 { encoder_free(e); return false, av_error(tprint("FFmpeg cannot write %", path), result), tried_text; }
|
||||
@ -111,12 +130,36 @@ encoder_write :: (e: *Encoder, pixels: *u8, pts: s64) -> ok: bool, error: string
|
||||
return drained, error;
|
||||
}
|
||||
|
||||
// Flushes the encoder, finishes the file and closes it.
|
||||
// One AAC frame of sound: e.audio_frame_size frames of interleaved stereo float, at pts in samples.
|
||||
// From the audio thread.
|
||||
encoder_write_audio :: (e: *Encoder, samples: *float, pts: s64) -> ok: bool, error: string {
|
||||
frame := e.audio_frame;
|
||||
result := av_frame_make_writable(frame);
|
||||
if result < 0 return false, av_error("FFmpeg cannot make a sound frame", result);
|
||||
left := cast(*float) frame.data[0];
|
||||
right := cast(*float) frame.data[1];
|
||||
for 0..e.audio_frame_size - 1 {
|
||||
left[it] = samples[it * 2];
|
||||
right[it] = samples[it * 2 + 1];
|
||||
}
|
||||
frame.pts = pts;
|
||||
result = avcodec_send_frame(e.audio, frame);
|
||||
if result < 0 return false, av_error("the AAC encoder", result);
|
||||
drained, error := encoder_drain(e, e.audio, e.audio_stream, e.audio_packet);
|
||||
return drained, error;
|
||||
}
|
||||
|
||||
// Flushes the encoders, sound first, finishes the file and closes it.
|
||||
encoder_close :: (e: *Encoder) -> ok: bool, error: string {
|
||||
ok, error := true, "";
|
||||
if e.audio && e.header_written {
|
||||
avcodec_send_frame(e.audio, null);
|
||||
ok, error = encoder_drain(e, e.audio, e.audio_stream, e.audio_packet);
|
||||
}
|
||||
if e.codec && e.header_written {
|
||||
avcodec_send_frame(e.codec, null);
|
||||
ok, error = encoder_drain(e);
|
||||
video_ok, video_error := encoder_drain(e);
|
||||
if ok ok, error = video_ok, video_error;
|
||||
result := av_write_trailer(e.format);
|
||||
if ok && result < 0 ok, error = false, av_error("FFmpeg cannot finish the MP4", result);
|
||||
}
|
||||
@ -186,22 +229,64 @@ encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat {
|
||||
return .NONE;
|
||||
}
|
||||
|
||||
encoder_drain :: (e: *Encoder) -> ok: bool, error: string {
|
||||
// The packets an encoder (the video's, unless another is given) has ready, into the file.
|
||||
encoder_drain :: (e: *Encoder, codec: *AVCodecContext = null, stream: *AVStream = null, packet: *AVPacket = null) -> ok: bool, error: string {
|
||||
AVERROR_EAGAIN :: -11;
|
||||
if !codec codec, stream, packet = e.codec, e.stream, e.packet;
|
||||
while true {
|
||||
result := avcodec_receive_packet(e.codec, e.packet);
|
||||
result := avcodec_receive_packet(codec, packet);
|
||||
if result == AVERROR_EAGAIN || result == cast,no_check(s32) AVERROR_EOF return true, "";
|
||||
if result < 0 return false, av_error(tprint("% failed", e.name), result);
|
||||
av_packet_rescale_ts(e.packet, e.codec.time_base, e.stream.time_base);
|
||||
e.packet.stream_index = e.stream.index;
|
||||
e.bytes += e.packet.size;
|
||||
result = av_interleaved_write_frame(e.format, e.packet); // takes the packet's data
|
||||
if result < 0 return false, av_error(tprint("% failed", ifx codec == e.audio then "the AAC encoder" else e.name), result);
|
||||
av_packet_rescale_ts(packet, codec.time_base, stream.time_base);
|
||||
packet.stream_index = stream.index;
|
||||
lock(*e.mux);
|
||||
e.bytes += packet.size;
|
||||
result = av_interleaved_write_frame(e.format, packet); // takes the packet's data
|
||||
unlock(*e.mux);
|
||||
if result < 0 return false, av_error("FFmpeg cannot write to the MP4", result);
|
||||
}
|
||||
return true, "";
|
||||
}
|
||||
|
||||
encoder_open_audio :: (e: *Encoder) -> ok: bool, error: string {
|
||||
codec := avcodec_find_encoder_by_name("aac");
|
||||
if !codec return false, "this FFmpeg has no AAC encoder";
|
||||
c := avcodec_alloc_context3(codec);
|
||||
if !c return false, "out of memory";
|
||||
c.sample_fmt = .FLTP;
|
||||
c.sample_rate = AUDIO_RATE;
|
||||
av_channel_layout_default(*c.ch_layout, AUDIO_CHANNELS);
|
||||
c.bit_rate = RECORD_AUDIO_BIT_RATE;
|
||||
c.time_base = .{1, AUDIO_RATE};
|
||||
if e.format.oformat.flags & AVFMT_GLOBALHEADER c.flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
|
||||
result := avcodec_open2(c, codec, null);
|
||||
if result < 0 { avcodec_free_context(*c); return false, av_error("the AAC encoder cannot open", result); }
|
||||
e.audio = c;
|
||||
e.audio_frame_size = ifx c.frame_size > 0 then c.frame_size else 1024;
|
||||
|
||||
e.audio_stream = avformat_new_stream(e.format, null);
|
||||
if !e.audio_stream return false, "FFmpeg cannot add a sound stream";
|
||||
avcodec_parameters_from_context(e.audio_stream.codecpar, c);
|
||||
e.audio_stream.time_base = c.time_base;
|
||||
|
||||
e.audio_packet = av_packet_alloc();
|
||||
e.audio_frame = av_frame_alloc();
|
||||
if !e.audio_packet || !e.audio_frame return false, "FFmpeg is out of memory";
|
||||
e.audio_frame.format = cast(s32) AVSampleFormat.FLTP;
|
||||
e.audio_frame.nb_samples = e.audio_frame_size;
|
||||
e.audio_frame.sample_rate = AUDIO_RATE;
|
||||
av_channel_layout_default(*e.audio_frame.ch_layout, AUDIO_CHANNELS);
|
||||
result = av_frame_get_buffer(e.audio_frame, 0);
|
||||
if result < 0 return false, av_error("FFmpeg cannot make a sound frame", result);
|
||||
return true, "";
|
||||
}
|
||||
|
||||
encoder_free :: (e: *Encoder) {
|
||||
if e.audio_frame av_frame_free(*e.audio_frame);
|
||||
if e.audio_packet av_packet_free(*e.audio_packet);
|
||||
if e.audio avcodec_free_context(*e.audio);
|
||||
e.audio_stream = null;
|
||||
if e.mux_ready { destroy(*e.mux); e.mux_ready = false; }
|
||||
if e.sws sws_freeContext(e.sws);
|
||||
if e.frame av_frame_free(*e.frame);
|
||||
if e.packet av_packet_free(*e.packet);
|
||||
|
||||
@ -13,6 +13,11 @@
|
||||
// one a second while nothing does, so players can seek. Each carries the time it was taken, less the
|
||||
// time spent paused (record_pause), so a pause leaves no gap in the video.
|
||||
//
|
||||
// Sound (system_audio, microphone) has a thread of its own (audio_source.jai's WASAPI): each packet is
|
||||
// put on the same timeline as the frames, by when it was heard, less the same pauses, into one mix,
|
||||
// and an AAC frame of it is encoded once every source has had its say up to there. A source that sends
|
||||
// nothing (the computer playing nothing) is silence.
|
||||
//
|
||||
// recorder: Recorder;
|
||||
// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0});
|
||||
// ...
|
||||
@ -66,6 +71,8 @@ Record_Options :: struct {
|
||||
capture: Record_Capture;
|
||||
legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe
|
||||
testing: Record_Testing;
|
||||
system_audio: bool; // what the computer plays (the default output device)
|
||||
microphone: bool; // the default input device
|
||||
}
|
||||
|
||||
Record_Stats :: struct {
|
||||
@ -81,6 +88,9 @@ Record_Stats :: struct {
|
||||
dropped: s64; // the encoder was too far behind
|
||||
lost: s64; // times the duplication was lost and made again
|
||||
paused_seconds: float64; // left out of the video
|
||||
audio: string; // the sound recorded: "Speakers (Realtek(R) Audio); Microphone (USB Audio)", or ""
|
||||
audio_frames: s64; // AAC frames encoded
|
||||
audio_errors: string; // sources that could not be opened, or were lost
|
||||
copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks
|
||||
bytes: s64; // encoded
|
||||
seconds: float64; // from start to stop
|
||||
@ -99,7 +109,15 @@ Recorder :: struct {
|
||||
start_ticks: s64;
|
||||
|
||||
capture_thread, encoder_thread: Thread;
|
||||
capture_started, encoder_started: bool;
|
||||
capture_started, encoder_started, audio_started: bool;
|
||||
audio_thread: Thread;
|
||||
audio_opened: Semaphore; // the audio thread has opened its sources
|
||||
audio_go: Semaphore; // the encoder is open: the audio thread may write
|
||||
audio_sources: [2] Audio_Source;
|
||||
audio_open: bool; // any source opened: the MP4 has sound
|
||||
zero_ticks: s64; // the recording's time 0, for frames and sound alike
|
||||
paused_ticks: s64; // time paused so far, for the capture and audio threads; set by record_pause
|
||||
pause_began: s64;
|
||||
running: bool;
|
||||
stopping: bool; // set by record_stop; the capture thread ends, then the encoder
|
||||
paused: bool; // set by record_pause; the capture thread takes nothing while it is
|
||||
@ -175,6 +193,7 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri
|
||||
QueryPerformanceFrequency(*frequency);
|
||||
r.frequency = frequency.QuadPart;
|
||||
r.start_ticks = ticks_now();
|
||||
r.zero_ticks = r.start_ticks;
|
||||
|
||||
// The source is opened on the capture thread, which uses it: WinRT's objects belong to the
|
||||
// apartment they are made in.
|
||||
@ -188,7 +207,18 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri
|
||||
return false, error, fall_back;
|
||||
}
|
||||
|
||||
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder);
|
||||
// The sound's sources are opened on their own thread (COM, for WASAPI), before the encoder, which
|
||||
// has a sound stream only when one opened.
|
||||
if r.options.system_audio || r.options.microphone {
|
||||
init(*r.audio_opened);
|
||||
init(*r.audio_go);
|
||||
thread_init(*r.audio_thread, record_audio_proc, name = "record audio");
|
||||
r.audio_started = true;
|
||||
thread_start(*r.audio_thread, r);
|
||||
wait_for(*r.audio_opened);
|
||||
}
|
||||
|
||||
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder, audio = r.audio_open);
|
||||
if tried log("[record] Encoders that did not open: %", tried);
|
||||
if !ok {
|
||||
error = copy_string(error,, temp);
|
||||
@ -196,6 +226,7 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri
|
||||
return false, error, false;
|
||||
}
|
||||
r.stats.encoder = copy_string(r.encoder.name);
|
||||
if r.audio_started signal(*r.audio_go);
|
||||
|
||||
thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder");
|
||||
r.encoder_started = true;
|
||||
@ -211,6 +242,7 @@ record_stop :: (r: *Recorder) -> ok: bool, error: string {
|
||||
atomic_swap(*r.stopping, true);
|
||||
thread_is_done(*r.capture_thread, -1);
|
||||
thread_is_done(*r.encoder_thread, -1);
|
||||
if r.audio_started thread_is_done(*r.audio_thread, -1);
|
||||
r.running = false;
|
||||
|
||||
closed, close_error := encoder_close(*r.encoder);
|
||||
@ -224,9 +256,23 @@ record_stop :: (r: *Recorder) -> ok: bool, error: string {
|
||||
return true, "";
|
||||
}
|
||||
|
||||
// Pauses or resumes. While paused nothing is taken, and the time is left out of the video.
|
||||
// Pauses or resumes. While paused nothing is taken, and the time is left out of the video, sound and
|
||||
// all: the time paused is kept here, once, for the capture and audio threads both.
|
||||
record_pause :: (r: *Recorder, pause: bool) {
|
||||
if r.running atomic_swap(*r.paused, pause);
|
||||
if !r.running || atomic_read(*r.paused) == pause return;
|
||||
now := ticks_now();
|
||||
if pause {
|
||||
atomic_write(*r.pause_began, now);
|
||||
atomic_swap(*r.paused, true);
|
||||
} else {
|
||||
atomic_add(*r.paused_ticks, now - atomic_read(*r.pause_began)); // before paused clears, so no thread sees it unpaused with the old total
|
||||
atomic_swap(*r.paused, false);
|
||||
}
|
||||
}
|
||||
|
||||
// Ticks on the recording's timeline: time 0 at its start, less the time paused.
|
||||
record_timeline :: (r: *Recorder, ticks: s64) -> s64 {
|
||||
return ticks - r.zero_ticks - atomic_read(*r.paused_ticks);
|
||||
}
|
||||
|
||||
record_paused :: (r: *Recorder) -> bool {
|
||||
@ -265,6 +311,14 @@ record_free :: (r: *Recorder) {
|
||||
atomic_swap(*r.stopping, true);
|
||||
if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; }
|
||||
if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; }
|
||||
if r.audio_started {
|
||||
signal(*r.audio_go); // in case it waits for an encoder that did not open
|
||||
thread_is_done(*r.audio_thread, -1);
|
||||
thread_deinit(*r.audio_thread);
|
||||
destroy(*r.audio_opened);
|
||||
destroy(*r.audio_go);
|
||||
r.audio_started = false;
|
||||
}
|
||||
if r.encoder.format encoder_close(*r.encoder);
|
||||
for * r.frames array_free(it.pixels);
|
||||
array_free(r.last_frame);
|
||||
@ -331,7 +385,7 @@ record_capture_proc :: (thread: *Thread) -> s64 {
|
||||
have_image, changed := false, false;
|
||||
staged: [RECORD_STAGING] Staged;
|
||||
emits: s64;
|
||||
paused_ticks, pause_start: s64; // time paused so far, left out of the frames' times; when this pause began
|
||||
was_paused := false;
|
||||
last_lost := ticks_now(); // for Record_Testing.LOSE_DUPLICATION
|
||||
|
||||
while !atomic_read(*r.stopping) {
|
||||
@ -381,15 +435,15 @@ record_capture_proc :: (thread: *Thread) -> s64 {
|
||||
// screen as it is now is sent at once.
|
||||
now = ticks_now();
|
||||
if atomic_read(*r.paused) {
|
||||
if !pause_start pause_start = now;
|
||||
lock(*r.mutex); r.stats.paused_seconds = cast(float64) (paused_ticks + now - pause_start) / r.frequency; unlock(*r.mutex);
|
||||
was_paused = true;
|
||||
lock(*r.mutex); r.stats.paused_seconds = cast(float64) (atomic_read(*r.paused_ticks) + now - atomic_read(*r.pause_began)) / r.frequency; unlock(*r.mutex);
|
||||
continue;
|
||||
}
|
||||
if pause_start {
|
||||
paused_ticks += now - pause_start;
|
||||
pause_start = 0;
|
||||
if was_paused {
|
||||
was_paused = false;
|
||||
changed = true;
|
||||
turn = now;
|
||||
lock(*r.mutex); r.stats.paused_seconds = cast(float64) atomic_read(*r.paused_ticks) / r.frequency; unlock(*r.mutex);
|
||||
}
|
||||
if !have_image continue;
|
||||
repeat := !changed && now - last_emit >= r.frequency;
|
||||
@ -400,7 +454,7 @@ record_capture_proc :: (thread: *Thread) -> s64 {
|
||||
slot := emits % RECORD_STAGING;
|
||||
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
|
||||
ID3D11DeviceContext_CopyResource(gpu.device_context, gpu.staging[slot], gpu.latest);
|
||||
staged[slot] = .{pending = true, ticks = now - paused_ticks};
|
||||
staged[slot] = .{pending = true, ticks = record_timeline(r, now)};
|
||||
if !r.using_wgc staged[slot].pointer, staged[slot].pointer_visible = r.dxgi.pointer_position, r.dxgi.pointer_visible;
|
||||
emits += 1;
|
||||
last_emit = now;
|
||||
@ -465,7 +519,6 @@ record_read_back :: (r: *Recorder, gpu: *Capture_Gpu, slot: s64, staged: Staged)
|
||||
|
||||
record_encoder_proc :: (thread: *Thread) -> s64 {
|
||||
r := cast(*Recorder) thread.data;
|
||||
first: s64 = -1;
|
||||
while true {
|
||||
wait_for(*r.ready, 100);
|
||||
lock(*r.mutex);
|
||||
@ -482,8 +535,7 @@ record_encoder_proc :: (thread: *Thread) -> s64 {
|
||||
}
|
||||
|
||||
frame := *r.frames[index];
|
||||
if first < 0 first = frame.ticks;
|
||||
pts := (frame.ticks - first) * 90000 / r.frequency;
|
||||
pts := max(frame.ticks, 0) * 90000 / r.frequency; // frame.ticks is on the timeline already
|
||||
t := ticks_now();
|
||||
ok, error := encoder_write(*r.encoder, frame.pixels.data, pts);
|
||||
|
||||
@ -510,6 +562,137 @@ record_encoder_proc :: (thread: *Thread) -> s64 {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// The audio thread: opens the sources asked for, waits for the encoder, then every 10 ms takes their
|
||||
// packets into the mix and encodes what every source has covered.
|
||||
record_audio_proc :: (thread: *Thread) -> s64 {
|
||||
r := cast(*Recorder) thread.data;
|
||||
audio_thread_start();
|
||||
defer audio_thread_end();
|
||||
|
||||
kinds: [2] Audio_Source_Kind = .[.SYSTEM, .MICROPHONE];
|
||||
wanted: [2] bool = .[r.options.system_audio, r.options.microphone];
|
||||
names, errors: [..] string;
|
||||
names.allocator, errors.allocator = temp, temp;
|
||||
for 0..1 {
|
||||
if !wanted[it] continue;
|
||||
s := *r.audio_sources[it];
|
||||
ok, error := audio_open(s, kinds[it]);
|
||||
what := ifx kinds[it] == .SYSTEM then "system sound" else "microphone";
|
||||
if ok { r.audio_open = true; array_add(*names, s.name); }
|
||||
else { audio_close(s); array_add(*errors, tprint("%: %", what, error)); log_error("[record] No %: %", what, error); }
|
||||
}
|
||||
r.stats.audio = join(..names, "; ");
|
||||
r.stats.audio_errors = join(..errors, "; ");
|
||||
signal(*r.audio_opened);
|
||||
wait_for(*r.audio_go);
|
||||
if !r.audio_open || !r.encoder.audio || atomic_read(*r.stopping) {
|
||||
for * r.audio_sources audio_free(it);
|
||||
return 0;
|
||||
}
|
||||
|
||||
mix: Audio_Mix;
|
||||
mix.frame_size = r.encoder.audio_frame_size;
|
||||
defer array_free(mix.samples);
|
||||
while true {
|
||||
stopping := atomic_read(*r.stopping);
|
||||
for * source, index: r.audio_sources {
|
||||
if !wanted[index] continue;
|
||||
if !source.open { // lost: opened again a second on, from the default device then
|
||||
if source.lost_at && ticks_now() - source.lost_at >= r.frequency && !stopping {
|
||||
if audio_open(source, kinds[index]) log("[record] % is back: %.", ifx kinds[index] == .SYSTEM then "System sound" else "The microphone", source.name);
|
||||
else { audio_close(source); source.lost_at = ticks_now(); }
|
||||
}
|
||||
continue;
|
||||
}
|
||||
packets, lost := audio_take(source, r.frequency);
|
||||
for packets if !atomic_read(*r.paused) audio_mix_add(*mix, r, index, it);
|
||||
if lost {
|
||||
log_error("[record] % was lost (unplugged, or the default device changed); trying again.", source.name);
|
||||
audio_close(source);
|
||||
source.lost_at = ticks_now();
|
||||
}
|
||||
}
|
||||
ok, error := audio_mix_encode(*mix, r, final = stopping);
|
||||
if !ok { record_fail(r, error); break; }
|
||||
if stopping break;
|
||||
reset_temporary_storage();
|
||||
sleep_milliseconds(10);
|
||||
}
|
||||
for * r.audio_sources audio_free(it);
|
||||
return 0;
|
||||
}
|
||||
|
||||
// The sound so far, from its base: one AAC frame's worth is encoded at a time.
|
||||
Audio_Mix :: struct {
|
||||
samples: [..] float; // interleaved stereo, from base on
|
||||
base: s64; // the frame (in samples, on the timeline) samples[0] is
|
||||
next: [2] s64; // each source's next frame, so packets that follow on are not moved by jitter
|
||||
started: [2] bool;
|
||||
frame_size: s32;
|
||||
}
|
||||
|
||||
AUDIO_SLACK :: AUDIO_RATE / 5; // 200 ms: how late a source's packets may come and still be in the mix
|
||||
AUDIO_FOLLOW_ON :: AUDIO_RATE / 33; // 30 ms: a packet that close to where its source left off follows on
|
||||
|
||||
// A packet into the mix at its place on the timeline, added to what the other source put there.
|
||||
audio_mix_add :: (mix: *Audio_Mix, r: *Recorder, source: s64, packet: Audio_Packet) {
|
||||
at := record_timeline(r, packet.ticks) * AUDIO_RATE / r.frequency;
|
||||
if mix.started[source] && abs(at - mix.next[source]) <= AUDIO_FOLLOW_ON at = mix.next[source];
|
||||
mix.started[source] = true;
|
||||
mix.next[source] = at + packet.frames;
|
||||
end := (at + packet.frames - mix.base) * AUDIO_CHANNELS;
|
||||
if end > mix.samples.count {
|
||||
old := mix.samples.count;
|
||||
array_resize(*mix.samples, end, initialize = false);
|
||||
for old..end - 1 mix.samples[it] = 0;
|
||||
}
|
||||
if packet.silent || !packet.samples.count return;
|
||||
for frame: max(at, mix.base)..at + packet.frames - 1 {
|
||||
into := (frame - mix.base) * AUDIO_CHANNELS;
|
||||
from := (frame - at) * AUDIO_CHANNELS;
|
||||
mix.samples[into] += packet.samples[from];
|
||||
mix.samples[into + 1] += packet.samples[from + 1];
|
||||
}
|
||||
}
|
||||
|
||||
// Encodes the frames every open source has covered: up to where each has sent, or, for one that has
|
||||
// sent nothing lately (silence), to AUDIO_SLACK ago. final: everything sent.
|
||||
audio_mix_encode :: (mix: *Audio_Mix, r: *Recorder, final: bool) -> ok: bool, error: string {
|
||||
// Paused, the timeline stands still where the pause began.
|
||||
now := ifx atomic_read(*r.paused) then atomic_read(*r.pause_began) else ticks_now();
|
||||
now_frame := record_timeline(r, now) * AUDIO_RATE / r.frequency;
|
||||
ready := now_frame - AUDIO_SLACK;
|
||||
for source, index: r.audio_sources {
|
||||
if !source.open continue;
|
||||
covered := max(mix.next[index], now_frame - AUDIO_SLACK);
|
||||
ready = min(ready, covered);
|
||||
}
|
||||
if final { ready = mix.base; for mix.next ready = max(ready, it); }
|
||||
|
||||
size := mix.frame_size;
|
||||
frame := NewArray(size * AUDIO_CHANNELS, float, initialized = false,, temp);
|
||||
done: s64;
|
||||
while mix.base + size <= ready {
|
||||
for i: 0..frame.count - 1 {
|
||||
at := done * AUDIO_CHANNELS + i;
|
||||
value := ifx at < mix.samples.count then mix.samples[at] else 0.0;
|
||||
frame[i] = clamp(value, -1.0, 1.0);
|
||||
}
|
||||
ok, error := encoder_write_audio(*r.encoder, frame.data, mix.base);
|
||||
if !ok return false, error;
|
||||
mix.base += size;
|
||||
done += size;
|
||||
lock(*r.mutex); r.stats.audio_frames += 1; unlock(*r.mutex);
|
||||
}
|
||||
// What was encoded leaves the front.
|
||||
gone := min(done * AUDIO_CHANNELS, mix.samples.count);
|
||||
if gone {
|
||||
memcpy(mix.samples.data, mix.samples.data + gone, (mix.samples.count - gone) * size_of(float));
|
||||
mix.samples.count -= gone;
|
||||
}
|
||||
return true, "";
|
||||
}
|
||||
|
||||
// Ends the recording from one of its threads; the first reason is kept.
|
||||
record_fail :: (r: *Recorder, error: string) {
|
||||
log_error("[record] %", error);
|
||||
@ -530,11 +713,13 @@ rect_inside :: (inner: RECT, outer: RECT) -> bool {
|
||||
#load "wgc_source.jai";
|
||||
#load "encoder.jai";
|
||||
#load "remux.jai";
|
||||
#load "audio_source.jai";
|
||||
#load "cursor.jai";
|
||||
|
||||
#import "Basic";
|
||||
#import "String";
|
||||
#import "Atomics";
|
||||
#import "Math";
|
||||
#import "Thread";
|
||||
#import "Windows";
|
||||
#import "Windows_Utf8";
|
||||
|
||||
@ -308,24 +308,6 @@ IID_DIRECT3D_DEVICE := #run guid("A37624AB-8D5F-4650-9D3E-9EAE3D9BC670");
|
||||
IID_DXGI_INTERFACE_ACCESS := #run guid("A9B3D012-3DF2-4EE3-B8D1-8695F457D3C1");
|
||||
IID_CLOSABLE := #run guid("30D5A829-7FA4-4026-83BB-D75BAE4EA99E");
|
||||
|
||||
guid :: (text: string) -> GUID {
|
||||
hex :: (s: string) -> u64 {
|
||||
value: u64;
|
||||
for 0..s.count - 1 {
|
||||
c := s[it];
|
||||
digit: u64 = ifx c >= #char "a" then cast(u64) (c - #char "a" + 10) else ifx c >= #char "A" then cast(u64) (c - #char "A" + 10) else cast(u64) (c - #char "0");
|
||||
value = value * 16 + digit;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
g: GUID;
|
||||
g.Data1 = cast(u32) hex(slice(text, 0, 8));
|
||||
g.Data2 = cast(u16) hex(slice(text, 9, 4));
|
||||
g.Data3 = cast(u16) hex(slice(text, 14, 4));
|
||||
for 0..1 g.Data4[it] = cast(u8) hex(slice(text, 19 + it * 2, 2));
|
||||
for 0..5 g.Data4[it + 2] = cast(u8) hex(slice(text, 24 + it * 2, 2));
|
||||
return g;
|
||||
}
|
||||
|
||||
// IUnknown, then IInspectable's three: every WinRT interface starts so.
|
||||
Inspectable_Methods :: struct {
|
||||
|
||||
@ -23,6 +23,7 @@ Screen_Capture_Job :: struct {
|
||||
source: string; // how the screen was taken: Record_Stats.source
|
||||
dropped: s64; // frames the encoder fell too far behind for
|
||||
paused_seconds: float64; // left out of it
|
||||
audio: string; // the sound's sources, "Speakers (...); Microphone (...)", or ""
|
||||
|
||||
thread: Thread;
|
||||
busy: bool; // cleared by the thread as it ends
|
||||
@ -142,6 +143,7 @@ screen_capture_job_free :: (job: *Screen_Capture_Job) {
|
||||
free(job.recording);
|
||||
free(job.encoder);
|
||||
free(job.source);
|
||||
free(job.audio);
|
||||
free(job);
|
||||
}
|
||||
|
||||
|
||||
@ -9,8 +9,8 @@
|
||||
//
|
||||
// While it records, a red border is drawn just outside the region and a toolbar beside it gives the
|
||||
// time, Pause, Stop and Discard (app/screen_record_win32.jai); both are kept out of the recording.
|
||||
// Time paused is left out of the video. The frame rate, the cursor and how the screen is taken are
|
||||
// [video] in the settings file.
|
||||
// Time paused is left out of the video. With sound: what the computer plays and the microphone, as
|
||||
// [video] in the settings file says, as are the frame rate, the cursor and how the screen is taken.
|
||||
//
|
||||
// A recording that stops by itself (the screen went away, the encoder failed) keeps what it has, and
|
||||
// so does quitting while recording. One left in the temporary folder by a crash is offered at the next
|
||||
@ -204,6 +204,8 @@ screen_record_start :: (immediate := false) {
|
||||
options.region = .{shot.left, shot.top, shot.left + shot.width, shot.top + shot.height};
|
||||
options.fps = cast(s32) app_user_settings.video_fps;
|
||||
options.cursor = app_user_settings.video_cursor;
|
||||
options.system_audio = app_user_settings.video_system_audio;
|
||||
options.microphone = app_user_settings.video_microphone;
|
||||
if app_user_settings.video_capture == {
|
||||
case "dxgi"; options.capture = .DXGI;
|
||||
case "wgc"; options.capture = .WGC;
|
||||
@ -224,7 +226,13 @@ screen_record_start :: (immediate := false) {
|
||||
if stats.fell_back_from log("[record] Desktop Duplication could not run: %", stats.fell_back_from);
|
||||
log("[record] Recording % x % at %, % (%) of \"%\" (%) with % and %, to %.", stats.width, stats.height, shot.left, shot.top,
|
||||
ifx shot.whole_screen then "the whole screen" else "a region", shot.window_title, shot.app_name, stats.source, stats.encoder, path);
|
||||
if stats.audio log("[record] Sound: %.", stats.audio);
|
||||
if stats.audio_errors {
|
||||
log_error("[record] Sound not recorded: %.", stats.audio_errors);
|
||||
toast_show(.Error, "Recording without some sound: %", stats.audio_errors);
|
||||
} else {
|
||||
toast_show(.Success, "Recording % x %: % stops", stats.width, stats.height, screen_record_hotkey_text());
|
||||
}
|
||||
record_ui_open(options.region, shot.whole_screen, stats.width, stats.height);
|
||||
}
|
||||
|
||||
@ -278,6 +286,7 @@ screen_record_stop :: (shutdown := false) {
|
||||
job.source = copy_string(stats.source,, context.default_allocator);
|
||||
job.dropped = stats.dropped;
|
||||
job.paused_seconds = stats.paused_seconds;
|
||||
job.audio = copy_string(stats.audio,, context.default_allocator);
|
||||
screen_record.shot = .{};
|
||||
screen_record.date, screen_record.path = "", "";
|
||||
screen_capture_job_start(job);
|
||||
@ -338,6 +347,8 @@ screen_record_write :: (job: *Screen_Capture_Job) -> path: string, error: string
|
||||
if job.encoder import_add(doc, info, "Encoder", import_text(job.encoder));
|
||||
if job.source import_add(doc, info, "Capture", import_text(job.source));
|
||||
import_add_number(doc, info, "DroppedFrames", xx job.dropped);
|
||||
capture_object_add(doc, *info.root, "Audio", capture_boolean(video.has_audio));
|
||||
if job.audio import_add(doc, info, "AudioSources", import_text(job.audio));
|
||||
if job.paused_seconds > 0 import_add_number(doc, info, "Paused", import_milliseconds(job.paused_seconds));
|
||||
|
||||
name := import_out_name(job.date, hash);
|
||||
|
||||
@ -783,6 +783,37 @@ screen_record_tests :: () {
|
||||
file_delete(leftover_path);
|
||||
}
|
||||
|
||||
// With sound: an AAC stream beside the video, kept through the remux, and capture.json says so.
|
||||
encoder = .{};
|
||||
opened, open_error = encoder_open(*encoder, mp4, WIDTH, HEIGHT, 30, "", audio = true);
|
||||
expect(opened && encoder.audio != null, "an encoder with sound opened: %", open_error);
|
||||
if opened {
|
||||
tone := NewArray(encoder.audio_frame_size * 2, float,, temp);
|
||||
for frame: 0..29 encoder_write(*encoder, pixels.data, frame * 3000);
|
||||
for block: 0..46 { // about a second of 440 Hz
|
||||
for i: 0..encoder.audio_frame_size - 1 {
|
||||
v := sin(cast(float) (block * encoder.audio_frame_size + i) * 2 * PI * 440 / 48000) * 0.5;
|
||||
tone[i * 2], tone[i * 2 + 1] = v, v;
|
||||
}
|
||||
encoder_write_audio(*encoder, tone.data, block * encoder.audio_frame_size);
|
||||
}
|
||||
encoder_close(*encoder);
|
||||
sounding: Screen_Capture_Job;
|
||||
sounding.video = true;
|
||||
sounding.recording = mp4;
|
||||
sounding.date = "2026-10-10 08:30:00";
|
||||
sounding.audio = "Speakers (test)";
|
||||
sounding.shot.width, sounding.shot.height = WIDTH, HEIGHT;
|
||||
sounding_path, sounding_error := screen_record_write(*sounding);
|
||||
expect(!sounding_error, "a recording with sound is written: %", sounding_error);
|
||||
if !sounding_error {
|
||||
m4, parsed4 := mcap_parse(cast([] u8) read_entire_file(sounding_path,, temp));
|
||||
expect(parsed4 && find_index_from_left(cast(string) m4.original, "mp4a") >= 0, "its MP4 has the AAC stream (mp4a)");
|
||||
expect(parsed4 && contains(m4.text, "Speakers (test)"), "capture.json names the sound's source");
|
||||
file_delete(sounding_path);
|
||||
}
|
||||
}
|
||||
|
||||
// One a crash cut short: a frame every 100 ms for 5 s makes fragments at 0, 2 and 4 s (a
|
||||
// keyframe each 2 s, however few the frames); cut at the last, as a crash would, it goes into the
|
||||
// library as a regular MP4 (header first, no fragments) of the 4 s before the cut.
|
||||
|
||||
@ -311,6 +311,8 @@ App_User_Settings :: struct {
|
||||
video_fps: s64 = VIDEO_FPS_DEFAULT;
|
||||
video_cursor := true;
|
||||
video_capture: string;
|
||||
video_system_audio := true; // what the computer plays
|
||||
video_microphone: bool; // off unless asked for
|
||||
|
||||
// The [hotkeys] lines. Only meaningful when the section was seen; see hotkeys_install.
|
||||
hotkey_bindings: [..] Hotkey_Binding;
|
||||
@ -380,12 +382,12 @@ app_load_user_settings_file :: () -> write_back: bool {
|
||||
if hotkeys_missing.count || global_missing.count return true;
|
||||
|
||||
#if OS == .LINUX seen.global_hotkeys = true; // not written there: Linux has no global hotkeys yet
|
||||
return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.video_fps && seen.video_cursor && seen.video_capture);
|
||||
return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.video_fps && seen.video_cursor && seen.video_capture && seen.video_system_audio && seen.video_microphone);
|
||||
}
|
||||
|
||||
Settings_Seen :: struct {
|
||||
theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys, ocr_model: bool;
|
||||
video_fps, video_cursor, video_capture: bool;
|
||||
video_fps, video_cursor, video_capture, video_system_audio, video_microphone: bool;
|
||||
}
|
||||
|
||||
// Of the file's bytes as last loaded or written, so a watcher event for our own save, or for an
|
||||
@ -520,6 +522,10 @@ settings_parse :: (path: string, file_data: string, out: *App_User_Settings) ->
|
||||
seen.video_fps = settings_read_int(*out.video_fps, path, line_number, key, value, VIDEO_FPS_MIN, VIDEO_FPS_MAX);
|
||||
} else if section == "[video]" && key == "cursor" {
|
||||
seen.video_cursor = settings_read_bool(*out.video_cursor, path, line_number, key, value);
|
||||
} else if section == "[video]" && key == "system_audio" {
|
||||
seen.video_system_audio = settings_read_bool(*out.video_system_audio, path, line_number, key, value);
|
||||
} else if section == "[video]" && key == "microphone" {
|
||||
seen.video_microphone = settings_read_bool(*out.video_microphone, path, line_number, key, value);
|
||||
} else if section == "[video]" && key == "capture" {
|
||||
capture := to_lower_copy(settings_unquote(value),, temp);
|
||||
if capture == "auto" || capture == "dxgi" || capture == "wgc" {
|
||||
@ -664,6 +670,8 @@ app_save_user_settings_file :: () -> bool {
|
||||
if seen.event_log app_user_settings.event_log_capacity = on_disk.event_log_capacity;
|
||||
if seen.video_fps app_user_settings.video_fps = on_disk.video_fps;
|
||||
if seen.video_cursor app_user_settings.video_cursor = on_disk.video_cursor;
|
||||
if seen.video_system_audio app_user_settings.video_system_audio = on_disk.video_system_audio;
|
||||
if seen.video_microphone app_user_settings.video_microphone = on_disk.video_microphone;
|
||||
if seen.video_capture {
|
||||
free(app_user_settings.video_capture,, context.default_allocator);
|
||||
app_user_settings.video_capture = copy_string(on_disk.video_capture,, context.default_allocator);
|
||||
@ -691,10 +699,13 @@ app_save_user_settings_file :: () -> bool {
|
||||
append(*builder, "[video]\n");
|
||||
append(*builder, "# Screen recording (Ctrl+PrintScreen; Windows). fps: frames a second at most, written only when the\n");
|
||||
append(*builder, "# screen changes. cursor: true or false. capture: auto (Desktop Duplication, else\n");
|
||||
append(*builder, "# Windows.Graphics.Capture), dxgi or wgc.\n");
|
||||
append(*builder, "# Windows.Graphics.Capture), dxgi or wgc. system_audio: what the computer plays; microphone: the\n");
|
||||
append(*builder, "# default input device. Each true or false.\n");
|
||||
print_to_builder(*builder, "fps: %\n", app_user_settings.video_fps);
|
||||
print_to_builder(*builder, "cursor: %\n", ifx app_user_settings.video_cursor then "true" else "false");
|
||||
print_to_builder(*builder, "capture: %\n\n", ifx app_user_settings.video_capture.count then app_user_settings.video_capture else "auto");
|
||||
print_to_builder(*builder, "capture: %\n", ifx app_user_settings.video_capture.count then app_user_settings.video_capture else "auto");
|
||||
print_to_builder(*builder, "system_audio: %\n", ifx app_user_settings.video_system_audio then "true" else "false");
|
||||
print_to_builder(*builder, "microphone: %\n\n", ifx app_user_settings.video_microphone then "true" else "false");
|
||||
append(*builder, "[logging]\n");
|
||||
append(*builder, "# Entries the Event Log keeps in memory. Past it the oldest are dropped; the log file keeps every line.\n");
|
||||
print_to_builder(*builder, "event_log_capacity: %\n", app_user_settings.event_log_capacity);
|
||||
|
||||
@ -21,6 +21,9 @@
|
||||
// -animate moves a square across the region the whole time, so the screen changes every
|
||||
// frame: for measuring throughput.
|
||||
// -remux IN OUT a recording's fragmented MP4 made a regular one, as the app does, and nothing else.
|
||||
// -audio WHICH records sound too: system (what the computer plays), mic, or both.
|
||||
// -sync-test every 2 s a white square flashes on the region and a 1 kHz beep plays; then the
|
||||
// flashes and beeps are found in the file and their offsets given, and how they drift.
|
||||
// -check FILE reads FILE back with FFmpeg and says what it holds, and nothing else: for a
|
||||
// recording cut short by a crash.
|
||||
// -snapshot DIR the last frame captured (captured.png) and a frame decoded back from the file
|
||||
@ -41,6 +44,7 @@ main :: () {
|
||||
list_only, cuda, exclude_test, dpi_unaware, animate, border_test, cursor_test := false, false, false, false, false, false, false;
|
||||
snapshot_folder: string;
|
||||
pause_seconds: float64;
|
||||
sync_test := false;
|
||||
i := 1;
|
||||
number :: (index: s64) -> s32 #expand { return cast(s32) string_to_int(`args[index]); }
|
||||
while i < args.count {
|
||||
@ -69,6 +73,14 @@ main :: () {
|
||||
if a == "-simulate-unsupported" { options.testing |= .DXGI_UNSUPPORTED; i += 1; continue; }
|
||||
if a == "-poll" { options.testing |= .WGC_POLL; i += 1; continue; }
|
||||
if a == "-lose" { options.testing |= .LOSE_DUPLICATION; i += 1; continue; }
|
||||
if a == "-sync-test" { sync_test = true; i += 1; continue; }
|
||||
if a == "-audio" && more >= 1 {
|
||||
which := args[i + 1];
|
||||
options.system_audio = which == "system" || which == "both";
|
||||
options.microphone = which == "mic" || which == "both";
|
||||
if !options.system_audio && !options.microphone { print("-audio takes system, mic or both\n"); exit(2); }
|
||||
i += 2; continue;
|
||||
}
|
||||
if a == "-pause" && more >= 1 { pause_seconds = string_to_float64(args[i + 1]); i += 2; continue; }
|
||||
if a == "-cursor-test" { cursor_test = true; i += 1; continue; }
|
||||
if a == "-capture" && more >= 1 {
|
||||
@ -149,6 +161,15 @@ main :: () {
|
||||
cursor_at = .{region.left + (region.right - region.left) / 2, region.top + (region.bottom - region.top) / 2};
|
||||
SetCursorPos(cursor_at.x, cursor_at.y);
|
||||
}
|
||||
flash: HWND;
|
||||
beep: [] u8;
|
||||
next_flash: float64 = 1;
|
||||
if sync_test {
|
||||
beep = beep_wav();
|
||||
side := min(cast(s32) 200, (region.right - region.left) / 3, (region.bottom - region.top) / 3);
|
||||
flash = square_window("record-probe flash", 0x00FFFFFF, region.left + (region.right - region.left - side) / 2, region.top + (region.bottom - region.top - side) / 2, side);
|
||||
ShowWindow(flash, SW_HIDE);
|
||||
}
|
||||
mover: HWND;
|
||||
if animate mover = square_window("record-probe animate", 0x0000A5FF, region.left, region.top, 120);
|
||||
|
||||
@ -173,6 +194,16 @@ main :: () {
|
||||
print(" % s: % captured, % sent (% repeats), % written, % dropped, % lost, % KB\n",
|
||||
FF(to_float64_seconds(last_print - start), 0), s.captured, s.emitted, s.repeats, s.written, s.dropped, s.lost, s.bytes / 1024);
|
||||
}
|
||||
if flash { // a flash and a beep together every 2 s, the flash for 200 ms
|
||||
elapsed := to_float64_seconds(current_time_monotonic() - start);
|
||||
if elapsed >= next_flash {
|
||||
ShowWindow(flash, SW_SHOWNOACTIVATE);
|
||||
PlaySoundW(beep.data, null, SND_MEMORY | SND_ASYNC);
|
||||
next_flash += 2;
|
||||
} else if elapsed >= next_flash - 2 + 0.2 {
|
||||
ShowWindow(flash, SW_HIDE);
|
||||
}
|
||||
}
|
||||
if pause_seconds > 0 {
|
||||
elapsed := to_float64_seconds(current_time_monotonic() - start);
|
||||
should := elapsed >= 2 && elapsed < 2 + pause_seconds;
|
||||
@ -189,6 +220,7 @@ main :: () {
|
||||
|
||||
last, width, height := record_last_frame(*recorder);
|
||||
if mover DestroyWindow(mover);
|
||||
if flash DestroyWindow(flash);
|
||||
if cursor_test SetCursorPos(saved_cursor.x, saved_cursor.y);
|
||||
stopped, stop_error := record_stop(*recorder);
|
||||
s := recorder.stats;
|
||||
@ -208,6 +240,29 @@ main :: () {
|
||||
print(" each: copy % (per image), read back % , encode % (per frame)\n",
|
||||
ms_each(s.copy_ticks, s.captured), ms_each(s.readback_ticks, s.emitted), ms_each(s.encode_ticks, s.written));
|
||||
print(" file: % KB, % MB a minute\n", file_size / 1024, FF(cast(float64) file_size / (1024 * 1024) / s.seconds * 60, 1));
|
||||
if options.system_audio || options.microphone {
|
||||
print(" sound: %; % AAC frames%\n", ifx s.audio then s.audio else "none opened", s.audio_frames, ifx s.audio_errors then tprint(" (%)", s.audio_errors) else "");
|
||||
sound := sound_report(options.path);
|
||||
if sound.found print(" in file: % s of sound, % dB RMS, % onsets\n", FF(sound.seconds, 2), FF(sound.rms_db, 1), sound.onsets.count);
|
||||
else print(" in file: no sound stream %\n", sound.error);
|
||||
if sync_test && sound.found {
|
||||
flashes := flash_times(options.path);
|
||||
print(" sync: % flashes, % beeps\n", flashes.count, sound.onsets.count);
|
||||
offsets: [..] float64;
|
||||
offsets.allocator = temp;
|
||||
for flash_time: flashes {
|
||||
best: float64 = 1000000000.0;
|
||||
for sound.onsets if abs(it - flash_time) < abs(best) best = it - flash_time;
|
||||
if abs(best) < 0.5 array_add(*offsets, best);
|
||||
}
|
||||
if offsets.count {
|
||||
text: [..] string; text.allocator = temp;
|
||||
for offsets array_add(*text, tprint("%", FF(it * 1000, 0)));
|
||||
print(" offsets: beep after flash, ms: %\n", join(..text, " ",, temp));
|
||||
print(" drift: % ms from the first to the last\n", FF((offsets[offsets.count - 1] - offsets[0]) * 1000, 0));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if exclude_test {
|
||||
excluded_rect := window_rect_in(excluded, region);
|
||||
@ -405,6 +460,134 @@ FF :: (value: float64, decimals: s64) -> FormatFloat #expand {
|
||||
SetProcessDpiAwarenessContext :: (value: HANDLE) -> BOOL #foreign user32_dll;
|
||||
SetWindowDisplayAffinity :: (window: HWND, affinity: u32) -> BOOL #foreign user32_dll;
|
||||
SetCursorPos :: (x: s32, y: s32) -> BOOL #foreign user32_dll;
|
||||
|
||||
// ---------------------------------------------------------------- sound and sync
|
||||
|
||||
// A beep of 1 kHz for 150 ms, as a WAV in memory, for PlaySound.
|
||||
beep_wav :: () -> [] u8 {
|
||||
RATE :: 48000;
|
||||
COUNT :: RATE * 150 / 1000;
|
||||
data := NewArray(44 + COUNT * 2, u8,, context.default_allocator);
|
||||
put32 :: (at: s64, v: u32) #expand { for 0..3 `data[at + it] = cast,trunc(u8) (v >> (8 * it)); }
|
||||
put16 :: (at: s64, v: u16) #expand { `data[at] = cast,trunc(u8) v; `data[at + 1] = cast,trunc(u8) (v >> 8); }
|
||||
memcpy(data.data, "RIFF".data, 4); put32(4, cast(u32) (36 + COUNT * 2)); memcpy(data.data + 8, "WAVEfmt ".data, 8);
|
||||
put32(16, 16); put16(20, 1); put16(22, 1); put32(24, RATE); put32(28, RATE * 2); put16(32, 2); put16(34, 16);
|
||||
memcpy(data.data + 36, "data".data, 4); put32(40, COUNT * 2);
|
||||
for i: 0..COUNT - 1 {
|
||||
v := cast(s16) (sin(cast(float) i * 2 * PI * 1000 / RATE) * 20000);
|
||||
put16(44 + i * 2, cast,no_check(u16) v);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
// What the file's sound holds: its length, its level, and when it goes from quiet to loud (onsets).
|
||||
Sound_Report :: struct { found: bool; seconds: float64; rms_db: float64; onsets: [..] float64; error: string; }
|
||||
|
||||
sound_report :: (path: string) -> Sound_Report {
|
||||
report: Sound_Report;
|
||||
report.onsets.allocator = temp;
|
||||
format: *AVFormatContext;
|
||||
if avformat_open_input(*format, temp_c_string(path), null, null) < 0 { report.error = "cannot open"; return report; }
|
||||
defer avformat_close_input(*format);
|
||||
avformat_find_stream_info(format, null);
|
||||
index := -1;
|
||||
for 0..format.nb_streams - 1 if format.streams[it].codecpar.codec_type == .AUDIO { index = it; break; }
|
||||
if index < 0 return report;
|
||||
report.found = true;
|
||||
stream := format.streams[index];
|
||||
codec := avcodec_find_decoder(stream.codecpar.codec_id);
|
||||
c := avcodec_alloc_context3(codec);
|
||||
defer avcodec_free_context(*c);
|
||||
avcodec_parameters_to_context(c, stream.codecpar);
|
||||
if avcodec_open2(c, codec, null) < 0 { report.error = "no decoder"; return report; }
|
||||
packet := av_packet_alloc(); defer av_packet_free(*packet);
|
||||
frame := av_frame_alloc(); defer av_frame_free(*frame);
|
||||
|
||||
WINDOW :: 480; // 10 ms at 48 kHz
|
||||
sum_all: float64; count_all: s64;
|
||||
window_sum: float64; window_count: s64; window_start: float64 = -1;
|
||||
quiet_for: s64 = 100; // windows in a row below the threshold
|
||||
first_time: float64 = -1; last_time: float64;
|
||||
handle :: () #expand {
|
||||
while avcodec_receive_frame(`c, `frame) >= 0 {
|
||||
t := cast(float64) `frame.pts * `stream.time_base.num / `stream.time_base.den;
|
||||
if `first_time < 0 `first_time = t;
|
||||
left := cast(*float) `frame.data[0];
|
||||
for i: 0..`frame.nb_samples - 1 {
|
||||
v := cast(float64) left[i];
|
||||
`sum_all += v * v; `count_all += 1;
|
||||
if `window_start < 0 `window_start = t + cast(float64) i / 48000;
|
||||
`window_sum += v * v; `window_count += 1;
|
||||
if `window_count == WINDOW {
|
||||
rms := sqrt(`window_sum / WINDOW);
|
||||
if rms > 0.05 {
|
||||
if `quiet_for >= 20 array_add(*`report.onsets, `window_start); // after 200 ms of quiet
|
||||
`quiet_for = 0;
|
||||
} else {
|
||||
`quiet_for += 1;
|
||||
}
|
||||
`window_sum, `window_count, `window_start = 0, 0, -1;
|
||||
}
|
||||
}
|
||||
`last_time = t + cast(float64) `frame.nb_samples / 48000;
|
||||
}
|
||||
}
|
||||
while av_read_frame(format, packet) >= 0 {
|
||||
if packet.stream_index == index { avcodec_send_packet(c, packet); handle(); }
|
||||
av_packet_unref(packet);
|
||||
}
|
||||
avcodec_send_packet(c, null);
|
||||
handle();
|
||||
report.seconds = last_time - max(first_time, 0);
|
||||
report.rms_db = ifx count_all && sum_all > 0 then 10 * log(sum_all / count_all) / log(10.0) else -200;
|
||||
return report;
|
||||
}
|
||||
|
||||
// When the video's middle turns white: the sync test's flashes, as times in seconds.
|
||||
flash_times :: (path: string) -> [..] float64 {
|
||||
times: [..] float64;
|
||||
times.allocator = temp;
|
||||
format: *AVFormatContext;
|
||||
if avformat_open_input(*format, temp_c_string(path), null, null) < 0 return times;
|
||||
defer avformat_close_input(*format);
|
||||
avformat_find_stream_info(format, null);
|
||||
index := -1;
|
||||
for 0..format.nb_streams - 1 if format.streams[it].codecpar.codec_type == .VIDEO { index = it; break; }
|
||||
if index < 0 return times;
|
||||
stream := format.streams[index];
|
||||
codec := avcodec_find_decoder(stream.codecpar.codec_id);
|
||||
c := avcodec_alloc_context3(codec);
|
||||
defer avcodec_free_context(*c);
|
||||
avcodec_parameters_to_context(c, stream.codecpar);
|
||||
if avcodec_open2(c, codec, null) < 0 return times;
|
||||
packet := av_packet_alloc(); defer av_packet_free(*packet);
|
||||
frame := av_frame_alloc(); defer av_frame_free(*frame);
|
||||
was_white := false;
|
||||
handle :: () #expand {
|
||||
while avcodec_receive_frame(`c, `frame) >= 0 {
|
||||
// The luma of the middle 32 x 32 (the first plane in every YUV format the encoders give).
|
||||
w, h := `frame.width, `frame.height;
|
||||
sum: s64;
|
||||
for y: h / 2 - 16..h / 2 + 15 for x: w / 2 - 16..w / 2 + 15 sum += `frame.data[0][y * `frame.linesize[0] + x];
|
||||
white := sum / 1024 > 200;
|
||||
if white && !`was_white array_add(*`times, cast(float64) `frame.best_effort_timestamp * `stream.time_base.num / `stream.time_base.den);
|
||||
`was_white = white;
|
||||
}
|
||||
}
|
||||
while av_read_frame(format, packet) >= 0 {
|
||||
if packet.stream_index == index { avcodec_send_packet(c, packet); handle(); }
|
||||
av_packet_unref(packet);
|
||||
}
|
||||
avcodec_send_packet(c, null);
|
||||
handle();
|
||||
return times;
|
||||
}
|
||||
|
||||
PlaySoundW :: (sound: *void, module: *void, flags: u32) -> s32 #foreign winmm_dll;
|
||||
winmm_dll :: #library,system "winmm";
|
||||
SND_ASYNC :: 0x1;
|
||||
SND_MEMORY :: 0x4;
|
||||
|
||||
user32_dll :: #library,system "user32";
|
||||
|
||||
#import "Basic";
|
||||
@ -417,3 +600,4 @@ user32_dll :: #library,system "user32";
|
||||
#import "Screen_Record";
|
||||
#import "Image_Decode";
|
||||
#import "stb_image_write";
|
||||
#import "ffmpeg";
|
||||
|
||||
@ -68,7 +68,7 @@
|
||||
|
||||
<header>
|
||||
<h1>Screen video capture in mcap</h1>
|
||||
<p>Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · <strong>phases 1 to 4 built, mcap 0.1.6</strong></p>
|
||||
<p>Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · <strong>all five phases built, mcap 0.1.8</strong></p>
|
||||
</header>
|
||||
|
||||
<div class="summary">
|
||||
@ -83,6 +83,7 @@
|
||||
<li><span class="tag decided">Done</span> <strong>Phase 1</strong> (2026-10-10, mcap 0.1.3): <code>modules/Screen_Record</code> and <code>record-probe</code>. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see <a href="#phases">phase 1's result</a>).</li>
|
||||
<li><span class="tag decided">Done</span> <strong>Phase 2</strong> (2026-10-10, mcap 0.1.4): the Windows.Graphics.Capture fallback. Automatic mode falls back to it on a simulated <code>DXGI_ERROR_UNSUPPORTED</code>. It records 4K at 60 fps with nothing dropped, draws the cursor exactly as Desktop Duplication's code does, and the yellow border can be turned off on Windows 11.</li>
|
||||
<li><span class="tag decided">Done</span> <strong>Phase 3</strong> (2026-10-10, mcap 0.1.5): recording in the app. Ctrl+PrintScreen opens the overlay's video mode; Ctrl+PrintScreen again stops; the recording becomes a video <code>.mcap</code> in Screen Captures. In the app, Desktop Duplication first failed with <code>DXGI_ERROR_UNSUPPORTED</code>: only the main thread was DPI aware. The capture thread now makes itself aware.</li>
|
||||
<li><span class="tag decided">Done</span> <strong>Phase 5</strong> (2026-10-10, mcap 0.1.8): sound. What the computer plays, and the microphone if asked for, are mixed in sync with the video into an AAC stream. A beep matched to a flash every 2 s stays within about 20 ms, without drifting.</li>
|
||||
<li><span class="tag decided">Done</span> <strong>Phase 4</strong> (2026-10-10, mcap 0.1.6): the border and toolbar, both kept out of the recording; pause, with no gap in the video; discard, confirmed in the toolbar; recovery when the screen is lost; leftover recordings offered at startup; the <code>[video]</code> settings.</li>
|
||||
<li><span class="tag later">Later</span> Audio is phase 5. Recording one window, and regions that span two screens, come after.</li>
|
||||
</ul>
|
||||
@ -466,9 +467,41 @@ Captured_Frame :: struct {
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 5: Audio</h3>
|
||||
<h3>Phase 5: Audio <span class="tag decided">Done</span></h3>
|
||||
<p>Section 8: <code>swresample</code> bindings, loopback and microphone, mixing, AAC.</p>
|
||||
<p><strong>Done when:</strong> a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.</p>
|
||||
<div class="note"><strong>Result (2026-10-10).</strong>
|
||||
<ul>
|
||||
<li><strong>Built:</strong>
|
||||
<ul>
|
||||
<li><code>modules/Screen_Record/audio_source.jai</code>: WASAPI for both sources. What the computer plays comes in loopback from the default output device; the microphone is the default input device.</li>
|
||||
<li>An audio thread in <code>module.jai</code> that mixes them.</li>
|
||||
<li>An AAC stream in <code>encoder.jai</code>, with the two threads' packets meeting in the file under a lock.</li>
|
||||
<li><code>system_audio</code> and <code>microphone</code> in <code>[video]</code>, defaulting to true and false.</li>
|
||||
</ul></li>
|
||||
<li><strong>Sync, by measurement:</strong> <code>record-probe -sync-test</code> flashes a white square and plays a 1 kHz beep together every 2 s, then finds both in the file.
|
||||
<ul>
|
||||
<li>20 s with system sound: 10 flashes, 10 beeps, each beep within 0 to 21 ms of its flash, with no drift (−10 ms first to last, within the spread).</li>
|
||||
<li>Both sources mixed: within 5 to 18 ms.</li>
|
||||
<li>With a 6 s pause: sound and video lose the same 6 s (14.1 s of sound, 14.2 s of video), and the beeps after the pause line up as before.</li>
|
||||
<li>3 minutes with system sound: 92 flashes, 90 beeps found. 85 beeps were within 3 to 24 ms of their flash, and from the first to the last the offset moved −14 ms: no drift. Five were 53 to 147 ms off, in two clusters about 40 s and 60 s in, near where the two beeps went missing. That points to Windows playing the test beep late or not at all then, but it isn't proven; a 10-minute run with someone talking, as the done-when asks, is still to do by hand.</li>
|
||||
</ul>
|
||||
The offsets include Windows' own delay in playing the beep, so they are an upper bound on the true offset.</li>
|
||||
<li><strong>In the app:</strong> <code>-record 6</code> recorded the monitor's speakers. The <code>.mcap</code>'s MP4 has its AAC stream, and its <code>capture.json</code> has <code>Audio: true</code> and the device's name. Your settings file gained <code>system_audio</code> and <code>microphone</code>.</li>
|
||||
<li><strong>Tests:</strong> 4 more checks, 30 in the recording group. An MP4 made with sound by the module's encoder keeps its AAC stream through the remux into the library, and <code>capture.json</code> names its source. Every suite passes.</li>
|
||||
</ul>
|
||||
<p><strong>Changes from the plan:</strong></p>
|
||||
<ul>
|
||||
<li><strong>No <code>swresample</code>.</strong> WASAPI converts any device to 48 kHz stereo float itself (<code>AUTOCONVERTPCM</code>), so no bindings were generated. A device that won't convert is taken as it is if it gives 48 kHz float, its channels made stereo here.</li>
|
||||
<li><strong>How the timeline holds together:</strong>
|
||||
<ul>
|
||||
<li>Each packet is placed by the time Windows says it was heard, on the video's own timeline: the same time 0, less the same pauses. <code>record_pause</code> now keeps one account of time paused for both threads.</li>
|
||||
<li>A packet within 30 ms of where its source left off follows on exactly, so timing jitter doesn't leave clicks.</li>
|
||||
<li>A source that sends nothing, as loopback does while nothing plays, is silence, and the mix moves on without it 200 ms late.</li>
|
||||
</ul></li>
|
||||
<li><strong>Not in the toolbar.</strong> The sources are chosen in <code>[video]</code>, not switched in the toolbar. An MP4 can't gain a sound stream once its header is written, so a source can't be added mid-recording.</li>
|
||||
<li><strong>A lost device</strong> (a headset unplugged, the default changed) is opened again from the new default once a second.</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
|
||||
<h2 id="risks">10. Risks</h2>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user