diff --git a/README.md b/README.md index 6fa53fa..94976bf 100644 --- a/README.md +++ b/README.md @@ -187,7 +187,9 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S go of it starts nothing; drag again. - Escape or a right click cancels. 3. **Recording starts** at once, at up to 30 frames a second, cursor included. A frame is written - only when the screen changes, so a still screen costs almost nothing. + only when the screen changes, so a still screen costs almost nothing. **Sound:** what the + computer plays is recorded with it, and the microphone too if `microphone` is set in `[video]` + (off by default). - **A red border** is drawn just outside the region. - **A toolbar** beside it shows the time, the size and any frames dropped, with **Pause** (the time paused is left out of the video), **Stop**, and **Discard**, which asks first. @@ -206,8 +208,12 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S recording carries on once it is back. - A recording a crash cut short is offered at the next start: into the library, or deleted. It keeps all but its last two seconds or so. -- **Settings:** `[video]` in the settings file: `fps` (30), `cursor` (true), and `capture`: `auto` - (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`. +- **Settings:** `[video]` in the settings file: + - `fps` (30) and `cursor` (true). + - `capture`: `auto` (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`. + - `system_audio` (true): what the computer plays, from the default output device. + - `microphone` (false): the default input device. + - Sound is AAC at 160 kb/s. A device unplugged mid-recording is picked up again from the new default. - **PrintScreen while recording** takes the whole screen under the cursor at once, without freezing the screens, which would show in the recording. - **The key** is `record_region` in `[global_hotkeys]`. diff --git a/check_targets.jai b/check_targets.jai index 20318ec..d6fba91 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.1.7"; +VERSION :: "0.1.8"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 0e75081..d86f3ed 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.1.7"; +VERSION :: "0.1.8"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; diff --git a/modules/Capture_File/README.md b/modules/Capture_File/README.md index 6e7f165..5c229ea 100644 --- a/modules/Capture_File/README.md +++ b/modules/Capture_File/README.md @@ -136,6 +136,9 @@ Capture: "DXGI DuplicateOutput1 on NVIDIA GeForce RTX 5080 Laptop GPU" or "Windows.Graphics.Capture on ..., border off" DroppedFrames: 0 frames the encoder fell too far behind for Paused: 10.2 seconds paused, left out of the video; only when it was paused +Audio: true whether it has sound (AAC) +AudioSources: "SAMSUNG (NVIDIA High Definition Audio); Microphone Array (Realtek(R) Audio)" + the devices heard; only when it has sound ``` A recording a crash cut short, put in the library at the next start, has no window, Encoder or diff --git a/modules/Screen_Record/audio_source.jai b/modules/Screen_Record/audio_source.jai new file mode 100644 index 0000000..9fe5484 --- /dev/null +++ b/modules/Screen_Record/audio_source.jai @@ -0,0 +1,263 @@ +// Sound for a recording, with WASAPI: what the computer plays (the default output device, in loopback) +// and the microphone (the default input device). Each is asked for in shared mode as 48 kHz stereo +// float, which Windows converts any device's own format to (AUTOCONVERTPCM), so nothing here +// resamples. Where a device will not convert, its own format is taken if it is 48 kHz float, and its +// channels are made stereo here. +// +// Each packet comes with the QueryPerformanceCounter time its first sample was heard, which places it +// on the recording's timeline (module.jai's audio thread mixes them). Loopback sends nothing while +// nothing plays: those gaps are silence. A device unplugged mid-recording (a headset) is opened +// again, from the new default, once a second. + +AUDIO_RATE :: 48000; +AUDIO_CHANNELS :: 2; + +Audio_Source_Kind :: enum u8 { SYSTEM; MICROPHONE; } + +Audio_Source :: struct { + kind: Audio_Source_Kind; + name: string; // "Speakers (Realtek(R) Audio)"; heap + client: *Audio_Client; + capture: *Audio_Capture_Client; + channels: s32; // as the device gives them: 2, unless it would not convert + open: bool; + lost_at: s64; // when it was lost (ticks), to open it again a second on; 0 while open + error: string; // why it could not be opened; heap +} + +Audio_Packet :: struct { + samples: [] float; // interleaved stereo, AUDIO_CHANNELS a frame; empty for silence + frames: s64; + ticks: s64; // when its first frame was heard, in QueryPerformanceCounter ticks + silent: bool; +} + +// The default device of its kind, started. False with why: none, or it will not open. +audio_open :: (s: *Audio_Source, kind: Audio_Source_Kind) -> ok: bool, error: string { + s.kind = kind; + enumerator: *Device_Enumerator; + hr := audio_CoCreateInstance(*CLSID_MM_DEVICE_ENUMERATOR, null, CLSCTX_ALL, *IID_MM_DEVICE_ENUMERATOR, xx *enumerator); + if FAILED(hr) return false, hresult_text("the audio device list", hr); + defer com_release(*enumerator); + device: *Audio_Device; + flow: s32 = ifx kind == .SYSTEM then E_RENDER else E_CAPTURE; + hr = enumerator.vtable.GetDefaultAudioEndpoint(enumerator, flow, E_CONSOLE, *device); + if FAILED(hr) return false, ifx kind == .SYSTEM then "no sound output device" else "no microphone"; + defer com_release(*device); + free(s.name); + s.name = copy_string(audio_device_name(device)); + + hr = device.vtable.Activate(device, *IID_AUDIO_CLIENT, CLSCTX_ALL, null, xx *s.client); + if FAILED(hr) return false, hresult_text(tprint("% (%)", s.name, "IAudioClient"), hr); + + flags: u32 = AUTOCONVERTPCM | SRC_DEFAULT_QUALITY; + if kind == .SYSTEM flags |= STREAMFLAGS_LOOPBACK; + wanted := WAVEFORMATEX.{wFormatTag = WAVE_FORMAT_IEEE_FLOAT, nChannels = AUDIO_CHANNELS, nSamplesPerSec = AUDIO_RATE, + nAvgBytesPerSec = AUDIO_RATE * AUDIO_CHANNELS * 4, nBlockAlign = AUDIO_CHANNELS * 4, wBitsPerSample = 32}; + BUFFER :: 2_000_000; // 200 ms, in 100 ns: room for the audio thread being late + hr = s.client.vtable.Initialize(s.client, SHAREMODE_SHARED, flags, BUFFER, 0, *wanted, null); + s.channels = AUDIO_CHANNELS; + if FAILED(hr) { + // No conversion here: the device's own format, if it is 48 kHz float, its channels made stereo after. + mix: *WAVEFORMATEX; + if FAILED(s.client.vtable.GetMixFormat(s.client, *mix)) return false, hresult_text(tprint("% will not give 48 kHz stereo", s.name), hr); + defer audio_CoTaskMemFree(mix); + is_float := mix.wFormatTag == WAVE_FORMAT_IEEE_FLOAT || (mix.wFormatTag == WAVE_FORMAT_EXTENSIBLE && mix.wBitsPerSample == 32); + if !is_float || mix.nSamplesPerSec != AUDIO_RATE + return false, tprint("% gives % Hz %, not 48 kHz float, and will not convert", s.name, mix.nSamplesPerSec, ifx is_float then "float" else "PCM"); + com_release(*s.client); + hr = device.vtable.Activate(device, *IID_AUDIO_CLIENT, CLSCTX_ALL, null, xx *s.client); + if FAILED(hr) return false, hresult_text(s.name, hr); + hr = s.client.vtable.Initialize(s.client, SHAREMODE_SHARED, flags & ~(AUTOCONVERTPCM | SRC_DEFAULT_QUALITY), BUFFER, 0, mix, null); + if FAILED(hr) return false, hresult_text(tprint("% (Initialize)", s.name), hr); + s.channels = mix.nChannels; + } + hr = s.client.vtable.GetService(s.client, *IID_AUDIO_CAPTURE_CLIENT, xx *s.capture); + if FAILED(hr) return false, hresult_text(tprint("% (IAudioCaptureClient)", s.name), hr); + hr = s.client.vtable.Start(s.client); + if FAILED(hr) return false, hresult_text(tprint("% (Start)", s.name), hr); + s.open = true; + s.lost_at = 0; + return true, ""; +} + +// The packets waiting, each as stereo float (temp), oldest first. lost: the device went away +// (unplugged, or the default changed): audio_close, and open it again later. +audio_take :: (s: *Audio_Source, frequency: s64) -> [] Audio_Packet, lost: bool { + packets: [..] Audio_Packet; + packets.allocator = temp; + while true { + next: u32; + hr := s.capture.vtable.GetNextPacketSize(s.capture, *next); + if hresult_is(hr, AUDCLNT_E_DEVICE_INVALIDATED) return packets, true; + if FAILED(hr) || !next break; + data: *u8; + frames, flags: u32; + device_position, qpc_position: u64; // qpc_position: QueryPerformanceCounter time, in 100 ns + hr = s.capture.vtable.GetBuffer(s.capture, *data, *frames, *flags, *device_position, *qpc_position); + if hresult_is(hr, AUDCLNT_E_DEVICE_INVALIDATED) return packets, true; + if FAILED(hr) || hresult_is(hr, AUDCLNT_S_BUFFER_EMPTY) break; + + packet: Audio_Packet; + packet.frames = frames; + packet.ticks = cast(s64) (cast(float64) qpc_position * frequency / 10_000_000.0); + packet.silent = (flags & BUFFERFLAGS_SILENT) != 0; + if !packet.silent && frames { + packet.samples = NewArray(cast(s64) frames * AUDIO_CHANNELS, float, initialized = false,, temp); + input := cast(*float) data; + if s.channels == AUDIO_CHANNELS { + memcpy(packet.samples.data, input, cast(s64) frames * AUDIO_CHANNELS * size_of(float)); + } else { + // Mono to both sides; more channels than two: the first two (front left and right). + channels := cast(s64) s.channels; + for f: 0..cast(s64) frames - 1 { + left := input[f * channels]; + right := ifx channels > 1 then input[f * channels + 1] else left; + packet.samples[f * 2] = left; + packet.samples[f * 2 + 1] = right; + } + } + } + s.capture.vtable.ReleaseBuffer(s.capture, frames); + array_add(*packets, packet); + } + return packets, false; +} + +audio_close :: (s: *Audio_Source) { + if s.client && s.open s.client.vtable.Stop(s.client); + com_release(*s.capture); + com_release(*s.client); + s.open = false; +} + +audio_free :: (s: *Audio_Source) { + audio_close(s); + free(s.name); + free(s.error); + s.name, s.error = "", ""; +} + +// COM on this thread, in the multithreaded apartment, as WASAPI's objects are used from it only. +audio_thread_start :: () { + audio_CoInitializeEx(null, COINIT_MULTITHREADED); +} + +audio_thread_end :: () { + audio_CoUninitialize(); +} + +#scope_file + +audio_device_name :: (device: *Audio_Device) -> string { + store: *Property_Store; + if FAILED(device.vtable.OpenPropertyStore(device, STGM_READ, *store)) return ""; + defer com_release(*store); + value: PROPVARIANT; + if FAILED(store.vtable.GetValue(store, *PKEY_DEVICE_FRIENDLY_NAME, *value)) return ""; + defer audio_PropVariantClear(*value); + VT_LPWSTR :: 31; + if value.vt != VT_LPWSTR || !value.pointer return ""; + name, ok := wide_to_utf8(cast(*u16) value.pointer,, temp); + return ifx ok then name else ""; +} + +// ---------------------------------------------------------------- bindings + +ole32_dll :: #library,system "ole32"; + +audio_CoInitializeEx :: (reserved: *void, flags: u32) -> HRESULT #foreign ole32_dll "CoInitializeEx"; +audio_CoUninitialize :: () #foreign ole32_dll "CoUninitialize"; +audio_CoCreateInstance :: (clsid: *GUID, outer: *void, class_context: u32, iid: *GUID, out: **void) -> HRESULT #foreign ole32_dll "CoCreateInstance"; +audio_CoTaskMemFree :: (memory: *void) #foreign ole32_dll "CoTaskMemFree"; +audio_PropVariantClear :: (value: *PROPVARIANT) -> HRESULT #foreign ole32_dll "PropVariantClear"; + +COINIT_MULTITHREADED : u32 : 0; +CLSCTX_ALL : u32 : 0x17; +E_RENDER : s32 : 0; +E_CAPTURE : s32 : 1; +E_CONSOLE : s32 : 0; +STGM_READ : u32 : 0; +SHAREMODE_SHARED : s32 : 0; +STREAMFLAGS_LOOPBACK : u32 : 0x00020000; +AUTOCONVERTPCM : u32 : 0x80000000; +SRC_DEFAULT_QUALITY : u32 : 0x08000000; +BUFFERFLAGS_SILENT : u32 : 0x2; +WAVE_FORMAT_IEEE_FLOAT : u16 : 3; +WAVE_FORMAT_EXTENSIBLE : u16 : 0xFFFE; +AUDCLNT_S_BUFFER_EMPTY :: 0x08890001; +AUDCLNT_E_DEVICE_INVALIDATED :: 0x88890004; + +CLSID_MM_DEVICE_ENUMERATOR := #run guid("BCDE0395-E52F-467C-8E3D-C4579291692E"); +IID_MM_DEVICE_ENUMERATOR := #run guid("A95664D2-9614-4F35-A746-DE8DB63617E6"); +IID_AUDIO_CLIENT := #run guid("1CB9AD4C-DBFA-4C32-B178-C2F568A703B2"); +IID_AUDIO_CAPTURE_CLIENT := #run guid("C8ADBD64-E71E-48A0-A4DE-185C395CD317"); + +PROPERTYKEY :: struct { fmtid: GUID; pid: u32; } +PKEY_DEVICE_FRIENDLY_NAME := PROPERTYKEY.{#run guid("A45C254E-DF1C-4EFD-8020-67D146A850E0"), 14}; + +// A PROPVARIANT, as far as a string in it: its type, then (after 6 reserved bytes) the pointer. +PROPVARIANT :: struct { + vt: u16; + reserved: [3] u16; + pointer: *void; + padding: *void; +} + +WAVEFORMATEX :: struct { + wFormatTag: u16; + nChannels: u16; + nSamplesPerSec: u32; + nAvgBytesPerSec: u32; + nBlockAlign: u16; + wBitsPerSample: u16; + cbSize: u16; +} + +Unknown_Methods :: struct { + QueryInterface: (this: *void, iid: *GUID, out: **void) -> HRESULT #c_call; + AddRef: (this: *void) -> u32 #c_call; + Release: (this: *void) -> u32 #c_call; +} + +Device_Enumerator :: struct { vtable: *struct { + using base: Unknown_Methods; + EnumAudioEndpoints: *void; + GetDefaultAudioEndpoint: (this: *Device_Enumerator, flow: s32, role: s32, out: **Audio_Device) -> HRESULT #c_call; +}; } + +Audio_Device :: struct { vtable: *struct { + using base: Unknown_Methods; + Activate: (this: *Audio_Device, iid: *GUID, class_context: u32, parameters: *void, out: **void) -> HRESULT #c_call; + OpenPropertyStore: (this: *Audio_Device, access: u32, out: **Property_Store) -> HRESULT #c_call; +}; } + +Property_Store :: struct { vtable: *struct { + using base: Unknown_Methods; + GetCount: *void; + GetAt: *void; + GetValue: (this: *Property_Store, key: *PROPERTYKEY, out: *PROPVARIANT) -> HRESULT #c_call; +}; } + +Audio_Client :: struct { vtable: *struct { + using base: Unknown_Methods; + Initialize: (this: *Audio_Client, share_mode: s32, flags: u32, buffer_duration: s64, periodicity: s64, format: *WAVEFORMATEX, session: *GUID) -> HRESULT #c_call; + GetBufferSize: *void; + GetStreamLatency: *void; + GetCurrentPadding: *void; + IsFormatSupported: *void; + GetMixFormat: (this: *Audio_Client, out: **WAVEFORMATEX) -> HRESULT #c_call; + GetDevicePeriod: *void; + Start: (this: *Audio_Client) -> HRESULT #c_call; + Stop: (this: *Audio_Client) -> HRESULT #c_call; + Reset: *void; + SetEventHandle: *void; + GetService: (this: *Audio_Client, iid: *GUID, out: **void) -> HRESULT #c_call; +}; } + +Audio_Capture_Client :: struct { vtable: *struct { + using base: Unknown_Methods; + GetBuffer: (this: *Audio_Capture_Client, data: **u8, frames: *u32, flags: *u32, device_position: *u64, qpc_position: *u64) -> HRESULT #c_call; + ReleaseBuffer: (this: *Audio_Capture_Client, frames: u32) -> HRESULT #c_call; + GetNextPacketSize: (this: *Audio_Capture_Client, out: *u32) -> HRESULT #c_call; +}; } diff --git a/modules/Screen_Record/capture_gpu.jai b/modules/Screen_Record/capture_gpu.jai index 4b64b09..941f607 100644 --- a/modules/Screen_Record/capture_gpu.jai +++ b/modules/Screen_Record/capture_gpu.jai @@ -78,6 +78,26 @@ gpu_thread_dpi_aware :: () { user32_dll :: #library,system "user32"; SetThreadDpiAwarenessContext :: (value: HANDLE) -> HANDLE #foreign user32_dll; +// A GUID from its text, "A95664D2-9614-4F35-A746-DE8DB63617E6": for #run. +guid :: (text: string) -> GUID { + hex :: (s: string) -> u64 { + value: u64; + for 0..s.count - 1 { + c := s[it]; + digit: u64 = ifx c >= #char "a" then cast(u64) (c - #char "a" + 10) else ifx c >= #char "A" then cast(u64) (c - #char "A" + 10) else cast(u64) (c - #char "0"); + value = value * 16 + digit; + } + return value; + } + g: GUID; + g.Data1 = cast(u32) hex(slice(text, 0, 8)); + g.Data2 = cast(u16) hex(slice(text, 9, 4)); + g.Data3 = cast(u16) hex(slice(text, 14, 4)); + for 0..1 g.Data4[it] = cast(u8) hex(slice(text, 19 + it * 2, 2)); + for 0..5 g.Data4[it + 2] = cast(u8) hex(slice(text, 24 + it * 2, 2)); + return g; +} + com_release :: (p: *$T) { if p.* { IUnknown_Release(cast(*IUnknown) p.*); p.* = null; } } diff --git a/modules/Screen_Record/encoder.jai b/modules/Screen_Record/encoder.jai index cf570e4..f96c8aa 100644 --- a/modules/Screen_Record/encoder.jai +++ b/modules/Screen_Record/encoder.jai @@ -4,6 +4,9 @@ // BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with // swscale (BT.709, limited range, as tagged). // +// With sound, an AAC stream (FFmpeg's own encoder, 160 kb/s, 48 kHz stereo) beside it, written from the +// audio thread: the two streams' packets meet in the file under a lock (mux). +// // The file is written fragmented, a fragment at each keyframe, so one cut short by a crash still plays // up to its last fragment. Frames are written only when the screen changes, so a count of frames can // span any time: a keyframe is made every RECORD_KEYFRAME_SECONDS of time instead, which bounds what a @@ -11,6 +14,7 @@ RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"]; RECORD_KEYFRAME_SECONDS :: 2; +RECORD_AUDIO_BIT_RATE :: 160_000; Encoder :: struct { format: *AVFormatContext; @@ -26,11 +30,20 @@ Encoder :: struct { last_key_pts: s64 = -1; // the last keyframe asked for header_written: bool; bytes: s64; // of packets written + + // Sound, when asked for: AAC, written from the audio thread. + audio: *AVCodecContext; + audio_stream: *AVStream; + audio_frame: *AVFrame; + audio_packet: *AVPacket; + audio_frame_size: s32; // samples a frame: AAC's 1024 + mux: Mutex; // the video and audio threads' packets into the file + mux_ready: bool; } // Opens the file and the first encoder that will take width x height (wanted alone, when given). // Failures to open an encoder are listed in tried (temp), for the log. -encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string) -> ok: bool, error: string, tried: string { +encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string, audio := false) -> ok: bool, error: string, tried: string { av_log_set_level(AV_LOG_ERROR); e.width, e.height = width, height; result := avformat_alloc_output_context2(*e.format, null, "mp4", temp_c_string(path)); @@ -55,6 +68,12 @@ encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, w if !e.stream { encoder_free(e); return false, "FFmpeg cannot add a stream", tried_text; } avcodec_parameters_from_context(e.stream.codecpar, e.codec); e.stream.time_base = e.codec.time_base; + if audio { + audio_ok, audio_error := encoder_open_audio(e); + if !audio_ok { encoder_free(e); return false, audio_error, tried_text; } + } + init(*e.mux, "record mux"); + e.mux_ready = true; result = avio_open(*e.format.pb, temp_c_string(path), AVIO_FLAG_WRITE); if result < 0 { encoder_free(e); return false, av_error(tprint("FFmpeg cannot write %", path), result), tried_text; } @@ -111,12 +130,36 @@ encoder_write :: (e: *Encoder, pixels: *u8, pts: s64) -> ok: bool, error: string return drained, error; } -// Flushes the encoder, finishes the file and closes it. +// One AAC frame of sound: e.audio_frame_size frames of interleaved stereo float, at pts in samples. +// From the audio thread. +encoder_write_audio :: (e: *Encoder, samples: *float, pts: s64) -> ok: bool, error: string { + frame := e.audio_frame; + result := av_frame_make_writable(frame); + if result < 0 return false, av_error("FFmpeg cannot make a sound frame", result); + left := cast(*float) frame.data[0]; + right := cast(*float) frame.data[1]; + for 0..e.audio_frame_size - 1 { + left[it] = samples[it * 2]; + right[it] = samples[it * 2 + 1]; + } + frame.pts = pts; + result = avcodec_send_frame(e.audio, frame); + if result < 0 return false, av_error("the AAC encoder", result); + drained, error := encoder_drain(e, e.audio, e.audio_stream, e.audio_packet); + return drained, error; +} + +// Flushes the encoders, sound first, finishes the file and closes it. encoder_close :: (e: *Encoder) -> ok: bool, error: string { ok, error := true, ""; + if e.audio && e.header_written { + avcodec_send_frame(e.audio, null); + ok, error = encoder_drain(e, e.audio, e.audio_stream, e.audio_packet); + } if e.codec && e.header_written { avcodec_send_frame(e.codec, null); - ok, error = encoder_drain(e); + video_ok, video_error := encoder_drain(e); + if ok ok, error = video_ok, video_error; result := av_write_trailer(e.format); if ok && result < 0 ok, error = false, av_error("FFmpeg cannot finish the MP4", result); } @@ -186,22 +229,64 @@ encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat { return .NONE; } -encoder_drain :: (e: *Encoder) -> ok: bool, error: string { +// The packets an encoder (the video's, unless another is given) has ready, into the file. +encoder_drain :: (e: *Encoder, codec: *AVCodecContext = null, stream: *AVStream = null, packet: *AVPacket = null) -> ok: bool, error: string { AVERROR_EAGAIN :: -11; + if !codec codec, stream, packet = e.codec, e.stream, e.packet; while true { - result := avcodec_receive_packet(e.codec, e.packet); + result := avcodec_receive_packet(codec, packet); if result == AVERROR_EAGAIN || result == cast,no_check(s32) AVERROR_EOF return true, ""; - if result < 0 return false, av_error(tprint("% failed", e.name), result); - av_packet_rescale_ts(e.packet, e.codec.time_base, e.stream.time_base); - e.packet.stream_index = e.stream.index; - e.bytes += e.packet.size; - result = av_interleaved_write_frame(e.format, e.packet); // takes the packet's data + if result < 0 return false, av_error(tprint("% failed", ifx codec == e.audio then "the AAC encoder" else e.name), result); + av_packet_rescale_ts(packet, codec.time_base, stream.time_base); + packet.stream_index = stream.index; + lock(*e.mux); + e.bytes += packet.size; + result = av_interleaved_write_frame(e.format, packet); // takes the packet's data + unlock(*e.mux); if result < 0 return false, av_error("FFmpeg cannot write to the MP4", result); } return true, ""; } +encoder_open_audio :: (e: *Encoder) -> ok: bool, error: string { + codec := avcodec_find_encoder_by_name("aac"); + if !codec return false, "this FFmpeg has no AAC encoder"; + c := avcodec_alloc_context3(codec); + if !c return false, "out of memory"; + c.sample_fmt = .FLTP; + c.sample_rate = AUDIO_RATE; + av_channel_layout_default(*c.ch_layout, AUDIO_CHANNELS); + c.bit_rate = RECORD_AUDIO_BIT_RATE; + c.time_base = .{1, AUDIO_RATE}; + if e.format.oformat.flags & AVFMT_GLOBALHEADER c.flags |= AV_CODEC_FLAG_GLOBAL_HEADER; + result := avcodec_open2(c, codec, null); + if result < 0 { avcodec_free_context(*c); return false, av_error("the AAC encoder cannot open", result); } + e.audio = c; + e.audio_frame_size = ifx c.frame_size > 0 then c.frame_size else 1024; + + e.audio_stream = avformat_new_stream(e.format, null); + if !e.audio_stream return false, "FFmpeg cannot add a sound stream"; + avcodec_parameters_from_context(e.audio_stream.codecpar, c); + e.audio_stream.time_base = c.time_base; + + e.audio_packet = av_packet_alloc(); + e.audio_frame = av_frame_alloc(); + if !e.audio_packet || !e.audio_frame return false, "FFmpeg is out of memory"; + e.audio_frame.format = cast(s32) AVSampleFormat.FLTP; + e.audio_frame.nb_samples = e.audio_frame_size; + e.audio_frame.sample_rate = AUDIO_RATE; + av_channel_layout_default(*e.audio_frame.ch_layout, AUDIO_CHANNELS); + result = av_frame_get_buffer(e.audio_frame, 0); + if result < 0 return false, av_error("FFmpeg cannot make a sound frame", result); + return true, ""; +} + encoder_free :: (e: *Encoder) { + if e.audio_frame av_frame_free(*e.audio_frame); + if e.audio_packet av_packet_free(*e.audio_packet); + if e.audio avcodec_free_context(*e.audio); + e.audio_stream = null; + if e.mux_ready { destroy(*e.mux); e.mux_ready = false; } if e.sws sws_freeContext(e.sws); if e.frame av_frame_free(*e.frame); if e.packet av_packet_free(*e.packet); diff --git a/modules/Screen_Record/module.jai b/modules/Screen_Record/module.jai index 2f6e209..9ed55a3 100644 --- a/modules/Screen_Record/module.jai +++ b/modules/Screen_Record/module.jai @@ -13,6 +13,11 @@ // one a second while nothing does, so players can seek. Each carries the time it was taken, less the // time spent paused (record_pause), so a pause leaves no gap in the video. // +// Sound (system_audio, microphone) has a thread of its own (audio_source.jai's WASAPI): each packet is +// put on the same timeline as the frames, by when it was heard, less the same pauses, into one mix, +// and an AAC frame of it is encoded once every source has had its say up to there. A source that sends +// nothing (the computer playing nothing) is silence. +// // recorder: Recorder; // ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0}); // ... @@ -66,6 +71,8 @@ Record_Options :: struct { capture: Record_Capture; legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe testing: Record_Testing; + system_audio: bool; // what the computer plays (the default output device) + microphone: bool; // the default input device } Record_Stats :: struct { @@ -81,6 +88,9 @@ Record_Stats :: struct { dropped: s64; // the encoder was too far behind lost: s64; // times the duplication was lost and made again paused_seconds: float64; // left out of the video + audio: string; // the sound recorded: "Speakers (Realtek(R) Audio); Microphone (USB Audio)", or "" + audio_frames: s64; // AAC frames encoded + audio_errors: string; // sources that could not be opened, or were lost copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks bytes: s64; // encoded seconds: float64; // from start to stop @@ -99,7 +109,15 @@ Recorder :: struct { start_ticks: s64; capture_thread, encoder_thread: Thread; - capture_started, encoder_started: bool; + capture_started, encoder_started, audio_started: bool; + audio_thread: Thread; + audio_opened: Semaphore; // the audio thread has opened its sources + audio_go: Semaphore; // the encoder is open: the audio thread may write + audio_sources: [2] Audio_Source; + audio_open: bool; // any source opened: the MP4 has sound + zero_ticks: s64; // the recording's time 0, for frames and sound alike + paused_ticks: s64; // time paused so far, for the capture and audio threads; set by record_pause + pause_began: s64; running: bool; stopping: bool; // set by record_stop; the capture thread ends, then the encoder paused: bool; // set by record_pause; the capture thread takes nothing while it is @@ -175,6 +193,7 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri QueryPerformanceFrequency(*frequency); r.frequency = frequency.QuadPart; r.start_ticks = ticks_now(); + r.zero_ticks = r.start_ticks; // The source is opened on the capture thread, which uses it: WinRT's objects belong to the // apartment they are made in. @@ -188,7 +207,18 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri return false, error, fall_back; } - ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder); + // The sound's sources are opened on their own thread (COM, for WASAPI), before the encoder, which + // has a sound stream only when one opened. + if r.options.system_audio || r.options.microphone { + init(*r.audio_opened); + init(*r.audio_go); + thread_init(*r.audio_thread, record_audio_proc, name = "record audio"); + r.audio_started = true; + thread_start(*r.audio_thread, r); + wait_for(*r.audio_opened); + } + + ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder, audio = r.audio_open); if tried log("[record] Encoders that did not open: %", tried); if !ok { error = copy_string(error,, temp); @@ -196,6 +226,7 @@ record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: stri return false, error, false; } r.stats.encoder = copy_string(r.encoder.name); + if r.audio_started signal(*r.audio_go); thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder"); r.encoder_started = true; @@ -211,6 +242,7 @@ record_stop :: (r: *Recorder) -> ok: bool, error: string { atomic_swap(*r.stopping, true); thread_is_done(*r.capture_thread, -1); thread_is_done(*r.encoder_thread, -1); + if r.audio_started thread_is_done(*r.audio_thread, -1); r.running = false; closed, close_error := encoder_close(*r.encoder); @@ -224,9 +256,23 @@ record_stop :: (r: *Recorder) -> ok: bool, error: string { return true, ""; } -// Pauses or resumes. While paused nothing is taken, and the time is left out of the video. +// Pauses or resumes. While paused nothing is taken, and the time is left out of the video, sound and +// all: the time paused is kept here, once, for the capture and audio threads both. record_pause :: (r: *Recorder, pause: bool) { - if r.running atomic_swap(*r.paused, pause); + if !r.running || atomic_read(*r.paused) == pause return; + now := ticks_now(); + if pause { + atomic_write(*r.pause_began, now); + atomic_swap(*r.paused, true); + } else { + atomic_add(*r.paused_ticks, now - atomic_read(*r.pause_began)); // before paused clears, so no thread sees it unpaused with the old total + atomic_swap(*r.paused, false); + } +} + +// Ticks on the recording's timeline: time 0 at its start, less the time paused. +record_timeline :: (r: *Recorder, ticks: s64) -> s64 { + return ticks - r.zero_ticks - atomic_read(*r.paused_ticks); } record_paused :: (r: *Recorder) -> bool { @@ -265,6 +311,14 @@ record_free :: (r: *Recorder) { atomic_swap(*r.stopping, true); if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; } if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; } + if r.audio_started { + signal(*r.audio_go); // in case it waits for an encoder that did not open + thread_is_done(*r.audio_thread, -1); + thread_deinit(*r.audio_thread); + destroy(*r.audio_opened); + destroy(*r.audio_go); + r.audio_started = false; + } if r.encoder.format encoder_close(*r.encoder); for * r.frames array_free(it.pixels); array_free(r.last_frame); @@ -331,7 +385,7 @@ record_capture_proc :: (thread: *Thread) -> s64 { have_image, changed := false, false; staged: [RECORD_STAGING] Staged; emits: s64; - paused_ticks, pause_start: s64; // time paused so far, left out of the frames' times; when this pause began + was_paused := false; last_lost := ticks_now(); // for Record_Testing.LOSE_DUPLICATION while !atomic_read(*r.stopping) { @@ -381,15 +435,15 @@ record_capture_proc :: (thread: *Thread) -> s64 { // screen as it is now is sent at once. now = ticks_now(); if atomic_read(*r.paused) { - if !pause_start pause_start = now; - lock(*r.mutex); r.stats.paused_seconds = cast(float64) (paused_ticks + now - pause_start) / r.frequency; unlock(*r.mutex); + was_paused = true; + lock(*r.mutex); r.stats.paused_seconds = cast(float64) (atomic_read(*r.paused_ticks) + now - atomic_read(*r.pause_began)) / r.frequency; unlock(*r.mutex); continue; } - if pause_start { - paused_ticks += now - pause_start; - pause_start = 0; + if was_paused { + was_paused = false; changed = true; turn = now; + lock(*r.mutex); r.stats.paused_seconds = cast(float64) atomic_read(*r.paused_ticks) / r.frequency; unlock(*r.mutex); } if !have_image continue; repeat := !changed && now - last_emit >= r.frequency; @@ -400,7 +454,7 @@ record_capture_proc :: (thread: *Thread) -> s64 { slot := emits % RECORD_STAGING; if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]); ID3D11DeviceContext_CopyResource(gpu.device_context, gpu.staging[slot], gpu.latest); - staged[slot] = .{pending = true, ticks = now - paused_ticks}; + staged[slot] = .{pending = true, ticks = record_timeline(r, now)}; if !r.using_wgc staged[slot].pointer, staged[slot].pointer_visible = r.dxgi.pointer_position, r.dxgi.pointer_visible; emits += 1; last_emit = now; @@ -465,7 +519,6 @@ record_read_back :: (r: *Recorder, gpu: *Capture_Gpu, slot: s64, staged: Staged) record_encoder_proc :: (thread: *Thread) -> s64 { r := cast(*Recorder) thread.data; - first: s64 = -1; while true { wait_for(*r.ready, 100); lock(*r.mutex); @@ -482,8 +535,7 @@ record_encoder_proc :: (thread: *Thread) -> s64 { } frame := *r.frames[index]; - if first < 0 first = frame.ticks; - pts := (frame.ticks - first) * 90000 / r.frequency; + pts := max(frame.ticks, 0) * 90000 / r.frequency; // frame.ticks is on the timeline already t := ticks_now(); ok, error := encoder_write(*r.encoder, frame.pixels.data, pts); @@ -510,6 +562,137 @@ record_encoder_proc :: (thread: *Thread) -> s64 { return 0; } +// The audio thread: opens the sources asked for, waits for the encoder, then every 10 ms takes their +// packets into the mix and encodes what every source has covered. +record_audio_proc :: (thread: *Thread) -> s64 { + r := cast(*Recorder) thread.data; + audio_thread_start(); + defer audio_thread_end(); + + kinds: [2] Audio_Source_Kind = .[.SYSTEM, .MICROPHONE]; + wanted: [2] bool = .[r.options.system_audio, r.options.microphone]; + names, errors: [..] string; + names.allocator, errors.allocator = temp, temp; + for 0..1 { + if !wanted[it] continue; + s := *r.audio_sources[it]; + ok, error := audio_open(s, kinds[it]); + what := ifx kinds[it] == .SYSTEM then "system sound" else "microphone"; + if ok { r.audio_open = true; array_add(*names, s.name); } + else { audio_close(s); array_add(*errors, tprint("%: %", what, error)); log_error("[record] No %: %", what, error); } + } + r.stats.audio = join(..names, "; "); + r.stats.audio_errors = join(..errors, "; "); + signal(*r.audio_opened); + wait_for(*r.audio_go); + if !r.audio_open || !r.encoder.audio || atomic_read(*r.stopping) { + for * r.audio_sources audio_free(it); + return 0; + } + + mix: Audio_Mix; + mix.frame_size = r.encoder.audio_frame_size; + defer array_free(mix.samples); + while true { + stopping := atomic_read(*r.stopping); + for * source, index: r.audio_sources { + if !wanted[index] continue; + if !source.open { // lost: opened again a second on, from the default device then + if source.lost_at && ticks_now() - source.lost_at >= r.frequency && !stopping { + if audio_open(source, kinds[index]) log("[record] % is back: %.", ifx kinds[index] == .SYSTEM then "System sound" else "The microphone", source.name); + else { audio_close(source); source.lost_at = ticks_now(); } + } + continue; + } + packets, lost := audio_take(source, r.frequency); + for packets if !atomic_read(*r.paused) audio_mix_add(*mix, r, index, it); + if lost { + log_error("[record] % was lost (unplugged, or the default device changed); trying again.", source.name); + audio_close(source); + source.lost_at = ticks_now(); + } + } + ok, error := audio_mix_encode(*mix, r, final = stopping); + if !ok { record_fail(r, error); break; } + if stopping break; + reset_temporary_storage(); + sleep_milliseconds(10); + } + for * r.audio_sources audio_free(it); + return 0; +} + +// The sound so far, from its base: one AAC frame's worth is encoded at a time. +Audio_Mix :: struct { + samples: [..] float; // interleaved stereo, from base on + base: s64; // the frame (in samples, on the timeline) samples[0] is + next: [2] s64; // each source's next frame, so packets that follow on are not moved by jitter + started: [2] bool; + frame_size: s32; +} + +AUDIO_SLACK :: AUDIO_RATE / 5; // 200 ms: how late a source's packets may come and still be in the mix +AUDIO_FOLLOW_ON :: AUDIO_RATE / 33; // 30 ms: a packet that close to where its source left off follows on + +// A packet into the mix at its place on the timeline, added to what the other source put there. +audio_mix_add :: (mix: *Audio_Mix, r: *Recorder, source: s64, packet: Audio_Packet) { + at := record_timeline(r, packet.ticks) * AUDIO_RATE / r.frequency; + if mix.started[source] && abs(at - mix.next[source]) <= AUDIO_FOLLOW_ON at = mix.next[source]; + mix.started[source] = true; + mix.next[source] = at + packet.frames; + end := (at + packet.frames - mix.base) * AUDIO_CHANNELS; + if end > mix.samples.count { + old := mix.samples.count; + array_resize(*mix.samples, end, initialize = false); + for old..end - 1 mix.samples[it] = 0; + } + if packet.silent || !packet.samples.count return; + for frame: max(at, mix.base)..at + packet.frames - 1 { + into := (frame - mix.base) * AUDIO_CHANNELS; + from := (frame - at) * AUDIO_CHANNELS; + mix.samples[into] += packet.samples[from]; + mix.samples[into + 1] += packet.samples[from + 1]; + } +} + +// Encodes the frames every open source has covered: up to where each has sent, or, for one that has +// sent nothing lately (silence), to AUDIO_SLACK ago. final: everything sent. +audio_mix_encode :: (mix: *Audio_Mix, r: *Recorder, final: bool) -> ok: bool, error: string { + // Paused, the timeline stands still where the pause began. + now := ifx atomic_read(*r.paused) then atomic_read(*r.pause_began) else ticks_now(); + now_frame := record_timeline(r, now) * AUDIO_RATE / r.frequency; + ready := now_frame - AUDIO_SLACK; + for source, index: r.audio_sources { + if !source.open continue; + covered := max(mix.next[index], now_frame - AUDIO_SLACK); + ready = min(ready, covered); + } + if final { ready = mix.base; for mix.next ready = max(ready, it); } + + size := mix.frame_size; + frame := NewArray(size * AUDIO_CHANNELS, float, initialized = false,, temp); + done: s64; + while mix.base + size <= ready { + for i: 0..frame.count - 1 { + at := done * AUDIO_CHANNELS + i; + value := ifx at < mix.samples.count then mix.samples[at] else 0.0; + frame[i] = clamp(value, -1.0, 1.0); + } + ok, error := encoder_write_audio(*r.encoder, frame.data, mix.base); + if !ok return false, error; + mix.base += size; + done += size; + lock(*r.mutex); r.stats.audio_frames += 1; unlock(*r.mutex); + } + // What was encoded leaves the front. + gone := min(done * AUDIO_CHANNELS, mix.samples.count); + if gone { + memcpy(mix.samples.data, mix.samples.data + gone, (mix.samples.count - gone) * size_of(float)); + mix.samples.count -= gone; + } + return true, ""; +} + // Ends the recording from one of its threads; the first reason is kept. record_fail :: (r: *Recorder, error: string) { log_error("[record] %", error); @@ -530,11 +713,13 @@ rect_inside :: (inner: RECT, outer: RECT) -> bool { #load "wgc_source.jai"; #load "encoder.jai"; #load "remux.jai"; +#load "audio_source.jai"; #load "cursor.jai"; #import "Basic"; #import "String"; #import "Atomics"; +#import "Math"; #import "Thread"; #import "Windows"; #import "Windows_Utf8"; diff --git a/modules/Screen_Record/wgc_source.jai b/modules/Screen_Record/wgc_source.jai index da51695..6ae0497 100644 --- a/modules/Screen_Record/wgc_source.jai +++ b/modules/Screen_Record/wgc_source.jai @@ -308,24 +308,6 @@ IID_DIRECT3D_DEVICE := #run guid("A37624AB-8D5F-4650-9D3E-9EAE3D9BC670"); IID_DXGI_INTERFACE_ACCESS := #run guid("A9B3D012-3DF2-4EE3-B8D1-8695F457D3C1"); IID_CLOSABLE := #run guid("30D5A829-7FA4-4026-83BB-D75BAE4EA99E"); -guid :: (text: string) -> GUID { - hex :: (s: string) -> u64 { - value: u64; - for 0..s.count - 1 { - c := s[it]; - digit: u64 = ifx c >= #char "a" then cast(u64) (c - #char "a" + 10) else ifx c >= #char "A" then cast(u64) (c - #char "A" + 10) else cast(u64) (c - #char "0"); - value = value * 16 + digit; - } - return value; - } - g: GUID; - g.Data1 = cast(u32) hex(slice(text, 0, 8)); - g.Data2 = cast(u16) hex(slice(text, 9, 4)); - g.Data3 = cast(u16) hex(slice(text, 14, 4)); - for 0..1 g.Data4[it] = cast(u8) hex(slice(text, 19 + it * 2, 2)); - for 0..5 g.Data4[it + 2] = cast(u8) hex(slice(text, 24 + it * 2, 2)); - return g; -} // IUnknown, then IInspectable's three: every WinRT interface starts so. Inspectable_Methods :: struct { diff --git a/src/app/screen_capture.jai b/src/app/screen_capture.jai index 96ac49c..86740cf 100644 --- a/src/app/screen_capture.jai +++ b/src/app/screen_capture.jai @@ -23,6 +23,7 @@ Screen_Capture_Job :: struct { source: string; // how the screen was taken: Record_Stats.source dropped: s64; // frames the encoder fell too far behind for paused_seconds: float64; // left out of it + audio: string; // the sound's sources, "Speakers (...); Microphone (...)", or "" thread: Thread; busy: bool; // cleared by the thread as it ends @@ -142,6 +143,7 @@ screen_capture_job_free :: (job: *Screen_Capture_Job) { free(job.recording); free(job.encoder); free(job.source); + free(job.audio); free(job); } diff --git a/src/app/screen_record.jai b/src/app/screen_record.jai index a3ced58..4241afa 100644 --- a/src/app/screen_record.jai +++ b/src/app/screen_record.jai @@ -9,8 +9,8 @@ // // While it records, a red border is drawn just outside the region and a toolbar beside it gives the // time, Pause, Stop and Discard (app/screen_record_win32.jai); both are kept out of the recording. -// Time paused is left out of the video. The frame rate, the cursor and how the screen is taken are -// [video] in the settings file. +// Time paused is left out of the video. With sound: what the computer plays and the microphone, as +// [video] in the settings file says, as are the frame rate, the cursor and how the screen is taken. // // A recording that stops by itself (the screen went away, the encoder failed) keeps what it has, and // so does quitting while recording. One left in the temporary folder by a crash is offered at the next @@ -204,6 +204,8 @@ screen_record_start :: (immediate := false) { options.region = .{shot.left, shot.top, shot.left + shot.width, shot.top + shot.height}; options.fps = cast(s32) app_user_settings.video_fps; options.cursor = app_user_settings.video_cursor; + options.system_audio = app_user_settings.video_system_audio; + options.microphone = app_user_settings.video_microphone; if app_user_settings.video_capture == { case "dxgi"; options.capture = .DXGI; case "wgc"; options.capture = .WGC; @@ -224,7 +226,13 @@ screen_record_start :: (immediate := false) { if stats.fell_back_from log("[record] Desktop Duplication could not run: %", stats.fell_back_from); log("[record] Recording % x % at %, % (%) of \"%\" (%) with % and %, to %.", stats.width, stats.height, shot.left, shot.top, ifx shot.whole_screen then "the whole screen" else "a region", shot.window_title, shot.app_name, stats.source, stats.encoder, path); - toast_show(.Success, "Recording % x %: % stops", stats.width, stats.height, screen_record_hotkey_text()); + if stats.audio log("[record] Sound: %.", stats.audio); + if stats.audio_errors { + log_error("[record] Sound not recorded: %.", stats.audio_errors); + toast_show(.Error, "Recording without some sound: %", stats.audio_errors); + } else { + toast_show(.Success, "Recording % x %: % stops", stats.width, stats.height, screen_record_hotkey_text()); + } record_ui_open(options.region, shot.whole_screen, stats.width, stats.height); } @@ -278,6 +286,7 @@ screen_record_stop :: (shutdown := false) { job.source = copy_string(stats.source,, context.default_allocator); job.dropped = stats.dropped; job.paused_seconds = stats.paused_seconds; + job.audio = copy_string(stats.audio,, context.default_allocator); screen_record.shot = .{}; screen_record.date, screen_record.path = "", ""; screen_capture_job_start(job); @@ -338,6 +347,8 @@ screen_record_write :: (job: *Screen_Capture_Job) -> path: string, error: string if job.encoder import_add(doc, info, "Encoder", import_text(job.encoder)); if job.source import_add(doc, info, "Capture", import_text(job.source)); import_add_number(doc, info, "DroppedFrames", xx job.dropped); + capture_object_add(doc, *info.root, "Audio", capture_boolean(video.has_audio)); + if job.audio import_add(doc, info, "AudioSources", import_text(job.audio)); if job.paused_seconds > 0 import_add_number(doc, info, "Paused", import_milliseconds(job.paused_seconds)); name := import_out_name(job.date, hash); diff --git a/src/test/import_tests.jai b/src/test/import_tests.jai index a482cef..a520dd2 100644 --- a/src/test/import_tests.jai +++ b/src/test/import_tests.jai @@ -783,6 +783,37 @@ screen_record_tests :: () { file_delete(leftover_path); } + // With sound: an AAC stream beside the video, kept through the remux, and capture.json says so. + encoder = .{}; + opened, open_error = encoder_open(*encoder, mp4, WIDTH, HEIGHT, 30, "", audio = true); + expect(opened && encoder.audio != null, "an encoder with sound opened: %", open_error); + if opened { + tone := NewArray(encoder.audio_frame_size * 2, float,, temp); + for frame: 0..29 encoder_write(*encoder, pixels.data, frame * 3000); + for block: 0..46 { // about a second of 440 Hz + for i: 0..encoder.audio_frame_size - 1 { + v := sin(cast(float) (block * encoder.audio_frame_size + i) * 2 * PI * 440 / 48000) * 0.5; + tone[i * 2], tone[i * 2 + 1] = v, v; + } + encoder_write_audio(*encoder, tone.data, block * encoder.audio_frame_size); + } + encoder_close(*encoder); + sounding: Screen_Capture_Job; + sounding.video = true; + sounding.recording = mp4; + sounding.date = "2026-10-10 08:30:00"; + sounding.audio = "Speakers (test)"; + sounding.shot.width, sounding.shot.height = WIDTH, HEIGHT; + sounding_path, sounding_error := screen_record_write(*sounding); + expect(!sounding_error, "a recording with sound is written: %", sounding_error); + if !sounding_error { + m4, parsed4 := mcap_parse(cast([] u8) read_entire_file(sounding_path,, temp)); + expect(parsed4 && find_index_from_left(cast(string) m4.original, "mp4a") >= 0, "its MP4 has the AAC stream (mp4a)"); + expect(parsed4 && contains(m4.text, "Speakers (test)"), "capture.json names the sound's source"); + file_delete(sounding_path); + } + } + // One a crash cut short: a frame every 100 ms for 5 s makes fragments at 0, 2 and 4 s (a // keyframe each 2 s, however few the frames); cut at the last, as a crash would, it goes into the // library as a regular MP4 (header first, no fragments) of the 4 s before the cut. diff --git a/src/ui/imgui_theme_registry.jai b/src/ui/imgui_theme_registry.jai index 7e52b4d..7fb16c9 100644 --- a/src/ui/imgui_theme_registry.jai +++ b/src/ui/imgui_theme_registry.jai @@ -311,6 +311,8 @@ App_User_Settings :: struct { video_fps: s64 = VIDEO_FPS_DEFAULT; video_cursor := true; video_capture: string; + video_system_audio := true; // what the computer plays + video_microphone: bool; // off unless asked for // The [hotkeys] lines. Only meaningful when the section was seen; see hotkeys_install. hotkey_bindings: [..] Hotkey_Binding; @@ -380,12 +382,12 @@ app_load_user_settings_file :: () -> write_back: bool { if hotkeys_missing.count || global_missing.count return true; #if OS == .LINUX seen.global_hotkeys = true; // not written there: Linux has no global hotkeys yet - return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.video_fps && seen.video_cursor && seen.video_capture); + return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.video_fps && seen.video_cursor && seen.video_capture && seen.video_system_audio && seen.video_microphone); } Settings_Seen :: struct { theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys, ocr_model: bool; - video_fps, video_cursor, video_capture: bool; + video_fps, video_cursor, video_capture, video_system_audio, video_microphone: bool; } // Of the file's bytes as last loaded or written, so a watcher event for our own save, or for an @@ -520,6 +522,10 @@ settings_parse :: (path: string, file_data: string, out: *App_User_Settings) -> seen.video_fps = settings_read_int(*out.video_fps, path, line_number, key, value, VIDEO_FPS_MIN, VIDEO_FPS_MAX); } else if section == "[video]" && key == "cursor" { seen.video_cursor = settings_read_bool(*out.video_cursor, path, line_number, key, value); + } else if section == "[video]" && key == "system_audio" { + seen.video_system_audio = settings_read_bool(*out.video_system_audio, path, line_number, key, value); + } else if section == "[video]" && key == "microphone" { + seen.video_microphone = settings_read_bool(*out.video_microphone, path, line_number, key, value); } else if section == "[video]" && key == "capture" { capture := to_lower_copy(settings_unquote(value),, temp); if capture == "auto" || capture == "dxgi" || capture == "wgc" { @@ -664,6 +670,8 @@ app_save_user_settings_file :: () -> bool { if seen.event_log app_user_settings.event_log_capacity = on_disk.event_log_capacity; if seen.video_fps app_user_settings.video_fps = on_disk.video_fps; if seen.video_cursor app_user_settings.video_cursor = on_disk.video_cursor; + if seen.video_system_audio app_user_settings.video_system_audio = on_disk.video_system_audio; + if seen.video_microphone app_user_settings.video_microphone = on_disk.video_microphone; if seen.video_capture { free(app_user_settings.video_capture,, context.default_allocator); app_user_settings.video_capture = copy_string(on_disk.video_capture,, context.default_allocator); @@ -691,10 +699,13 @@ app_save_user_settings_file :: () -> bool { append(*builder, "[video]\n"); append(*builder, "# Screen recording (Ctrl+PrintScreen; Windows). fps: frames a second at most, written only when the\n"); append(*builder, "# screen changes. cursor: true or false. capture: auto (Desktop Duplication, else\n"); - append(*builder, "# Windows.Graphics.Capture), dxgi or wgc.\n"); + append(*builder, "# Windows.Graphics.Capture), dxgi or wgc. system_audio: what the computer plays; microphone: the\n"); + append(*builder, "# default input device. Each true or false.\n"); print_to_builder(*builder, "fps: %\n", app_user_settings.video_fps); print_to_builder(*builder, "cursor: %\n", ifx app_user_settings.video_cursor then "true" else "false"); - print_to_builder(*builder, "capture: %\n\n", ifx app_user_settings.video_capture.count then app_user_settings.video_capture else "auto"); + print_to_builder(*builder, "capture: %\n", ifx app_user_settings.video_capture.count then app_user_settings.video_capture else "auto"); + print_to_builder(*builder, "system_audio: %\n", ifx app_user_settings.video_system_audio then "true" else "false"); + print_to_builder(*builder, "microphone: %\n\n", ifx app_user_settings.video_microphone then "true" else "false"); append(*builder, "[logging]\n"); append(*builder, "# Entries the Event Log keeps in memory. Past it the oldest are dropped; the log file keeps every line.\n"); print_to_builder(*builder, "event_log_capacity: %\n", app_user_settings.event_log_capacity); diff --git a/tools/record_probe.jai b/tools/record_probe.jai index 1368694..844333d 100644 --- a/tools/record_probe.jai +++ b/tools/record_probe.jai @@ -21,6 +21,9 @@ // -animate moves a square across the region the whole time, so the screen changes every // frame: for measuring throughput. // -remux IN OUT a recording's fragmented MP4 made a regular one, as the app does, and nothing else. +// -audio WHICH records sound too: system (what the computer plays), mic, or both. +// -sync-test every 2 s a white square flashes on the region and a 1 kHz beep plays; then the +// flashes and beeps are found in the file and their offsets given, and how they drift. // -check FILE reads FILE back with FFmpeg and says what it holds, and nothing else: for a // recording cut short by a crash. // -snapshot DIR the last frame captured (captured.png) and a frame decoded back from the file @@ -41,6 +44,7 @@ main :: () { list_only, cuda, exclude_test, dpi_unaware, animate, border_test, cursor_test := false, false, false, false, false, false, false; snapshot_folder: string; pause_seconds: float64; + sync_test := false; i := 1; number :: (index: s64) -> s32 #expand { return cast(s32) string_to_int(`args[index]); } while i < args.count { @@ -69,6 +73,14 @@ main :: () { if a == "-simulate-unsupported" { options.testing |= .DXGI_UNSUPPORTED; i += 1; continue; } if a == "-poll" { options.testing |= .WGC_POLL; i += 1; continue; } if a == "-lose" { options.testing |= .LOSE_DUPLICATION; i += 1; continue; } + if a == "-sync-test" { sync_test = true; i += 1; continue; } + if a == "-audio" && more >= 1 { + which := args[i + 1]; + options.system_audio = which == "system" || which == "both"; + options.microphone = which == "mic" || which == "both"; + if !options.system_audio && !options.microphone { print("-audio takes system, mic or both\n"); exit(2); } + i += 2; continue; + } if a == "-pause" && more >= 1 { pause_seconds = string_to_float64(args[i + 1]); i += 2; continue; } if a == "-cursor-test" { cursor_test = true; i += 1; continue; } if a == "-capture" && more >= 1 { @@ -149,6 +161,15 @@ main :: () { cursor_at = .{region.left + (region.right - region.left) / 2, region.top + (region.bottom - region.top) / 2}; SetCursorPos(cursor_at.x, cursor_at.y); } + flash: HWND; + beep: [] u8; + next_flash: float64 = 1; + if sync_test { + beep = beep_wav(); + side := min(cast(s32) 200, (region.right - region.left) / 3, (region.bottom - region.top) / 3); + flash = square_window("record-probe flash", 0x00FFFFFF, region.left + (region.right - region.left - side) / 2, region.top + (region.bottom - region.top - side) / 2, side); + ShowWindow(flash, SW_HIDE); + } mover: HWND; if animate mover = square_window("record-probe animate", 0x0000A5FF, region.left, region.top, 120); @@ -173,6 +194,16 @@ main :: () { print(" % s: % captured, % sent (% repeats), % written, % dropped, % lost, % KB\n", FF(to_float64_seconds(last_print - start), 0), s.captured, s.emitted, s.repeats, s.written, s.dropped, s.lost, s.bytes / 1024); } + if flash { // a flash and a beep together every 2 s, the flash for 200 ms + elapsed := to_float64_seconds(current_time_monotonic() - start); + if elapsed >= next_flash { + ShowWindow(flash, SW_SHOWNOACTIVATE); + PlaySoundW(beep.data, null, SND_MEMORY | SND_ASYNC); + next_flash += 2; + } else if elapsed >= next_flash - 2 + 0.2 { + ShowWindow(flash, SW_HIDE); + } + } if pause_seconds > 0 { elapsed := to_float64_seconds(current_time_monotonic() - start); should := elapsed >= 2 && elapsed < 2 + pause_seconds; @@ -189,6 +220,7 @@ main :: () { last, width, height := record_last_frame(*recorder); if mover DestroyWindow(mover); + if flash DestroyWindow(flash); if cursor_test SetCursorPos(saved_cursor.x, saved_cursor.y); stopped, stop_error := record_stop(*recorder); s := recorder.stats; @@ -208,6 +240,29 @@ main :: () { print(" each: copy % (per image), read back % , encode % (per frame)\n", ms_each(s.copy_ticks, s.captured), ms_each(s.readback_ticks, s.emitted), ms_each(s.encode_ticks, s.written)); print(" file: % KB, % MB a minute\n", file_size / 1024, FF(cast(float64) file_size / (1024 * 1024) / s.seconds * 60, 1)); + if options.system_audio || options.microphone { + print(" sound: %; % AAC frames%\n", ifx s.audio then s.audio else "none opened", s.audio_frames, ifx s.audio_errors then tprint(" (%)", s.audio_errors) else ""); + sound := sound_report(options.path); + if sound.found print(" in file: % s of sound, % dB RMS, % onsets\n", FF(sound.seconds, 2), FF(sound.rms_db, 1), sound.onsets.count); + else print(" in file: no sound stream %\n", sound.error); + if sync_test && sound.found { + flashes := flash_times(options.path); + print(" sync: % flashes, % beeps\n", flashes.count, sound.onsets.count); + offsets: [..] float64; + offsets.allocator = temp; + for flash_time: flashes { + best: float64 = 1000000000.0; + for sound.onsets if abs(it - flash_time) < abs(best) best = it - flash_time; + if abs(best) < 0.5 array_add(*offsets, best); + } + if offsets.count { + text: [..] string; text.allocator = temp; + for offsets array_add(*text, tprint("%", FF(it * 1000, 0))); + print(" offsets: beep after flash, ms: %\n", join(..text, " ",, temp)); + print(" drift: % ms from the first to the last\n", FF((offsets[offsets.count - 1] - offsets[0]) * 1000, 0)); + } + } + } if exclude_test { excluded_rect := window_rect_in(excluded, region); @@ -405,6 +460,134 @@ FF :: (value: float64, decimals: s64) -> FormatFloat #expand { SetProcessDpiAwarenessContext :: (value: HANDLE) -> BOOL #foreign user32_dll; SetWindowDisplayAffinity :: (window: HWND, affinity: u32) -> BOOL #foreign user32_dll; SetCursorPos :: (x: s32, y: s32) -> BOOL #foreign user32_dll; + +// ---------------------------------------------------------------- sound and sync + +// A beep of 1 kHz for 150 ms, as a WAV in memory, for PlaySound. +beep_wav :: () -> [] u8 { + RATE :: 48000; + COUNT :: RATE * 150 / 1000; + data := NewArray(44 + COUNT * 2, u8,, context.default_allocator); + put32 :: (at: s64, v: u32) #expand { for 0..3 `data[at + it] = cast,trunc(u8) (v >> (8 * it)); } + put16 :: (at: s64, v: u16) #expand { `data[at] = cast,trunc(u8) v; `data[at + 1] = cast,trunc(u8) (v >> 8); } + memcpy(data.data, "RIFF".data, 4); put32(4, cast(u32) (36 + COUNT * 2)); memcpy(data.data + 8, "WAVEfmt ".data, 8); + put32(16, 16); put16(20, 1); put16(22, 1); put32(24, RATE); put32(28, RATE * 2); put16(32, 2); put16(34, 16); + memcpy(data.data + 36, "data".data, 4); put32(40, COUNT * 2); + for i: 0..COUNT - 1 { + v := cast(s16) (sin(cast(float) i * 2 * PI * 1000 / RATE) * 20000); + put16(44 + i * 2, cast,no_check(u16) v); + } + return data; +} + +// What the file's sound holds: its length, its level, and when it goes from quiet to loud (onsets). +Sound_Report :: struct { found: bool; seconds: float64; rms_db: float64; onsets: [..] float64; error: string; } + +sound_report :: (path: string) -> Sound_Report { + report: Sound_Report; + report.onsets.allocator = temp; + format: *AVFormatContext; + if avformat_open_input(*format, temp_c_string(path), null, null) < 0 { report.error = "cannot open"; return report; } + defer avformat_close_input(*format); + avformat_find_stream_info(format, null); + index := -1; + for 0..format.nb_streams - 1 if format.streams[it].codecpar.codec_type == .AUDIO { index = it; break; } + if index < 0 return report; + report.found = true; + stream := format.streams[index]; + codec := avcodec_find_decoder(stream.codecpar.codec_id); + c := avcodec_alloc_context3(codec); + defer avcodec_free_context(*c); + avcodec_parameters_to_context(c, stream.codecpar); + if avcodec_open2(c, codec, null) < 0 { report.error = "no decoder"; return report; } + packet := av_packet_alloc(); defer av_packet_free(*packet); + frame := av_frame_alloc(); defer av_frame_free(*frame); + + WINDOW :: 480; // 10 ms at 48 kHz + sum_all: float64; count_all: s64; + window_sum: float64; window_count: s64; window_start: float64 = -1; + quiet_for: s64 = 100; // windows in a row below the threshold + first_time: float64 = -1; last_time: float64; + handle :: () #expand { + while avcodec_receive_frame(`c, `frame) >= 0 { + t := cast(float64) `frame.pts * `stream.time_base.num / `stream.time_base.den; + if `first_time < 0 `first_time = t; + left := cast(*float) `frame.data[0]; + for i: 0..`frame.nb_samples - 1 { + v := cast(float64) left[i]; + `sum_all += v * v; `count_all += 1; + if `window_start < 0 `window_start = t + cast(float64) i / 48000; + `window_sum += v * v; `window_count += 1; + if `window_count == WINDOW { + rms := sqrt(`window_sum / WINDOW); + if rms > 0.05 { + if `quiet_for >= 20 array_add(*`report.onsets, `window_start); // after 200 ms of quiet + `quiet_for = 0; + } else { + `quiet_for += 1; + } + `window_sum, `window_count, `window_start = 0, 0, -1; + } + } + `last_time = t + cast(float64) `frame.nb_samples / 48000; + } + } + while av_read_frame(format, packet) >= 0 { + if packet.stream_index == index { avcodec_send_packet(c, packet); handle(); } + av_packet_unref(packet); + } + avcodec_send_packet(c, null); + handle(); + report.seconds = last_time - max(first_time, 0); + report.rms_db = ifx count_all && sum_all > 0 then 10 * log(sum_all / count_all) / log(10.0) else -200; + return report; +} + +// When the video's middle turns white: the sync test's flashes, as times in seconds. +flash_times :: (path: string) -> [..] float64 { + times: [..] float64; + times.allocator = temp; + format: *AVFormatContext; + if avformat_open_input(*format, temp_c_string(path), null, null) < 0 return times; + defer avformat_close_input(*format); + avformat_find_stream_info(format, null); + index := -1; + for 0..format.nb_streams - 1 if format.streams[it].codecpar.codec_type == .VIDEO { index = it; break; } + if index < 0 return times; + stream := format.streams[index]; + codec := avcodec_find_decoder(stream.codecpar.codec_id); + c := avcodec_alloc_context3(codec); + defer avcodec_free_context(*c); + avcodec_parameters_to_context(c, stream.codecpar); + if avcodec_open2(c, codec, null) < 0 return times; + packet := av_packet_alloc(); defer av_packet_free(*packet); + frame := av_frame_alloc(); defer av_frame_free(*frame); + was_white := false; + handle :: () #expand { + while avcodec_receive_frame(`c, `frame) >= 0 { + // The luma of the middle 32 x 32 (the first plane in every YUV format the encoders give). + w, h := `frame.width, `frame.height; + sum: s64; + for y: h / 2 - 16..h / 2 + 15 for x: w / 2 - 16..w / 2 + 15 sum += `frame.data[0][y * `frame.linesize[0] + x]; + white := sum / 1024 > 200; + if white && !`was_white array_add(*`times, cast(float64) `frame.best_effort_timestamp * `stream.time_base.num / `stream.time_base.den); + `was_white = white; + } + } + while av_read_frame(format, packet) >= 0 { + if packet.stream_index == index { avcodec_send_packet(c, packet); handle(); } + av_packet_unref(packet); + } + avcodec_send_packet(c, null); + handle(); + return times; +} + +PlaySoundW :: (sound: *void, module: *void, flags: u32) -> s32 #foreign winmm_dll; +winmm_dll :: #library,system "winmm"; +SND_ASYNC :: 0x1; +SND_MEMORY :: 0x4; + user32_dll :: #library,system "user32"; #import "Basic"; @@ -417,3 +600,4 @@ user32_dll :: #library,system "user32"; #import "Screen_Record"; #import "Image_Decode"; #import "stb_image_write"; +#import "ffmpeg"; diff --git a/video-capture-plan.html b/video-capture-plan.html index 6b50c90..1fa0992 100644 --- a/video-capture-plan.html +++ b/video-capture-plan.html @@ -68,7 +68,7 @@

Screen video capture in mcap

-

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · phases 1 to 4 built, mcap 0.1.6

+

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · all five phases built, mcap 0.1.8

@@ -83,6 +83,7 @@
  • Done Phase 1 (2026-10-10, mcap 0.1.3): modules/Screen_Record and record-probe. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see phase 1's result).
  • Done Phase 2 (2026-10-10, mcap 0.1.4): the Windows.Graphics.Capture fallback. Automatic mode falls back to it on a simulated DXGI_ERROR_UNSUPPORTED. It records 4K at 60 fps with nothing dropped, draws the cursor exactly as Desktop Duplication's code does, and the yellow border can be turned off on Windows 11.
  • Done Phase 3 (2026-10-10, mcap 0.1.5): recording in the app. Ctrl+PrintScreen opens the overlay's video mode; Ctrl+PrintScreen again stops; the recording becomes a video .mcap in Screen Captures. In the app, Desktop Duplication first failed with DXGI_ERROR_UNSUPPORTED: only the main thread was DPI aware. The capture thread now makes itself aware.
    • +
  • Done Phase 5 (2026-10-10, mcap 0.1.8): sound. What the computer plays, and the microphone if asked for, are mixed in sync with the video into an AAC stream. A beep matched to a flash every 2 s stays within about 20 ms, without drifting.
  • Done Phase 4 (2026-10-10, mcap 0.1.6): the border and toolbar, both kept out of the recording; pause, with no gap in the video; discard, confirmed in the toolbar; recovery when the screen is lost; leftover recordings offered at startup; the [video] settings.
  • Later Audio is phase 5. Recording one window, and regions that span two screens, come after.
    • @@ -466,9 +467,41 @@ Captured_Frame :: struct {
    -

    Phase 5: Audio

    +

    Phase 5: Audio Done

    Section 8: swresample bindings, loopback and microphone, mixing, AAC.

    Done when: a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.

    +
    Result (2026-10-10). + +

    Changes from the plan:

    +

    10. Risks