// Default_Allocator: rpmalloc 2.0.1 for Jai, as a drop-in replacement for Jai's modules/Default_Allocator. // // Jai finds the default allocator by module name: Runtime_Support.jai sets the default context's // allocator to Default_Allocator.allocator_proc, and Process/posix.jai calls // Default_Allocator.unsafe_reset() and allocator_proc(.STARTUP) in a forked child. Put the directory // that holds this module ahead of Jai's modules directory on the import path, and every allocation // of the program, from the first one the runtime makes, goes through this allocator. // // The allocator is the port in my_modules/rpmalloc (tested by its tests/ suite). This file is the // repository's Default_Allocator/module.jai; install.jai copies it next to the port's shared files // (rpmalloc.jai, os_*.jai, version.jai) in PROJECT/modules/Default_Allocator, and only there does it // build. Upstream is dual licensed Unlicense / MIT; see LICENSE and LICENSES/. // // Read DIFFERENCES.md before switching a project over: it lists everything that behaves differently // from Jai's Default_Allocator. #module_parameters ( // The first two are Jai's Default_Allocator parameters, with the same names and defaults. // Internal consistency asserts, reported through Memory_Interface.error_callback if set, // otherwise context.assertion_failed. Upstream rpmalloc default: off. ENABLE_ASSERTS := true, // Refuse bad arguments at the public entry points: sizes at or above MAX_ALLOC_SIZE (negative // sizes included), count * size overflow in the calloc family, size + alignment overflow and // non-power-of-two alignments. Upstream rpmalloc default: off. ENABLE_VALIDATE_ARGS := true, // The rest are the port's (see my_modules/rpmalloc/module.jai). Runtime_Support imports this // module without parameters, so the defaults below are what a program's context allocator uses. // Global, per-thread and (with HEAP_STATISTICS) per-heap counters: global_statistics, // thread_statistics, dump_statistics, heap_statistics. ENABLE_STATISTICS := false, // At finalize, assert that every block was freed. Needs ENABLE_STATISTICS. ENABLE_LEAK_DETECTION := false, // Commit pages on demand, and decommit free pages beyond a per page type threshold. ENABLE_DECOMMIT := true, // Return address space to the OS (huge blocks; everything at finalize with unmap_on_finalize). ENABLE_UNMAP := true, // The heap_* API and Allocator_Mode.CREATE_HEAP / DESTROY_HEAP. FIRST_CLASS_HEAPS := true, // Per first-class heap allocated / committed / mapped sizes (heap_statistics). HEAP_STATISTICS := false, // Recently freed huge blocks kept for reuse (at most 128 MiB, 32 MiB each); 0 disables the cache. HUGE_CACHE_SLOT_COUNT := 32, // How much freed huge-block memory the cache may keep committed, in MiB; a single entry may take a // quarter of it. Upstream's 128. A program that frees and allocates huge blocks often runs faster // with more (churn_huge misses the cache on 26% of steps at 128, 5% at 256, 1% at 512), at the cost of // up to this much memory held while it does; entries idle for a second are decommitted (#jai fix E11). HUGE_CACHE_LIMIT_MB := 128, // Keep the thread heap in the Jai Context (field rpmalloc_default_allocator_thread_heap) as well as in OS thread-local storage, so // allocations and frees need no OS call to find it (#jai fix E10). Two imports of this module cannot // both add the field: turn it off for a second import with other parameters. CONTEXT_HEAP_CACHE := true, // Export internals and fault injection for the port's test suite. Adds exports only. TEST_HOOKS := false )(); #assert !ENABLE_LEAK_DETECTION || ENABLE_STATISTICS "ENABLE_LEAK_DETECTION needs ENABLE_STATISTICS"; // #jai Jai's Default_Allocator also builds for PS5, Switch 2, iOS, Android and 32-bit targets (as a // CRT malloc wrapper on most of them). The port has an OS layer for Windows, macOS and Linux only. #assert (OS == .WINDOWS || OS == .MACOS || OS == .LINUX) && (CPU == .X64 || CPU == .ARM64) "This Default_Allocator (rpmalloc 2.0.1 port) supports 64-bit Windows, macOS and Linux only. For other targets, take my_modules off the import path so Jai's own Default_Allocator is used."; /* Jai's Default_Allocator interface, as Runtime_Support, Process and other code use it: allocator Allocator.{allocator_proc, null} allocator_proc The Allocator protocol, below. unsafe_reset() For a forked child only, before it allocates. initialize_if_needed() Initialise now; true if this call did it. is_initialized() finalize() usable_size(ptr) is_this_yours(heap, ptr) heap_free_all(heap) global_statistics() Also global_statistics(*stats), the rpmalloc form. heap_statistics(heap) Not provided: allocator_default_heap. There is no single default heap; each thread has its own. Allocator protocol (allocator_proc): ALLOCATE, RESIZE, FREE The calling thread's heap, or the first-class heap in allocator_data. Out of memory returns null. With ENABLE_VALIDATE_ARGS, so does an impossible size (negative or near S64_MAX), and such a RESIZE leaves the block intact. FREE of null is a no-op. RESIZE copies old_size bytes when the block moves (all of the old block when old_size is 0). STARTUP Optional: initialize(). Returns null. SHUTDOWN finalize(). Allocating afterwards initialises again. THREAD_START, THREAD_STOP thread_initialize(), thread_finalize(). Both optional: a thread gets its heap on its first allocation, and gives it back at thread exit. CREATE_HEAP heap_acquire(): a first-class heap, to put in allocator_data. DESTROY_HEAP Frees every block still in the heap, then releases it. IS_THIS_YOURS With a heap: is the address inside a page of that heap? Without: inside a page of any thread heap (not of a first-class heap)? CAPS Everything above; CREATE_HEAP only with FIRST_CLASS_HEAPS. For a thread pool: call thread_idle() where a pool thread waits for work. It gives back what the thread's heap holds free and parks the heap, so blocks other threads free into it while it sleeps are given back at once; the thread's next allocation takes the heap back. The whole rpmalloc API of the port is exported as well: initialize, thread_collect, trim, thread_idle, rpmalloc, rpaligned_alloc, heap_acquire, and the rest (my_modules/rpmalloc/module.jai lists it). */ allocator :: Allocator.{allocator_proc, null}; allocator_proc :: (mode: Allocator_Mode, requested_size: s64, old_size: s64, old_memory: *void, allocator_data: *void) -> *void { heap := cast(*Heap) allocator_data; if #complete mode == { case .ALLOCATE; #if FIRST_CLASS_HEAPS if heap return heap_alloc(heap, requested_size); return rpmalloc(requested_size); case .RESIZE; return allocator_proc_resize(heap, old_memory, requested_size, old_size); case .FREE; #if FIRST_CLASS_HEAPS if heap { heap_free(heap, old_memory); return null; } // Null is a no-op (#jai fix U18). rpfree(old_memory); return null; case .STARTUP; initialize(); return null; // Process/posix.jai stores this in context.allocator.data: null means the thread heap. case .SHUTDOWN; finalize(); return null; case .THREAD_START; thread_initialize(); return null; case .THREAD_STOP; thread_finalize(); return null; case .CREATE_HEAP; #if FIRST_CLASS_HEAPS return heap_acquire(); else return null; case .DESTROY_HEAP; // #jai fix M4: destroying a heap frees every block still in it. #if FIRST_CLASS_HEAPS if heap { heap_free_all(heap); heap_release(heap); } return null; case .IS_THIS_YOURS; return cast(*void) cast(s64) is_this_yours(heap, old_memory); case .CAPS; if old_memory { (cast(*string) old_memory).* = CAPS_VERSION_STRING; } caps := Allocator_Caps.HINT_I_AM_A_GENERAL_HEAP_ALLOCATOR | .MULTIPLE_THREADS | .FREE | .ACTUALLY_RESIZE | .IS_THIS_YOURS; #if FIRST_CLASS_HEAPS caps |= .CREATE_HEAP; return cast(*void) caps; case; context.handling_assertion_failure = true; context.assertion_failed(#location(), "Invalid or corrupt mode passed to Default_Allocator.allocator_proc.\n"); context.handling_assertion_failure = false; return null; } } // Jai's Default_Allocator interface: initialise now, unless already done. True if this call did it. initialize_if_needed :: () -> did_init: bool { if is_initialized() return false; initialize(); return true; } is_initialized :: inline () -> bool { return atomic_load(*global_rpmalloc_init_state) == RPMALLOC_INIT_DONE; } // Jai's Default_Allocator form of global_statistics. With ENABLE_STATISTICS off (the default) every // field is 0, as with Jai's. global_statistics :: () -> Global_Statistics { stats: Global_Statistics; global_statistics(*stats); return stats; } // :AllocatorAfterFork: Only for the child of fork(), before it allocates (Process/posix.jai). The // child has one thread, the one that called fork; any lock another thread of the parent held at that // moment would never be released, so the next allocation would wait forever. This releases them. // // #jai Jai's Default_Allocator forgets its whole state here and starts afresh. This keeps it: the // forking thread's heap and every block it holds stay valid, and the heaps of the parent's other // threads stay allocated (in the child nobody frees them, as with fork and any allocator). Like Jai's, // it cannot repair a list another thread was in the middle of changing; the window is a few // instructions under the global heap lock or the huge cache lock. unsafe_reset :: () { atomic_store(*global_heap_lock, 0); global_heap_creating = 0; // Another thread was mapping a block of heaps; the child would wait for it. #if HUGE_CACHE_SLOT_COUNT atomic_store(*global_huge_cache.lock, 0); // Another thread was initialising: start again (the forking thread was not, it was in fork). if atomic_load(*global_rpmalloc_init_state) == RPMALLOC_INIT_RUNNING atomic_store(*global_rpmalloc_init_state, RPMALLOC_INIT_UNINIT); } // // Public types (2.0.1: rpmalloc.h). Kept identical to my_modules/rpmalloc/module.jai. // #load "version.jai"; // VERSION, VERSION_REVISION, RPMALLOC_VERSION RPMALLOC_MAX_ALIGNMENT :: 256 * 1024; // 2.0.1: rpmalloc.h:69 // Flags for rpaligned_realloc / heap_realloc (2.0.1: rpmalloc.h:79-84). Realloc_Flags :: enum_flags u32 { NO_PRESERVE :: 1; // Do not preserve content in reallocation. GROW_OR_FAIL :: 2; // Fail (return null) if the block cannot grow in place; the original stays valid. } // 2.0.1: rpmalloc.h rpmalloc_interface_t. Pass one to initialize / initialize_config to replace // the OS calls; it is copied, so later changes to it have no effect. If memory_map or memory_unmap // is null, all four OS defaults are used; otherwise a null memory_commit or memory_decommit uses // the OS's (#jai fix U8). Memory_Interface :: struct { // Map `size` bytes aligned to `alignment` (0 or the span size). May store an alignment offset // and the mapped size, both passed back to memory_unmap. memory_map: (size: u64, alignment: u64, offset: *u64, mapped_size: *u64) -> *void; memory_commit: (address: *void, size: u64) -> s32; // Non-zero on failure. memory_decommit: (address: *void, size: u64) -> s32; // Non-zero on failure. memory_unmap: (address: *void, offset: u64, mapped_size: u64); map_fail_callback: (size: u64) -> s32; // Non-zero: retry the map. error_callback: (message: string); } // 2.0.1: rpmalloc.h rpmalloc_config_t. Config :: struct { page_size: u64; // 0: the OS page size. enable_huge_pages: bool; enable_thp: bool; // Linux / Android only. disable_decommit: bool; page_name: string; huge_page_name: string; unmap_on_finalize: bool; disable_thp: bool; // Linux / Android only. } // 2.0.1: rpmalloc.h rpmalloc_global_statistics_t. All zero without ENABLE_STATISTICS. The memory // committed now is active, not committed: committed and decommitted are running totals, as upstream. Global_Statistics :: struct { mapped: u64; // Address space mapped now, committed or not. mapped_peak: u64; committed: u64; // Running total of bytes ever committed; it only grows. decommitted: u64; // Running total of bytes ever decommitted; it only grows. active: u64; // Bytes committed now: what the OS counts (API-07 checks it). active_peak: u64; huge_alloc: u64; // Bytes in huge blocks allocated now. huge_alloc_peak: u64; heap_count: u64; // Heaps in use now. } // 2.0.1: rpmalloc.h rpmalloc_thread_statistics_t. Thread_Statistics :: struct { sizecache: u64; // Bytes available in thread size class caches. spancache: u64; // Bytes available in thread page caches. span_use: [5] struct { current: u64; map_calls: u64; }; size_use: [128] struct { alloc_current: u64; alloc_peak: u64; alloc_total: u64; free_total: u64; }; } // 2.0.1: rpmalloc.h rpmalloc_heap_statistics_t. Heap_Statistics :: struct { allocated_size: u64; committed_size: u64; mapped_size: u64; } #scope_module // #jai fix E10: the thread heap, cached in the Context (see get_thread_heap in rpmalloc.jai). #if CONTEXT_HEAP_CACHE { #add_context rpmalloc_default_allocator_thread_heap: *void; context_thread_heap :: () -> **void #expand { return *context.rpmalloc_default_allocator_thread_heap; } } // A literal, not built from VERSION with #run: a #run would wait for the default allocator, which is // this module, whose CAPS needs this string. The assert keeps it in step with version.jai. CAPS_VERSION_STRING :: "rpmalloc 2.0.1-jai.16 (Jai port) as Default_Allocator"; #assert #run text_is(CAPS_VERSION_STRING, "rpmalloc ", VERSION, " (Jai port) as Default_Allocator") "Update CAPS_VERSION_STRING to the VERSION in version.jai"; #load "rpmalloc.jai"; #if OS == .WINDOWS #load "os_windows.jai"; else #load "os_posix.jai";