Today I Learned

6/7

MADV_POPULATE_READ – it's a madvise which warms up the page cache + populate page table. It's a good replacement for doing compiler hacks to ignore "dead code" (which is actually for prefetching some data)

26/6

 [INFO] [06-26|07:58:01.737] [mem] memory stats                       Rss=54.1GB Size=0B Pss=53.7GB SharedClean=847.1MB SharedDirty=0B PrivateClean=6.7GB PrivateDirty=46.6GB Referenced=53.9GB Anonymous=46.0GB Swap=0B alloc=29.2GB sys=46.2GB

• Anonymous is about – this page can't be linked back to a file; this can happen for heap allocations of a program + anonymous mmaps
• RSS is the total mem used by the process - Anonymous+ file-backed memory (mmap); so used by mmap is RSS-anonymous; this is rough, RSS also includes shared libs and binary text
• Anonymous can be shared – cow pages after process fork; so SharedDirty and PrivateDirty; above SharedDirty=0, so all anonymous pages are private to the process

25/6

the pages madvise hints cause to be pulled

no madvise / MADVISE_NORMAL – 32 pages (centered i.e. 16 behind, 15 ahead) 128kb MADV_RANDOM - 1 page (4kb) MADV_SEQUENTIAL - 32 pages (128kb) - all 32 ahead of the fault

22/6

https://en.wikipedia.org/wiki/Amdahl's_law

21/6

Aho-Corasick algorithm – used to match words to a fixed dictionary of patterns.

15/6

in golang, allocating slice using make can allocate on stack if the slice doesn't escape (stack analysis does this) and its size is less than MaxImplicitStackVarSize (64kb).

• make([]int, 32) – non-escaping => on stack
• make([]int, n) – non-escaping => compiler stack allocates, but inserts a runtime sizeguard which at runtime can allocate on stack or on heap (using runtime.makeslice)
• it the slice escapes – it's always heap

10/6

agent forwarding – it allows to send data from server A to server B, from your local machine.

ssh-add ~/.ssh/id_ed25519_erigon   # load key into agent if not already

ssh -A -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_erigon user1@server1 \
  'rsync -P --inplace -e "ssh -o StrictHostKeyChecking=accept-new -c aes128-ctr" \
    /source_file \
    user2@server1:/destination'

1/6

USE framework for performance characterization of linux machine

4/5

golang bit: bytes.Clone gets attributed with the allocated bytes (and not the caller) in pprof. To avoid this use copy functionality

10/4/

du vs df

I've been using df to find free/used disk space; and du for breakdowns or total. Typically they should report the same "used space" value, but it can differ sometimes. e.g. df tells how many blocks are allocated and that can include files which have been "rm'ed but not closed" or "rm'ed/closed but not munmapped" by running processes.

df reports the value from the filesystem's perspective. du walks the directory tree, stat-ing each inode and summing up the st_blocks field (actual blocks allocated on disk, excluding holes; com,pare with st_size which includes holes).

This is the reason for some discrepancies between what du vs df might report: - "unlinked but held" files by running processes will show up in df (because block is still reserved) but not in du (because no inode) - filesystem metadata overhead: journal/inode tables etc. are not "files" so du won't see and report space used by them - CoW filesystems: df might account reused blocks (across different files) once, but du might see them separately.

du and df actually agree on sparse files.

9/4

/proc/<pid>/maps contain info about the process' VMA. Each line represents one VMA.

6d9dffa00000-6d9e01724000 r--s 00000000 09:03 166637280 /erigon-data/temp/erigon-sortable-buf-2634549773 (deleted)

• 6d9dffa00000-6d9e01724000: range of virtual memory address
• rwxp: read, write, execute, (p)rivate or (s)hared
• 166637280: inode number of backing file

in general you can look at complete format in man pages: man proc_pid_maps

27/3

• that jq tool language is turing complete. I always imagined it processes the passed json through path-traversal, transforming or pruning the branches (and turns out tools like json_pp do this);
• query -> AST -> bytecode for jq's stack VM
• use of generator as a way to produce 0,1 or many values. Everything is a generator

generators: - everything is a generator - .foo: a generator that yields 1 value - .[]: yields N values - select(cond): generator that yields 0 or 1 values..

24/3

Reservoir Sampling and Vitter's algorithm

Reservoir sampling is an algorithm for selecting k items uniformly at random from a stream of unknown or very large size n, using only O(k) memory.

Algorithm (k=1): 1. Keep the first item 2. For the i-th item, replace the kept item with probability 1/i

Algorithm (k items) — Vitter's Algorithm R: 1. Fill a reservoir array with the first k items 2. For each subsequent item i (where i > k), pick a random number j in [1, i] 3. If j ≤ k, replace reservoir[j] with item i

Why it works: After processing all n items, each item has exactly a k/n probability of being in the reservoir.

Use cases: - Sampling from a stream where you don't know the total size upfront - Sampling from data too large to fit in memory - Random line selection from a file (e.g., shuf -n 1 uses this internally)

4/3

about llms.txt and how it can enable agent-friendly documentation. Companies like supabase and resend are already using it. Here's my PR for erigon and turns out agents like it.

23/2

it's possible to have multiple VMA (virtual memory area) per file-descriptor. This is useful in cases where you have different usecases for the file, and each of those usecase have different access patterns which can benefit from MADVISE_* hints. Both mappings get mapped by the same page cache pages, so there's no "double memory cost".

source: https://github.com/erigontech/erigon/pull/19397/changes