RipGrep musl Segfault: Debugging Linux Binary Compatibility Issues

RipGrep is one of the most popular developer tools—a blazingly fast code search utility used by developers worldwide. But a recent GitHub issue reported that musl-linked RipGrep binaries occasionally segfault during very large searches, affecting users who rely on statically-linked binaries for portability.

This isn’t just a ripgrep bug—it’s a window into a deeper issue that affects anyone building or distributing Linux CLI tools: the trade-offs between musl and glibc, static versus dynamic linking, and the subtle compatibility issues that arise from these choices.

If you maintain a CLI tool, distribute Linux binaries, or have ever wondered why some tools ship separate musl and glibc builds, this case study will show you what’s at stake and how to debug these issues when they arise.

TL;DR

  • What happened: RipGrep 15.2.0’s musl-linked binaries segfault on very large searches (~20GB, 1.8M files) due to a Linux kernel race condition in zap_empty_pte_table (introduced in Linux 7.0)
  • Why it matters: musl’s stricter memory allocator detects the kernel’s memory corruption and crashes; glibc silently ignores it. This shows why “stricter” is often better than “permissive”
  • Root cause: Not a musl bug or stack-size issue—a kernel TLB shootdown race during concurrent munmap operations
  • Status: No fix in ripgrep releases as of July 2026; requires kernel patch or workarounds
  • Workarounds: Use glibc binaries, reduce concurrency with -j1, use Linux 6.19.10 or earlier, or switch allocators

What Are musl and glibc?

Before we dive into the segfault, let’s clarify what we’re talking about:

glibc (GNU C Library)

  • What it is: The standard C library used by most Linux distributions
  • Characteristics: Feature-rich, optimized for performance, large codebase
  • Linking: Typically dynamically linked, requiring the library to be present on the target system
  • Compatibility: Highly compatible but version-dependent

musl libc

  • What it is: A lightweight, standards-compliant C library designed for static linking
  • Characteristics: Small, clean codebase with emphasis on correctness and simplicity
  • Linking: Designed for static linking, producing standalone binaries
  • Portability: Binaries work across different Linux distributions without dependency issues

Static vs Dynamic Linking

Dynamic Linking (typical with glibc):

$ ldd /usr/bin/rg
    linux-vdso.so.1 (0x00007ffd9e3e6000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8b2e200000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8b2e000000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f8b2e600000)

The binary depends on system libraries. Smaller binary size, but requires compatible libraries on the target system.

Static Linking (typical with musl):

$ ldd rg-musl
    not a dynamic executable

All code is bundled into a single binary. Larger file size, but runs anywhere without dependencies.

The RipGrep Segfault: What Went Wrong?

The issue manifested in RipGrep 15.2.0 during very large searches:

Symptom: Segmentation fault during very large searches
Affected: musl-linked binaries (15.2.0) only
Unaffected: glibc-linked binaries
Status: Not fixed in subsequent releases as of July 2026 Trigger: ~20GB of files across 1.8M files, high concurrency (24+ cores), typically crashes within ~1 minute of repeated searches

The Root Cause: A Linux Kernel Race Condition

The segfault is not a musl bug or stack-size issue. It’s a Linux kernel race condition introduced in Linux 7.0 in the zap_empty_pte_table PTE-reclaim logic during munmap TLB shootdown.

What happens: When multiple threads call munmap concurrently (which RipGrep does during parallel searching), the kernel’s TLB shootdown can transiently expose the kernel zero page. A thread’s own writes to a freshly-faulted page become invisible—the data written is lost. musl’s mallocng allocator detects this inconsistency and crashes as a safety measure. glibc’s allocator doesn’t perform the same consistency check, so it doesn’t crash—but the underlying kernel bug is still present.

Key insight: This isn’t a case of “musl is broken.” It’s that musl’s allocator is stricter about detecting memory corruption, which is actually the correct behavior. The bug is in the Linux kernel, and ripgrep’s high concurrency on large datasets happens to trigger it reliably.

For technical details, see the kernel race condition analysis.

Debugging the Segfault

When you encounter this issue, here’s how to verify it:

1. Reproduce the Issue

# Create a test case with a very large directory structure
mkdir -p test-large
cd test-large

# Generate ~1.8M files to trigger the race condition
# (this is simplified; the real scenario involves tree-like structures)
for i in {1..100000}; do
  mkdir -p "dir$i"
  echo "test content $i" > "dir$i/file.txt"
done

# Compare musl vs glibc builds
./rg-musl "test" .      # Crashes with kernel race condition on high concurrency
./rg-glibc "test" .     # Works (doesn't crash, but kernel bug still present)

2. Check Your Setup

# Verify kernel version and ripgrep version
uname -a
rg --version

# Note: bug is present in Linux 7.0+ and ripgrep 15.2.0
# Workaround: use Linux 6.19.10 or earlier, or adjust concurrency

3. Apply Workarounds

# Option 1: Switch to glibc binary
rg-glibc "search-term" /path/to/large/directory

# Option 2: Reduce concurrency (avoids triggering the race)
rg -j1 "search-term" /path/to/large/directory

# Option 3: Use Linux 6.19.10 or earlier (patch not available for 7.0+)

# Option 4: Use alternative allocators for musl builds (mimalloc shows ~20x improvement)

Common Musl vs Glibc Differences That Cause Issues

While the RipGrep segfault is specifically a Linux kernel issue, musl and glibc do differ in ways that can affect CLI tools:

1. Allocator Strictness (Key to the RipGrep Issue)

musl’s mallocng allocator performs stricter consistency checks on allocated memory. When the kernel bug exposes the zero page during concurrent munmap operations, writes to freshly-faulted pages become invisible. musl detects this corruption and crashes—which is actually the correct behavior for data integrity.

glibc’s allocator is more permissive and doesn’t detect this corruption, allowing the process to continue with silent data loss.

Lesson: Don’t assume a crash in musl means musl is broken. It often means musl is catching a real bug that glibc is silently ignoring.

2. Stack Size Defaults

glibc: Default stack size is typically 8 MB
musl: Default stack size can be smaller (but this is not the RipGrep issue)

Deep recursion or large stack-allocated buffers can overflow musl’s stack:

// This might work with glibc but segfault with musl
void process_directory(const char *path) {
    char buffer[4096];  // Stack-allocated
    // ... recursive directory traversal
    process_directory(subdir);
}

Fix: Use heap allocation for large buffers or increase stack size at runtime:

#include <sys/resource.h>

void set_stack_size(size_t size) {
    struct rlimit rl;
    rl.rlim_cur = size;
    rl.rlim_max = size;
    setrlimit(RLIMIT_STACK, &rl);
}

3. Thread Stack Size

musl’s default thread stack size may also differ:

// Create thread with explicit stack size
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 2 * 1024 * 1024);  // 2 MB
pthread_create(&thread, &attr, worker, NULL);

4. Symbol Versioning

glibc uses symbol versioning to maintain ABI compatibility. musl doesn’t:

# glibc symbols include version info
nm /lib/x86_64-linux-gnu/libc.so.6 | grep malloc
# shows: malloc@@GLIBC_2.2.5

# musl symbols are unversioned
nm /lib/x86_64-linux-musl/libc.so | grep malloc
# shows: malloc

Practical Guidance for CLI Tool Developers

If you’re building and distributing Linux CLI tools, here’s how to navigate these trade-offs:

Choose Your Linking Strategy

Use Static Musl Linking When:

  • Portability across distributions is critical
  • You want a single binary that works everywhere
  • Binary size increase is acceptable
  • Your tool doesn’t need glibc-specific features

Use Dynamic Glibc Linking When:

  • You need glibc-specific functionality
  • Binary size must be minimized
  • Your tool is distribution-packaged (not standalone download)
  • Deep integration with system libraries is required

Ship Both Variants When:

  • Your user base is diverse
  • You can maintain both in CI/CD
  • You want to give users the choice

Testing Strategy

# .github/workflows/test.yml
name: Binary Compatibility Tests

on: [push, pull_request]

jobs:
  test-musl:
    runs-on: ubuntu-latest
    container: alpine:latest
    steps:
      - uses: actions/checkout@v3
      - name: Build musl binary
        run: |
          apk add rust cargo
          cargo build --release --target x86_64-unknown-linux-musl
      
      - name: Test with large dataset
        run: |
          # Generate large test case
          ./scripts/generate-large-test.sh
          
          # Test binary under realistic load
          ulimit -v 1048576  # Limit virtual memory
          ./target/x86_64-unknown-linux-musl/release/mytool --search test ./large-dataset

  test-glibc:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build glibc binary
        run: cargo build --release --target x86_64-unknown-linux-gnu
      
      - name: Test with large dataset
        run: |
          ./scripts/generate-large-test.sh
          ./target/x86_64-unknown-linux-gnu/release/mytool --search test ./large-dataset

Runtime Checks and Graceful Degradation

// Example: Detect and handle stack size issues
fn check_stack_size() -> Result<(), Box<dyn std::error::Error>> {
    use libc::{getrlimit, rlimit, RLIMIT_STACK};
    
    let mut rlim = rlimit {
        rlim_cur: 0,
        rlim_max: 0,
    };
    
    unsafe {
        if getrlimit(RLIMIT_STACK, &mut rlim) == 0 {
            let stack_mb = rlim.rlim_cur / (1024 * 1024);
            
            if stack_mb < 2 {
                eprintln!("Warning: Stack size is {}MB, which may be too small", stack_mb);
                eprintln!("Consider running: ulimit -s 8192");
            }
        }
    }
    
    Ok(())
}

Document the Trade-offs

In your README or documentation:

## Installation

### Pre-built Binaries

We provide two Linux binary variants:

- **`mytool-linux-x86_64-musl`**: Static binary, works on any Linux distribution
  - ✅ No dependencies, runs anywhere
  - ⚠️ Larger file size (~5MB vs 2MB)
  - ⚠️ May require `ulimit -s 8192` for very large workloads

- **`mytool-linux-x86_64-gnu`**: Dynamic binary, requires glibc 2.31+
  - ✅ Smaller file size
  - ✅ Better performance on very large datasets
  - ⚠️ Requires compatible glibc version

If unsure, use the musl variant. Switch to gnu if you encounter stack size issues.

Resolution and Workarounds

For RipGrep users encountering the 15.2.0 musl segfault on large searches:

Immediate Workarounds:

  1. Switch to glibc binary: Use the x86_64-linux-gnu variant instead (doesn’t crash, though kernel bug still present)
  2. Reduce concurrency: rg -j1 "search-term" /path/to/large/directory (avoids triggering the race condition)
  3. Downgrade kernel: Use Linux 6.19.10 or earlier (patch not available for 7.0+)
  4. Use alternative allocator: Build ripgrep with mimalloc instead of mallocng (~20x improvement reported)

For Tool Maintainers: The RipGrep maintainers are aware of the issue (GitHub #3494). Since this is a Linux kernel race condition, not a ripgrep bug, it requires either:

  • Users to upgrade to a patched kernel (once available)
  • Reducing concurrency in the tool’s internal threading
  • Using alternative memory allocators

At present, there is no release fix for ripgrep 15.2.0; workarounds are the recommended approach.

Key Takeaways

  1. Stricter detection isn’t a bug: When musl crashes and glibc doesn’t, it often means musl found a real corruption that glibc is silently accepting. Stricter checking is a feature, not a flaw.
  2. Not every musl issue is a musl problem: The RipGrep segfault is a kernel bug, not a C library bug. Always dig into the root cause before blaming the allocator.
  3. musl and glibc are not drop-in replacements: Differences in memory allocators, symbol versioning, and stack defaults can expose hidden system bugs.
  4. High-concurrency + large workloads = good test cases: The bug only manifests under specific conditions (high concurrency on large datasets). Your integration tests need realistic scale.
  5. Test both variants: If you ship both musl and glibc binaries, test against realistic production workloads with both. They behave differently under kernel stress.
  6. Document your workarounds: Users need to know which variant to use and what kernel versions are affected.

When you hit a segfault affecting only one libc variant, remember: the crash might be the right behavior, and the smooth operation might be hiding a bug.

Linux binary compatibility is complex, but understanding these fundamentals helps you debug issues faster and make informed trade-offs when distributing your tools.

Sources

Have you encountered musl vs glibc compatibility issues in your projects? What debugging techniques worked for you? Let us know in the comments.