Comment 42 for bug 490024

Created attachment 2187
Proposed fix.

Sorry about the broken bug; I hit enter accidentally.

The failing assertion is this one:
  assert(!victim || chunk_is_mmapped(mem2chunk(victim)) ||
  ar_ptr == arena_for_chunk(mem2chunk(victim)));

GDB's bigcore.c testcase triggers this assertion on several PowerPC systems I
tested. It starts by a malloc too large for the system to satisfy; when
_int_malloc fails, malloc creates and tries a new arena. This arena is saved
as the default arena for the main thread so future allocations come from that
arena instead of the main one.

Later the test tries a malloc which can be met by mmap. Eventually mmap
returns ENOMEM after a number of similar allocations:

mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xa810c000
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb010d000
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
-1 ENOMEM (Cannot alloca
te memory)
mmap(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
-1 ENOMEM (Cannot alloca
te memory)
brk(0x180c1000) = 0x180c1000

I do not know why brk succeeded (another seven times, all 0x8000000 bytes) when
mmap failed. But the result is a non-mmapped chunk allocated from the main
arena. The assert checks the thread's specific arena and fails. Updating
ar_ptr fixes the failure.

Patch attached.