A Battle Without a Victory: Chasing a glibc freeres Bug中文EN

I first ran into this problem over a year ago, but since it wasn't causing any trouble I never followed up on it.
Recently a new hire on the team triggered it again, so I finally resolved to dig into the cause of this bug once and for all..
First off, this post is a stream-of-consciousness log and frankly not very interesting. Second, I never actually found the true root cause — I only found a workaround and a hypothesis about what might be going on. If you're in a similar situation, maybe parts of this will help. And of course, I'd still love it if someone who knows the glibc source inside-out could weigh in with the real reason.
Here's the problem. One of our services depends on the Yar C library, but valgrind reports the following:

==25279== Invalid free() / delete / delete[] / realloc()
==25279==    at 0x4A072BA: free (vg_replace_malloc.c:446)
==25279==    by 0x34DC10ADAA: free_mem (in /lib64/libc-2.5.so)
==25279==    by 0x34DC10A9A1: __libc_freeres (in /lib64/libc-2.5.so)
==25279==    by 0x48025E9: _vgnU_freeres (vg_preloaded.c:62)
==25279==    by 0x34DC0334E4: exit (in /lib64/libc-2.5.so)
==25279==    by 0x34DC01D99A: (below main) (in /lib64/libc-2.5.so)
==25279==  Address 0x4e265c8 is not stack'd, malloc'd or (recently) free'd

Naturally, the first thing I did was Google it. A lot of the answers pointed to this being a bug in libc_freeres in older glibc versions (ours is glibc-2.5), and that passing --run-libc-freeres=no would sidestep the warning. I verified that it works.
Fine — at this point I was relieved it wasn't our code. But that doesn't sit well with our "question everything until it breaks" spirit. What about ours actually triggers this warning? Can I pin down the cause and work around it, given that some other libraries I tried don't have this problem?
First, let's try to figure out what that address actually is. After some pruning, I got down to this minimal reproduction. First, main.c, which does absolutely nothing:

//file main.c
int main(int argc, char **argv) {
    return 0;
}

Then compile and link the yar library:

gcc -Wl,-rpath,/home/huixinchen/local/yar/lib/ -L/home/huixinchen/local/yar/lib/ -lyar main.c

Test:

$ valgrind --run-libc-freeres=yes ./a.out
==7008== Memcheck, a memory error detector
==7008== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==7008== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==7008== Command: ./a.out
==7008==
==7008== Invalid free() / delete / delete[] / realloc()
==7008==    at 0x4A072BA: free (vg_replace_malloc.c:446)
==7008==    by 0x34DC10ADAA: free_mem (in /lib64/libc-2.5.so)
==7008==    by 0x34DC10A9A1: __libc_freeres (in /lib64/libc-2.5.so)
==7008==    by 0x48025E9: _vgnU_freeres (vg_preloaded.c:62)
==7008==    by 0x34DC0334E4: exit (in /lib64/libc-2.5.so)
==7008==    by 0x34DC01D99A: (below main) (in /lib64/libc-2.5.so)
==7008==  Address 0x4e265c8 is not stack'd, malloc'd or (recently) free'd
==7008==

On an X86 64-bit system, 0x4e265c8 looks a lot like the address of a global variable in the memory layout. But what could it be? A global in .bss? I tried sleeping and then cat'ing /proc/pid/maps, but valgrind changes the layout of loaded shared libraries, so first I need a way to reproduce this without valgrind.
Since this is supposedly a __libc_freeres problem, can we just call it directly to surface it?

//file main.c
extern void __libc_freeres();
int main(int argc, char **argv) {
    __libc_freeres();
    return 0;
}

Test:

$ ./a.out
*** glibc detected *** ./a.out: munmap_chunk(): invalid pointer: 0x00002ab9e8a21770 ***
======= Backtrace: =========
/lib64/libc.so.6(cfree+0x166)[0x34dc071756]
/lib64/libc.so.6[0x34dc10adab]
/lib64/libc.so.6(__libc_freeres+0x42)[0x34dc10a9a2]
./a.out[0x4005d1]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x34dc01d994]
./a.out[0x400509]

Sure enough — it triggers directly, no valgrind needed. But the yar library is huge and a pain to debug. Can we simplify further?
After a long, painful process (which I'll skip over), I finally got down to an even easier-to-reproduce library. Guess what it looks like?

#shared library

Yes, you read that right — the library has no content at all. Now compile it (as libtest.so):

gcc -shared -fPIC empty.c -Wl,-rpath,/home/huixinchen/dev/ -lm -o libtest.so

(Don't ask why rpath and -lm are both required. It's because I worked it out the hard way, during that long painful process. But if you must know: Yar uses them in a similar way, and I found this through elimination and a bit of Google... *sweat*)
Now link main.c against this new library:

$ gcc -g -o mem-error main.c ./libtest.so
$ ./mem-error
*** glibc detected *** ./mem-error: free(): invalid pointer: 0x00002b1a30252660 ***
======= Backtrace: =========
/lib64/libc.so.6[0x34dc0711df]
/lib64/libc.so.6(cfree+0x4b)[0x34dc07163b]
/lib64/libc.so.6[0x34dc10adab]
/lib64/libc.so.6(__libc_freeres+0x42)[0x34dc10a9a2]
./mem-error(main+0xe)[0x4005c6]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x34dc01d994]
./mem-error[0x400509]
======= Memory map: ========
00400000-00401000 r-xp 00000000 68:11 17932888                           /home/huixinchen/dev/mem-error
00600000-00601000 rw-p 00000000 68:11 17932888                           /home/huixinchen/dev/mem-error
04999000-049ba000 rw-p 04999000 00:00 0                                  [heap]
304a600000-304a60d000 r-xp 00000000 68:01 950291                         /lib64/libgcc_s-4.1.2-20080825.so.1
304a60d000-304a80d000 ---p 0000d000 68:01 950291                         /lib64/libgcc_s-4.1.2-20080825.so.1
304a80d000-304a80e000 rw-p 0000d000 68:01 950291                         /lib64/libgcc_s-4.1.2-20080825.so.1
31a6a00000-31a6a82000 r-xp 00000000 68:01 950274                         /lib64/libm-2.5.so
31a6a82000-31a6c81000 ---p 00082000 68:01 950274                         /lib64/libm-2.5.so
31a6c81000-31a6c82000 r--p 00081000 68:01 950274                         /lib64/libm-2.5.so
31a6c82000-31a6c83000 rw-p 00082000 68:01 950274                         /lib64/libm-2.5.so
34dbc00000-34dbc1c000 r-xp 00000000 68:01 950586                         /lib64/ld-2.5.so
34dbe1c000-34dbe1d000 r--p 0001c000 68:01 950586                         /lib64/ld-2.5.so
34dbe1d000-34dbe1e000 rw-p 0001d000 68:01 950586                         /lib64/ld-2.5.so
34dc000000-34dc14e000 r-xp 00000000 68:01 950587                         /lib64/libc-2.5.so
34dc14e000-34dc34d000 ---p 0014e000 68:01 950587                         /lib64/libc-2.5.so
34dc34d000-34dc351000 r--p 0014d000 68:01 950587                         /lib64/libc-2.5.so
34dc351000-34dc352000 rw-p 00151000 68:01 950587                         /lib64/libc-2.5.so
34dc352000-34dc357000 rw-p 34dc352000 00:00 0
2b1a30251000-2b1a30253000 rw-p 2b1a30251000 00:00 0
2b1a30253000-2b1a30254000 r-xp 00000000 68:11 17932574                   /home/huixinchen/dev/libtest.so
2b1a30254000-2b1a30453000 ---p 00001000 68:11 17932574                   /home/huixinchen/dev/libtest.so
2b1a30453000-2b1a30454000 rw-p 00000000 68:11 17932574                   /home/huixinchen/dev/libtest.so
 

So the address (0x00002b1a30252660) falls before libtest.so. It can't be one of our globals — and besides, there's no object in the code at all.... (That earlier address was indeed an artifact of valgrind.)
At this point we can be almost certain it's not caused by our own code. So what on earth is going on?
No choice — back to valgrind (revert main.c to the original version, don't call freeres directly). In the spirit of "might as well try," I attached with gdb:

0x0000000004a072ba in _vgr10050ZU_libcZdsoZa_free (p=0x4e265c8) at m_replacemalloc/vg_replace_malloc.c:446
446	 FREE(VG_Z_LIBC_SONAME,       free,                 free );
(gdb) bt
#0  0x0000000004a072ba in _vgr10050ZU_libcZdsoZa_free (p=0x4e265c8) at m_replacemalloc/vg_replace_malloc.c:446
#1  0x00000034dc10adab in free_mem () from /lib64/libc.so.6
#2  0x00000034dc10a9a2 in __libc_freeres () from /lib64/libc.so.6
#3  0x00000000048025ea in _vgnU_freeres () at vg_preloaded.c:62
#4  0x00000034dc0334e5 in exit () from /lib64/libc.so.6
#5  0x00000034dc01d99b in __libc_start_main () from /lib64/libc.so.6
#6  0x00000000004004b9 in _start ()

Wait — something's off here. Why does it look like it's freeing the SONAME? I never specified a SONAME!
So... what if I do specify one?

$ gcc -shared -fPIC empty.c -Wl,-rpath,/home/huixinchen/dev/  -Wl,-soname,libm.so -lm -o libtest.so
$ objdump -x libtest.so  | grep SONAME
  SONAME               libm.so

Then — you'll never guess what?
The problem disappeared..... Hahaha (this took from yesterday afternoon, through yesterday evening, to this morning :<) So it looks like: in older glibc versions, when you have rpath and a dependent library, freeres tries to free something without correctly checking whether a soname exists.. and that's what causes the error... Of course, this is just a guess based on surface evidence. If you know the real reason, please do leave a comment. Or maybe in another year I'll dig deeper again.... 🙂 (Postscript: let me sum up in two words — learning never ends. For someone who has never read the glibc source, this process was agonizing. You can only guess boldly and verify carefully. But human energy is finite; how can you possibly understand every single knowledge point you touch? The only answer is endless learning... ) thanks