heap-use-after-free in object_ref_cleanup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

heap-use-after-free in object_ref_cleanup

Orivej Desh
Hello,

I noticed in dmesg that in my repository svnserve occasionally crashes.
This happens at exit, so it is not visible to end users.  I captured a
few sessions at the svn protocol level that resulted in a crash; client
commands are quite different in each one; sending an input that crashed
`svnserve -t` again always crashes it; I did not manage to reproduce
this in a new repository.  I tested with subversion 1.9.0, 1.9.4 and
subversion trunk, apr 1.3.0 and apr 1.5.2 with the same result.

Here is the AddressSanitizer report of the problem (with subversion
1.9.4 and apr 1.3.0).  For me it seems that a piece of memory belonging
to the object_pool is registered for run_cleanups in the main pool, then
right before return from main() all child pools of the main pool are
destroyed, then run_cleanups of the main pool calls object_ref_cleanup
with a pointer to a freed memory.  Is this correct?  How is this
supposed to work?  Could you help me spot the difference between
expected and unexpected paths of code concerning the issue?

==329484==ERROR: AddressSanitizer: heap-use-after-free on address 0x6250000234d0 at pc 0x00000077f3f9 bp 0x7ffc53dac920 sp 0x7ffc53dac918
READ of size 8 at 0x6250000234d0 thread T0
    #0 0x77f3f8 in object_ref_cleanup /home/orivej/src/subversion/subversion/libsvn_subr/object_pool.c:148:45
    #1 0x9b7dd0 in run_cleanups /home/orivej/src/apr/memory/unix/apr_pools.c:2298:9
    #2 0x9b66e4 in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:783:5
    #3 0x5bbb65 in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1368:3
    #4 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #5 0x4edac8 in _start (/home/orivej/bin/svnserve+0x4edac8)

0x6250000234d0 is located 976 bytes inside of 8192-byte region [0x625000023100,0x625000025100)
freed by thread T0 here:
    #0 0x58bf9b in free /home/orivej/src/compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
    #1 0x9b5991 in allocator_free /home/orivej/src/apr/memory/unix/apr_pools.c:407:9
    #2 0x9b6a4f in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:825:5
    #3 0x9b66cf in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:780:9
    #4 0x5bbb65 in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1368:3
    #5 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

previously allocated by thread T0 here:
    #0 0x58c2ec in malloc /home/orivej/src/compiler-rt/lib/asan/asan_malloc_linux.cc:64:3
    #1 0x9b516e in allocator_alloc /home/orivej/src/apr/memory/unix/apr_pools.c:330:17
    #2 0x9b5d5a in apr_pool_create_ex /home/orivej/src/apr/memory/unix/apr_pools.c:860:17
    #3 0x62986e in svn_pool_create_ex /home/orivej/src/subversion/subversion/libsvn_subr/pool.c:70:3
    #4 0x77e654 in svn_object_pool__new_wrapper_pool /home/orivej/src/subversion/subversion/libsvn_subr/object_pool.c:356:10
    #5 0x6f04ae in auto_parse /home/orivej/src/subversion/subversion/libsvn_repos/config_pool.c:215:14
    #6 0x6eeab1 in svn_repos__config_pool_get /home/orivej/src/subversion/subversion/libsvn_repos/config_pool.c:523:17
    #7 0x5be05c in sub_main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1002:7
    #8 0x5bba6d in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1347:9
    #9 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap-use-after-free in object_ref_cleanup

Daniel Shahaf-2
Orivej Desh wrote on Tue, 23 May 2017 08:20 +0000:

> Hello,
>
> I noticed in dmesg that in my repository svnserve occasionally crashes.
> This happens at exit, so it is not visible to end users.  I captured a
> few sessions at the svn protocol level that resulted in a crash; client
> commands are quite different in each one; sending an input that crashed
> `svnserve -t` again always crashes it; I did not manage to reproduce
> this in a new repository.  I tested with subversion 1.9.0, 1.9.4 and
> subversion trunk, apr 1.3.0 and apr 1.5.2 with the same result.
>

I cannot reproduce this with apr 1.5.1-3 (debian), svn 1.9.4 (self compiled),
and clang 3.5, under 'make svnserveautocheck'.  I use 1.9.4 because that's the
traceback you posted; let's do further analysis on trunk if the problem
reproduces there.

> Here is the AddressSanitizer report of the problem (with subversion
> 1.9.4 and apr 1.3.0).  For me it seems that a piece of memory belonging
> to the object_pool is registered for run_cleanups in the main pool, then
> right before return from main() all child pools of the main pool are
> destroyed, then run_cleanups of the main pool calls object_ref_cleanup
> with a pointer to a freed memory.  Is this correct?  How is this
> supposed to work?  Could you help me spot the difference between
> expected and unexpected paths of code concerning the issue?
>

The report says that the use-after-free occured inside the cleanup handler.  It
doesn't say where the accessed object was allocated or freed; to get that info,
you'd have had to compile APR with pool debugging (--enable-pool-debug), then
the second and third traceback would have pointed to the site of the malloc()
and of the apr_pool_clear().

Cleanup handlers work as follows: «apr_pool_cleanup_register(pool, baton, f, g)»
calls «f(baton)» when POOL is cleared.  If BATON had been free()d before POOL
is cleared, then F will access freed memory.  

Therefore, there are a few options to fixing such issues. One could install the
cleanup handler F on a different pool, or duplicate the BATON object into a
different pool, handler is invoked), or sometimes to use apr_pool_pre_cleanup_register()
instead.  I'm not sure which of these options is best in this case (not
familiar with this module).

The pool cleanup is registered in add_object_ref().  Can you break in that
function and confirm which of its two callsites is involved?

Thanks for the report!

Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap-use-after-free in object_ref_cleanup

Stefan Fuhrmann
In reply to this post by Orivej Desh
On 23.05.2017 10:20, Orivej Desh wrote:

> Hello,
>
> I noticed in dmesg that in my repository svnserve occasionally crashes.
> This happens at exit, so it is not visible to end users.  I captured a
> few sessions at the svn protocol level that resulted in a crash; client
> commands are quite different in each one; sending an input that crashed
> `svnserve -t` again always crashes it; I did not manage to reproduce
> this in a new repository.  I tested with subversion 1.9.0, 1.9.4 and
> subversion trunk, apr 1.3.0 and apr 1.5.2 with the same result.
>
> Here is the AddressSanitizer report of the problem (with subversion
> 1.9.4 and apr 1.3.0).  For me it seems that a piece of memory belonging
> to the object_pool is registered for run_cleanups in the main pool, then
> right before return from main() all child pools of the main pool are
> destroyed, then run_cleanups of the main pool calls object_ref_cleanup
> with a pointer to a freed memory.  Is this correct?  How is this
> supposed to work?  Could you help me spot the difference between
> expected and unexpected paths of code concerning the issue?
Hi Orivej,

thanks for reporting this!

The callstacks suggests that this is a pool cleanup race.
Please try the attached patch and report the results.

Thanks you!

-- Stefan^2.


>
> ==329484==ERROR: AddressSanitizer: heap-use-after-free on address 0x6250000234d0 at pc 0x00000077f3f9 bp 0x7ffc53dac920 sp 0x7ffc53dac918
> READ of size 8 at 0x6250000234d0 thread T0
>      #0 0x77f3f8 in object_ref_cleanup /home/orivej/src/subversion/subversion/libsvn_subr/object_pool.c:148:45
>      #1 0x9b7dd0 in run_cleanups /home/orivej/src/apr/memory/unix/apr_pools.c:2298:9
>      #2 0x9b66e4 in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:783:5
>      #3 0x5bbb65 in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1368:3
>      #4 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
>      #5 0x4edac8 in _start (/home/orivej/bin/svnserve+0x4edac8)
>
> 0x6250000234d0 is located 976 bytes inside of 8192-byte region [0x625000023100,0x625000025100)
> freed by thread T0 here:
>      #0 0x58bf9b in free /home/orivej/src/compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
>      #1 0x9b5991 in allocator_free /home/orivej/src/apr/memory/unix/apr_pools.c:407:9
>      #2 0x9b6a4f in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:825:5
>      #3 0x9b66cf in apr_pool_destroy /home/orivej/src/apr/memory/unix/apr_pools.c:780:9
>      #4 0x5bbb65 in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1368:3
>      #5 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
>
> previously allocated by thread T0 here:
>      #0 0x58c2ec in malloc /home/orivej/src/compiler-rt/lib/asan/asan_malloc_linux.cc:64:3
>      #1 0x9b516e in allocator_alloc /home/orivej/src/apr/memory/unix/apr_pools.c:330:17
>      #2 0x9b5d5a in apr_pool_create_ex /home/orivej/src/apr/memory/unix/apr_pools.c:860:17
>      #3 0x62986e in svn_pool_create_ex /home/orivej/src/subversion/subversion/libsvn_subr/pool.c:70:3
>      #4 0x77e654 in svn_object_pool__new_wrapper_pool /home/orivej/src/subversion/subversion/libsvn_subr/object_pool.c:356:10
>      #5 0x6f04ae in auto_parse /home/orivej/src/subversion/subversion/libsvn_repos/config_pool.c:215:14
>      #6 0x6eeab1 in svn_repos__config_pool_get /home/orivej/src/subversion/subversion/libsvn_repos/config_pool.c:523:17
>      #7 0x5be05c in sub_main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1002:7
>      #8 0x5bba6d in main /home/orivej/src/subversion/subversion/svnserve/svnserve.c:1347:9
>      #9 0x7f59a0f5882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
>

object-reference.patch (637 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap-use-after-free in object_ref_cleanup

Orivej Desh
In reply to this post by Daniel Shahaf-2
* Daniel Shahaf <[hidden email]> [2017-05-23]
>
> The report says that the use-after-free occured inside the cleanup handler.  It
> doesn't say where the accessed object was allocated or freed; to get that info,
> you'd have had to compile APR with pool debugging (--enable-pool-debug), then
> the second and third traceback would have pointed to the site of the malloc()
> and of the apr_pool_clear().

Thanks for your response!  I did not make much progress: I know that
--config-file option is necessary for reproduction, that actual config
file is irrelevant (may be empty), and here is the ASAN report from
subversion r1796426 and apr 1.6.0 with -DAPR_POOL_DEBUG=13 (I replaced
path prefixes with // for convenience).

> The pool cleanup is registered in add_object_ref().  Can you break in that
> function and confirm which of its two callsites is involved?

When I comment out `add_object_ref' in `lookup' the bug remains, but
when I delete it from `insert' svnserve exits successfully.

==456372==ERROR: AddressSanitizer: heap-use-after-free on address 0x60600000ea80 at pc 0x00000097c7c9 bp 0x7fff2ae32eb0 sp 0x7fff2ae32ea8
READ of size 8 at 0x60600000ea80 thread T0
    #0 0x97c7c8 in object_ref_cleanup //subversion/subversion/libsvn_subr/object_pool.c:144:45
    #1 0x640010 in run_cleanups //apr/memory/unix/apr_pools.c:2629:9
    #2 0x63bbb1 in pool_clear_debug //apr/memory/unix/apr_pools.c:1830:5
    #3 0x63c0a8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #4 0x63b1ed in apr_pool_destroy_debug //apr/memory/unix/apr_pools.c:1957:5
    #5 0x63fdae in apr_pool_destroy //apr/memory/unix/apr_pools.c:2887:5
    #6 0x5c0d15 in main //subversion/subversion/svnserve/svnserve.c:1432:3
    #7 0x7f22c12d682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #8 0x4f2c78 in _start (/home/orivej/bin/svnserve+0x4f2c78)

0x60600000ea80 is located 0 bytes inside of 56-byte region [0x60600000ea80,0x60600000eab8)
freed by thread T0 here:
    #0 0x59114b in free //compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
    #1 0x63bf6c in pool_clear_debug //apr/memory/unix/apr_pools.c:1853:13
    #2 0x63c0a8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #3 0x63bb9c in pool_clear_debug //apr/memory/unix/apr_pools.c:1827:9
    #4 0x63c0a8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #5 0x63b1ed in apr_pool_destroy_debug //apr/memory/unix/apr_pools.c:1957:5
    #6 0x63fdae in apr_pool_destroy //apr/memory/unix/apr_pools.c:2887:5
    #7 0x5c0d15 in main //subversion/subversion/svnserve/svnserve.c:1432:3
    #8 0x7f22c12d682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

previously allocated by thread T0 here:
    #0 0x59149c in malloc //compiler-rt/lib/asan/asan_malloc_linux.cc:64:3
    #1 0x63b31b in pool_alloc //apr/memory/unix/apr_pools.c:1740:16
    #2 0x63b229 in apr_palloc_debug //apr/memory/unix/apr_pools.c:1781:11
    #3 0x63fd26 in apr_palloc //apr/memory/unix/apr_pools.c:2863:12
    #4 0x97c214 in insert //subversion/subversion/libsvn_subr/object_pool.c:230:20
    #5 0x97c0d6 in svn_object_pool__insert //subversion/subversion/libsvn_subr/object_pool.c:321:3
    #6 0x8fc7b0 in find_config //subversion/subversion/libsvn_repos/config_pool.c:89:7
    #7 0x8fc2dd in svn_repos__config_pool_get //subversion/subversion/libsvn_repos/config_pool.c:126:33
    #8 0x5c340f in sub_main //subversion/subversion/svnserve/svnserve.c:1063:7
    #9 0x5c0c1d in main //subversion/subversion/svnserve/svnserve.c:1411:9
    #10 0x7f22c12d682f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap-use-after-free in object_ref_cleanup

Orivej Desh
* Stefan Fuhrmann <[hidden email]> [2017-05-28]
> The callstacks suggests that this is a pool cleanup race.
> Please try the attached patch and report the results.

Thanks!  With this patch subversion from trunk no longer crashes, and
subversion 1.9.5 does not crash with an empty config, and crashes
differently with a config containing `authz-db'.

(Please CC me in response.)

==909821==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000002540 at pc 0x0000009b6109 bp 0x7ffd4d8df8d0 sp 0x7ffd4d8df8c8
READ of size 8 at 0x606000002540 thread T0
    #0 0x9b6108 in object_ref_cleanup //subversion/subversion/libsvn_subr/object_pool.c:148:45
    #1 0x63d760 in run_cleanups //apr/memory/unix/apr_pools.c:2629:9
    #2 0x639245 in pool_clear_debug //apr/memory/unix/apr_pools.c:1820:5
    #3 0x6397f8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #4 0x6392ec in pool_clear_debug //apr/memory/unix/apr_pools.c:1827:9
    #5 0x6397f8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #6 0x63893d in apr_pool_destroy_debug //apr/memory/unix/apr_pools.c:1957:5
    #7 0x63d4fe in apr_pool_destroy //apr/memory/unix/apr_pools.c:2887:5
    #8 0x5bfba5 in main //subversion/subversion/svnserve/svnserve.c:1368:3
    #9 0x7ff6e517082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #10 0x4f1b08 in _start (/home/orivej/bin/svnserve1+0x4f1b08)

0x606000002540 is located 0 bytes inside of 56-byte region [0x606000002540,0x606000002578)
freed by thread T0 here:
    #0 0x58ffdb in free //compiler-rt/lib/asan/asan_malloc_linux.cc:47:3
    #1 0x6396bc in pool_clear_debug //apr/memory/unix/apr_pools.c:1853:13
    #2 0x6397f8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #3 0x6392ec in pool_clear_debug //apr/memory/unix/apr_pools.c:1827:9
    #4 0x6397f8 in pool_destroy_debug //apr/memory/unix/apr_pools.c:1915:5
    #5 0x63893d in apr_pool_destroy_debug //apr/memory/unix/apr_pools.c:1957:5
    #6 0x63d4fe in apr_pool_destroy //apr/memory/unix/apr_pools.c:2887:5
    #7 0x5bfba5 in main //subversion/subversion/svnserve/svnserve.c:1368:3
    #8 0x7ff6e517082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

previously allocated by thread T0 here:
    #0 0x59032c in malloc //compiler-rt/lib/asan/asan_malloc_linux.cc:64:3
    #1 0x638a6b in pool_alloc //apr/memory/unix/apr_pools.c:1740:16
    #2 0x638979 in apr_palloc_debug //apr/memory/unix/apr_pools.c:1781:11
    #3 0x63d476 in apr_palloc //apr/memory/unix/apr_pools.c:2863:12
    #4 0x9b5aeb in insert //subversion/subversion/libsvn_subr/object_pool.c:263:20
    #5 0x9b57a9 in svn_object_pool__insert //subversion/subversion/libsvn_subr/object_pool.c:393:3
    #6 0x946edb in auto_parse //subversion/subversion/libsvn_repos/config_pool.c:230:3
    #7 0x9451d1 in svn_repos__config_pool_get //subversion/subversion/libsvn_repos/config_pool.c:523:17
    #8 0x94ffde in svn_repos__authz_pool_get //subversion/subversion/libsvn_repos/authz_pool.c:159:3
    #9 0x5eac98 in load_authz_config //subversion/subversion/svnserve/serve.c:322:15
    #10 0x5e8c6d in find_repos //subversion/subversion/svnserve/serve.c:3551:3
    #11 0x5c6bd2 in construct_server_baton //subversion/subversion/svnserve/serve.c:3878:29
    #12 0x5c7e93 in serve //subversion/subversion/svnserve/serve.c:4076:3
    #13 0x5c254b in sub_main //subversion/subversion/svnserve/svnserve.c:1045:13
    #14 0x5bfaad in main //subversion/subversion/svnserve/svnserve.c:1347:9
    #15 0x7ff6e517082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: heap-use-after-free in object_ref_cleanup

Stefan Fuhrmann
On 29.05.2017 05:45, Orivej Desh wrote:
> * Stefan Fuhrmann <[hidden email]> [2017-05-28]
>> The callstacks suggests that this is a pool cleanup race.
>> Please try the attached patch and report the results.
>
> Thanks!  With this patch subversion from trunk no longer crashes, and
> subversion 1.9.5 does not crash with an empty config, and crashes
> differently with a config containing `authz-db'.

Hi Orivej,

I'm having a hard time reproducing that setup.
Could you give the command line(s) etc., maybe
including config files etc?

I probably wouldn't the actual repository, though.

-- Stefan^2.
Loading...