Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Radek Krotil

Hi all.

Our application generates lot of concurrent read requests to subversion using svn: protocol. When we tested the multithreaded mode of svnserve after upgrade to 1.9.3, we noticed strange 'deadlock-like' behavior: at some point all the requests are blocked in svnserve and wait there for a few minutes (3 to 15 minutes, no CPU activity), after which they continue to work. This is making our application significantly slower.

 

Operating system: Windows 10, CentOS 6.6, CentOS 7.2

The release and/or revision of Subversion: 1.9.3

The compiler and configuration options you built Subversion with: WANDisco binaries for CentOS, Apache Haus binaries for Windows

 

The workaround on Linux is to run svnserve without -T switch, i.e. not using multithreaded mode. For Windows, there is no workaround as svnserve only supports the multi-threaded mode.

 

Here is a sample of thread dump of svnserve.exe during the 'deadlock' obtained on Windows 10 using Process Explorer:

ntoskrnl.exe!KeSynchronizeExecution+0x3de6

ntoskrnl.exe!KeWaitForMutexObject+0xc7a

ntoskrnl.exe!KeWaitForMutexObject+0x709

ntoskrnl.exe!KeWaitForMutexObject+0x375

ntoskrnl.exe!IoThreadToProcess+0xff0

ntoskrnl.exe!KeRemoveQueueEx+0x16ba

ntoskrnl.exe!KeWaitForMutexObject+0xe8e

ntoskrnl.exe!KeWaitForMutexObject+0x709

ntoskrnl.exe!KeWaitForMutexObject+0x375

ntoskrnl.exe!NtWaitForSingleObject+0xf2

ntoskrnl.exe!setjmpex+0x3963

ntdll.dll!NtWaitForSingleObject+0x14

MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155

MSWSOCK.dll+0x1bf1

WS2_32.dll!WSAAccept+0xce

WS2_32.dll!accept+0x12

libapr-1.dll!apr_socket_accept+0x46

svnserve.exe+0xc11c

svnserve.exe+0xbae5

svnserve.exe+0xaf6c

svnserve.exe+0x13ab

KERNEL32.DLL!BaseThreadInitThunk+0x22

ntdll.dll!RtlUserThreadStart+0x34

 

The similar stack can be seen with other threads too.

 

Radek

Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Ivan Zhakov-2
On 14 April 2016 at 19:14, Radek Krotil <[hidden email]> wrote:

> Hi all.
>
> Our application generates lot of concurrent read requests to subversion
> using svn: protocol. When we tested the multithreaded mode of svnserve after
> upgrade to 1.9.3, we noticed strange 'deadlock-like' behavior: at some point
> all the requests are blocked in svnserve and wait there for a few minutes (3
> to 15 minutes, no CPU activity), after which they continue to work. This is
> making our application significantly slower.
>
> Operating system: Windows 10, CentOS 6.6, CentOS 7.2
>
> The release and/or revision of Subversion: 1.9.3
>
> The compiler and configuration options you built Subversion with: WANDisco
> binaries for CentOS, Apache Haus binaries for Windows
>
> The workaround on Linux is to run svnserve without -T switch, i.e. not using
> multithreaded mode. For Windows, there is no workaround as svnserve only
> supports the multi-threaded mode.
>
> Here is a sample of thread dump of svnserve.exe during the 'deadlock'
> obtained on Windows 10 using Process Explorer:
>
> ntoskrnl.exe!KeSynchronizeExecution+0x3de6
> ntoskrnl.exe!KeWaitForMutexObject+0xc7a
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!IoThreadToProcess+0xff0
> ntoskrnl.exe!KeRemoveQueueEx+0x16ba
> ntoskrnl.exe!KeWaitForMutexObject+0xe8e
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!NtWaitForSingleObject+0xf2
> ntoskrnl.exe!setjmpex+0x3963
> ntdll.dll!NtWaitForSingleObject+0x14
> MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
> MSWSOCK.dll+0x1bf1
> WS2_32.dll!WSAAccept+0xce
>
> WS2_32.dll!accept+0x12
> libapr-1.dll!apr_socket_accept+0x46
> svnserve.exe+0xc11c
> svnserve.exe+0xbae5
> svnserve.exe+0xaf6c
> svnserve.exe+0x13ab
> KERNEL32.DLL!BaseThreadInitThunk+0x22
> ntdll.dll!RtlUserThreadStart+0x34
>
> The similar stack can be seen with other threads too.
>
1. Do you have debug symbols for Subversion binaries you're using?
2. Other threads should have different stack trace, because AFAIK only
one thread calls accept().
3. Could you please create full memory dump of locked process and send
it to me including debug symbols?

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Stefan Fuhrmann-4
In reply to this post by Radek Krotil
On 14.04.2016 18:14, Radek Krotil wrote:
> Hi all.
>
> Our application generates lot of concurrent read requests to subversion using
> svn: protocol. When we tested the multithreaded mode of svnserve after upgrade
> to 1.9.3, we noticed strange 'deadlock-like' behavior: at some point all the
> requests are blocked in svnserve and wait there for a few minutes (3 to 15
> minutes, no CPU activity), after which they continue to work. This is making our
> application significantly slower.

In 1.9.x we switched to thread pools and limited their number
to give some control over the memory usage.  The limiting code
might interact with your requests, though.

The number of concurrent connections should not be a problem,
the threads only become active whenever there is an actual
request from the client side (like a c/o, log etc.).  What
kind of requests are being served?  How long to the connections
live (one new connection per request?).

You might try setting "--min-threads" and "--max-threads" to
large values such as 64 for the lower limit (default is 1) and
1024 for the upper limit (default is 64 on 32 bits and 256 with
64 bit versions of svnserve).  Monitor the memory usage, though!

Let us know if that helped.

-- Stefan^2.
Reply | Threaded
Open this post in threaded view
|

RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Radek Krotil
In reply to this post by Ivan Zhakov-2
Hi Ivan.

Apologies for my late reply. I've been caught in other tasks and events, but
recently I've encountered the behavior again and also similar pattern was
observed in environment of one of our enterprise customer.

All the dumps from svnserve, our application and also the memory dump can be
downloaded at https://drive.google.com/open?id=0B-yalijSzAIULWNuRGNGUW9KeWM.
I've been using the binaries from
http://de.apachehaus.com/downloads/mod_svn-1.9.3-ap24-x64.zip.

So to answer your questions..
1. Do you have debug symbols for Subversion binaries you're using?
A: I'm not an expert in C development, so I will need to consult with one of
my developers to understand, what you actually ask for. Maybe you can figure
the necessary details from the binaries themselves, or maybe you can give me
bit more hint in how to get the information you seek.

2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
A: Correct. All the stacktraces are captured in the zip file above. Majority
of the threads are in the following state, but our application does not
receive any response for a long time.

ntoskrnl.exe!KeSynchronizeExecution+0x3de6
ntoskrnl.exe!KeWaitForMutexObject+0xc7a
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!IoThreadToProcess+0xff0
ntoskrnl.exe!KeRemoveQueueEx+0x16ba
ntoskrnl.exe!KeWaitForMutexObject+0xe8e
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!NtWaitForSingleObject+0xf2
ntoskrnl.exe!setjmpex+0x3963
ntdll.dll!NtWaitForSingleObject+0x14
MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
MSWSOCK.dll!WSPStartup+0x3b1c
WS2_32.dll!WSARecv+0x17d
libapr-1.dll!apr_socket_recv+0x4a
svnserve.exe+0x15895
svnserve.exe+0x10a6d
svnserve.exe+0x1175b
svnserve.exe+0x1f8f
svnserve.exe+0xbeb9
libaprutil-1.dll!apr_thread_pool_top+0x797
MSVCR110.dll!beginthreadex+0x107
MSVCR110.dll!endthreadex+0x192
KERNEL32.DLL!BaseThreadInitThunk+0x22
ntdll.dll!RtlUserThreadStart+0x34

3. Could you please create full memory dump of locked process and send it to
me including debug symbols?
Dump is part of the zip file and for the debug symbol, I'll follow up on
that during the next week with my developers.

Thank you,
Radek Krotil

-----Original Message-----
From: Ivan Zhakov [mailto:[hidden email]]
Sent: Friday, April 15, 2016 9:46 AM
To: Radek Krotil
Cc: [hidden email]
Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

On 14 April 2016 at 19:14, Radek Krotil <[hidden email]> wrote:

> Hi all.
>
> Our application generates lot of concurrent read requests to
> subversion using svn: protocol. When we tested the multithreaded mode
> of svnserve after upgrade to 1.9.3, we noticed strange 'deadlock-like'
> behavior: at some point all the requests are blocked in svnserve and
> wait there for a few minutes (3 to 15 minutes, no CPU activity), after
> which they continue to work. This is making our application significantly
> slower.
>
> Operating system: Windows 10, CentOS 6.6, CentOS 7.2
>
> The release and/or revision of Subversion: 1.9.3
>
> The compiler and configuration options you built Subversion with:
> WANDisco binaries for CentOS, Apache Haus binaries for Windows
>
> The workaround on Linux is to run svnserve without -T switch, i.e. not
> using multithreaded mode. For Windows, there is no workaround as
> svnserve only supports the multi-threaded mode.
>
> Here is a sample of thread dump of svnserve.exe during the 'deadlock'
> obtained on Windows 10 using Process Explorer:
>
> ntoskrnl.exe!KeSynchronizeExecution+0x3de6
> ntoskrnl.exe!KeWaitForMutexObject+0xc7a
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!IoThreadToProcess+0xff0
> ntoskrnl.exe!KeRemoveQueueEx+0x16ba
> ntoskrnl.exe!KeWaitForMutexObject+0xe8e
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!NtWaitForSingleObject+0xf2
> ntoskrnl.exe!setjmpex+0x3963
> ntdll.dll!NtWaitForSingleObject+0x14
> MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
> MSWSOCK.dll+0x1bf1
> WS2_32.dll!WSAAccept+0xce
>
> WS2_32.dll!accept+0x12
> libapr-1.dll!apr_socket_accept+0x46
> svnserve.exe+0xc11c
> svnserve.exe+0xbae5
> svnserve.exe+0xaf6c
> svnserve.exe+0x13ab
> KERNEL32.DLL!BaseThreadInitThunk+0x22
> ntdll.dll!RtlUserThreadStart+0x34
>
> The similar stack can be seen with other threads too.
>
1. Do you have debug symbols for Subversion binaries you're using?
2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
3. Could you please create full memory dump of locked process and send it to
me including debug symbols?

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Radek Krotil
In reply to this post by Ivan Zhakov-2
Hi Ivan.

I managed to get the debug symbols for the Subversion binaries, we've been
using. It can be downloaded at
http://www.apachehaus.de/subversion-1.9.3-ap24-x64_pdb.zip.

Thank you,
Radek Krotil

-----Original Message-----
From: Radek Krotil [mailto:[hidden email]]
Sent: Saturday, May 21, 2016 10:46 AM
To: 'Ivan Zhakov'
Cc: '[hidden email]'
Subject: RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Hi Ivan.

Apologies for my late reply. I've been caught in other tasks and events, but
recently I've encountered the behavior again and also similar pattern was
observed in environment of one of our enterprise customer.

All the dumps from svnserve, our application and also the memory dump can be
downloaded at https://drive.google.com/open?id=0B-yalijSzAIULWNuRGNGUW9KeWM.
I've been using the binaries from
http://de.apachehaus.com/downloads/mod_svn-1.9.3-ap24-x64.zip.

So to answer your questions..
1. Do you have debug symbols for Subversion binaries you're using?
A: I'm not an expert in C development, so I will need to consult with one of
my developers to understand, what you actually ask for. Maybe you can figure
the necessary details from the binaries themselves, or maybe you can give me
bit more hint in how to get the information you seek.

2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
A: Correct. All the stacktraces are captured in the zip file above. Majority
of the threads are in the following state, but our application does not
receive any response for a long time.

ntoskrnl.exe!KeSynchronizeExecution+0x3de6
ntoskrnl.exe!KeWaitForMutexObject+0xc7a
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!IoThreadToProcess+0xff0
ntoskrnl.exe!KeRemoveQueueEx+0x16ba
ntoskrnl.exe!KeWaitForMutexObject+0xe8e
ntoskrnl.exe!KeWaitForMutexObject+0x709
ntoskrnl.exe!KeWaitForMutexObject+0x375
ntoskrnl.exe!NtWaitForSingleObject+0xf2
ntoskrnl.exe!setjmpex+0x3963
ntdll.dll!NtWaitForSingleObject+0x14
MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
MSWSOCK.dll!WSPStartup+0x3b1c
WS2_32.dll!WSARecv+0x17d
libapr-1.dll!apr_socket_recv+0x4a
svnserve.exe+0x15895
svnserve.exe+0x10a6d
svnserve.exe+0x1175b
svnserve.exe+0x1f8f
svnserve.exe+0xbeb9
libaprutil-1.dll!apr_thread_pool_top+0x797
MSVCR110.dll!beginthreadex+0x107
MSVCR110.dll!endthreadex+0x192
KERNEL32.DLL!BaseThreadInitThunk+0x22
ntdll.dll!RtlUserThreadStart+0x34

3. Could you please create full memory dump of locked process and send it to
me including debug symbols?
Dump is part of the zip file and for the debug symbol, I'll follow up on
that during the next week with my developers.

Thank you,
Radek Krotil

-----Original Message-----
From: Ivan Zhakov [mailto:[hidden email]]
Sent: Friday, April 15, 2016 9:46 AM
To: Radek Krotil
Cc: [hidden email]
Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

On 14 April 2016 at 19:14, Radek Krotil <[hidden email]> wrote:

> Hi all.
>
> Our application generates lot of concurrent read requests to
> subversion using svn: protocol. When we tested the multithreaded mode
> of svnserve after upgrade to 1.9.3, we noticed strange 'deadlock-like'
> behavior: at some point all the requests are blocked in svnserve and
> wait there for a few minutes (3 to 15 minutes, no CPU activity), after
> which they continue to work. This is making our application significantly
> slower.
>
> Operating system: Windows 10, CentOS 6.6, CentOS 7.2
>
> The release and/or revision of Subversion: 1.9.3
>
> The compiler and configuration options you built Subversion with:
> WANDisco binaries for CentOS, Apache Haus binaries for Windows
>
> The workaround on Linux is to run svnserve without -T switch, i.e. not
> using multithreaded mode. For Windows, there is no workaround as
> svnserve only supports the multi-threaded mode.
>
> Here is a sample of thread dump of svnserve.exe during the 'deadlock'
> obtained on Windows 10 using Process Explorer:
>
> ntoskrnl.exe!KeSynchronizeExecution+0x3de6
> ntoskrnl.exe!KeWaitForMutexObject+0xc7a
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!IoThreadToProcess+0xff0
> ntoskrnl.exe!KeRemoveQueueEx+0x16ba
> ntoskrnl.exe!KeWaitForMutexObject+0xe8e
> ntoskrnl.exe!KeWaitForMutexObject+0x709
> ntoskrnl.exe!KeWaitForMutexObject+0x375
> ntoskrnl.exe!NtWaitForSingleObject+0xf2
> ntoskrnl.exe!setjmpex+0x3963
> ntdll.dll!NtWaitForSingleObject+0x14
> MSWSOCK.dll!Tcpip6_WSHSetSocketInformation+0x155
> MSWSOCK.dll+0x1bf1
> WS2_32.dll!WSAAccept+0xce
>
> WS2_32.dll!accept+0x12
> libapr-1.dll!apr_socket_accept+0x46
> svnserve.exe+0xc11c
> svnserve.exe+0xbae5
> svnserve.exe+0xaf6c
> svnserve.exe+0x13ab
> KERNEL32.DLL!BaseThreadInitThunk+0x22
> ntdll.dll!RtlUserThreadStart+0x34
>
> The similar stack can be seen with other threads too.
>
1. Do you have debug symbols for Subversion binaries you're using?
2. Other threads should have different stack trace, because AFAIK only one
thread calls accept().
3. Could you please create full memory dump of locked process and send it to
me including debug symbols?

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Ivan Zhakov-2
On 31 May 2016 at 14:43, Radek Krotil <[hidden email]> wrote:
> Hi Ivan.
>
> I managed to get the debug symbols for the Subversion binaries, we've been
> using. It can be downloaded at
> http://www.apachehaus.de/subversion-1.9.3-ap24-x64_pdb.zip.
>
As far I see all 22 workers threads are waiting for data from network:
[[
ntdll.dll!NtWaitForSingleObject ()    Unknown
mswsock.dll!SockWaitForSingleObject ()    Unknown
mswsock.dll!WSPRecv ()    Unknown
ws2_32.dll!WSARecv ()    Unknown
libapr-1.dll!0000000057bff0aa()    Unknown
svnserve.exe!sock_read_cb(void * baton, char * buffer, unsigned
__int64 * len) Line 120    C
svnserve.exe!readbuf_fill(svn_ra_svn_conn_st * conn, apr_pool_t *
pool) Line 391    C
svnserve.exe!svn_ra_svn__read_tuple(svn_ra_svn_conn_st * conn,
apr_pool_t * pool, const char * fmt, ...) Line 1379    C
svnserve.exe!serve_interruptable(int * terminate_p, connection_t *
connection, int(*)(connection_t *) is_busy, apr_pool_t * pool) Line
4057    C
svnserve.exe!serve_thread(apr_thread_t * tid, void * data) Line 598    C
]]]

It also seems that you're using some third-party (Polarion) svn://
client for Subversion. Is it true?

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Radek Krotil
Polarion is our application and it uses SvnKit (http://svnkit.com/) as a
connector to SVN. We use SVNKit version 1.8.12.

Dump from our application also shows that the threads are waiting for data
from network. Could it be an issue in Windows itself? Anyway, we have seen
this behavior both in Windows and Linux environment and never seen it with
Subversion 1.8 and earlier, so we suspect it's a regression in Subversion.

Thanks,
Radek

-----Original Message-----
From: Ivan Zhakov [mailto:[hidden email]]
Sent: Tuesday, May 31, 2016 2:10 PM
To: Radek Krotil
Cc: [hidden email]
Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

On 31 May 2016 at 14:43, Radek Krotil <[hidden email]> wrote:
> Hi Ivan.
>
> I managed to get the debug symbols for the Subversion binaries, we've
> been using. It can be downloaded at
> http://www.apachehaus.de/subversion-1.9.3-ap24-x64_pdb.zip.
>
As far I see all 22 workers threads are waiting for data from network:
[[
ntdll.dll!NtWaitForSingleObject ()    Unknown
mswsock.dll!SockWaitForSingleObject ()    Unknown
mswsock.dll!WSPRecv ()    Unknown
ws2_32.dll!WSARecv ()    Unknown
libapr-1.dll!0000000057bff0aa()    Unknown
svnserve.exe!sock_read_cb(void * baton, char * buffer, unsigned
__int64 * len) Line 120    C
svnserve.exe!readbuf_fill(svn_ra_svn_conn_st * conn, apr_pool_t *
pool) Line 391    C
svnserve.exe!svn_ra_svn__read_tuple(svn_ra_svn_conn_st * conn,
apr_pool_t * pool, const char * fmt, ...) Line 1379    C
svnserve.exe!serve_interruptable(int * terminate_p, connection_t *
connection, int(*)(connection_t *) is_busy, apr_pool_t * pool) Line
4057    C
svnserve.exe!serve_thread(apr_thread_t * tid, void * data) Line 598    C
]]]

It also seems that you're using some third-party (Polarion) svn:// client
for Subversion. Is it true?

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Ivan Zhakov-2
On 31 May 2016 at 17:19, Radek Krotil <[hidden email]> wrote:
> Polarion is our application and it uses SvnKit (http://svnkit.com/) as a
> connector to SVN. We use SVNKit version 1.8.12.
>
> Dump from our application also shows that the threads are waiting for data
> from network. Could it be an issue in Windows itself? Anyway, we have seen
> this behavior both in Windows and Linux environment and never seen it with
> Subversion 1.8 and earlier, so we suspect it's a regression in Subversion.
>
I suggest to check network dump prior deadlock.

--
Ivan Zhakov
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Tomas Bilek
Hello Ivan,
I am continuing in topic, which Radek has started previously. I have created networkdump where you can see communication on localhost. Dump was started approx 5 s before our application started but is starts using SVN later. There are relative time stamps.   Also I have packed process monitor result but unfortunetely from time after svn starts hang up, because when I was running process monitor I was not able to reproduce that problem, so I turned it on after the system hangs.

To logs.
1) Network (network dump https://drive.google.com/file/d/0ByHPI8jVgwP8ZHJxbHZ5Z3BScVU/view?usp=sharing)
when i filter only tcp.port = 3690 I see that communication stopped and there is window from 245s until 637s
and this is snippet from our app log
2016-11-07 12:54:20,753 [ObjectIndex-refreshIndex-1 | u:p] INFO  com.polarion.platform.startup  -  -- -- 631 from 731 objects were refreshed (86%)
2016-11-07 13:00:54,030 [ObjectIndex-refreshIndex-2 | u:p] INFO  com.polarion.platform.startup  -  -- -- 731 from 731 objects were refreshed (100%)
   /////Indexing take some time from 10-20s so the difference 300-400s is corresponding.

2) Process Monitor  (dumps https://drive.google.com/file/d/0ByHPI8jVgwP8Qm0xa1ExNEQxZFU/view?usp=sharing)
you can filter at start for svnserve.exe. at start 07.11.2016 13:00:52,5687177   there is "disconnect sequence"

Do you think, that it is enough info to determine, where is the problem?
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Daniel Shahaf-5
Ping.  Has anyone looked into the network dump to try and understand
what's happening here?
Reply | Threaded
Open this post in threaded view
|

RE: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Stroleny, Jakub
Hi,

we're struggling with this issue again, but we're able to reproduce it now.
We have tested issue on svn 1.9.7 and 1.10.0 and it seems it is still valid.

I wrote some simple application to reproduce the issue. I've
described it in comment, see related issue https://issues.apache.org/jira/browse/SVN-4626
The application needs to be run probably multiple times to reproduce it correctly.

Basically, it seems that it is reproducible only when svnserve is running on Windows machine.
I did some tests on svnserve running on Linux machine and issue did not occur.

The client application is stuck on socket read from svnserve and there is no response
from server. We have network dumps and process monitor dumps, but
there isn't any response or network traffic when issue occurs.
The client application is blocked on the connection and  no progress is done
until we free some other connection to svnserve.

Can someone help us with analysis for this issue? I can provide more details if needed.
I had suspicious on svnserve.c:min_thread_count property, which is by default set to 1, but
we have 5+ concurrent connection, so that pool do not have enough workers to process
incoming requests. I have tried suggestion with "--min-threads" and "--max-threads" as
described also in previous replies, but without any change - it has same behavior as before.
Is there anything more what can we do to debug/trace this issue more?


Thank you,
Jakub Stroleny


-----Original Message-----
From: Daniel Shahaf [mailto:[hidden email]]
Sent: Tuesday, November 22, 2016 5:53 PM
To: [hidden email]
Subject: Re: Deadlock-like behaviour of svnserve in multi-threaded mode (-T)

Ping.  Has anyone looked into the network dump to try and understand what's happening here?

-----------------
Siemens Industry Software, s.r.o.
Praha 4, Doudlebská 1699/5, PSČ 140 00
IČ 256 51 897
Zapsaná v obchodním rejstříku vedeném Městským soudem v Praze, oddíl C, vložka 58222

Důležité upozornění: Tato zpráva má jen informativní charakter. Obsah této zprávy odesílatele nezavazuje a odesílatel nemá v úmyslu touto zprávou uzavřít smlouvu, přijmout nabídku, potvrdit uzavření smlouvy ani nezakládá předsmluvní odpovědnost jejího odesílatele, ledaže je odesílatelem ve zprávě uvedeno výslovně jinak. Obsah této zprávy (včetně příloh) je důvěrný. Pokud nejste zamýšleným adresátem této zprávy, zpřístupnění, kopírování, distribuce nebo jiné užití obsahu zprávy je zakázáno a v takovém případě, prosím, okamžitě informujte odesílatele a poté zprávu (vč. příloh) odstraňte z Vašeho systému.