Slow ISVNClient.getChangelists on Linux/NFS share

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow ISVNClient.getChangelists on Linux/NFS share

Thomas Singer (SyntEvo)
Hi,

When performing following steps on my old Linux test machine (with slow
hard disk):

- have a SVN working copy at /home/user/test
- sudo apt install nfs-kernel-server
- add following line to /etc/exports:
   /home/user/test *(rw,sync,no_root_squash)
- start the NFS server:
   sudo systemctl start nfs-kernel-server.service
- mount the NFS share:
   sudo mount localhost:/home/user/test /home/user/test.nfs

and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9 JavaHL
binaries), adding/removing a file is very slow. It boils down to the
call ISVNClient.getChangelists which takes ~8s on the NFS share
(/home/user/test.nfs). First, I thought, it would be caused by the
native-Java overhead calling the call-back ~11,000 times for my working
copy, but when using the working copy directly (/home/user/test), the
method just takes <1s though the ~11,000 times call-back invocations are
still there.

My working copy has no local modifications, no untracked or ignored
files, no changelists.

Is it expected that this method (ISVNClient.getChangelists) is so slow
on a NFS share even if there are no changelists?

--
Best regards,
Thomas Singer
=============
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Branko Čibej
On 18.10.2017 13:31, Thomas Singer wrote:

> Hi,
>
> When performing following steps on my old Linux test machine (with
> slow hard disk):
>
> - have a SVN working copy at /home/user/test
> - sudo apt install nfs-kernel-server
> - add following line to /etc/exports:
>   /home/user/test *(rw,sync,no_root_squash)
> - start the NFS server:
>   sudo systemctl start nfs-kernel-server.service
> - mount the NFS share:
>   sudo mount localhost:/home/user/test /home/user/test.nfs
>
> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
> JavaHL binaries), adding/removing a file is very slow. It boils down
> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
> (/home/user/test.nfs). First, I thought, it would be caused by the
> native-Java overhead calling the call-back ~11,000 times for my
> working copy, but when using the working copy directly
> (/home/user/test), the method just takes <1s though the ~11,000 times
> call-back invocations are still there.
>
> My working copy has no local modifications, no untracked or ignored
> files, no changelists.
>
> Is it expected that this method (ISVNClient.getChangelists) is so slow
> on a NFS share even if there are no changelists?

I don't know if it's "expected" but I bet that NFS is killing SQLite
performance.

https://www.mail-archive.com/sqlite-users@.../msg88989.html

I'm not sure about the reason but the most likely answer, apart from
slow data rate and latency when compared to a local filesystem (which,
in your case on loopback, should not be an issue), is that the OS can't
really use a cache for files on NFS since it has no way to know whether
or not it's valid. With a lot of random-access reads and writes, that
can be a HUGE slowdown, as you found.


Also this:
https://sqlite.org/faq.html#q5

In other words, Subversion working copies on NFS are, and have always
been, a bad idea; not only because of SQLite but also because
Subversion's code itself relies on atomic renames, which NFS does not
provide.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Thomas Singer (SyntEvo)
On 18.10.2017 19:56, Branko Čibej wrote:

> On 18.10.2017 13:31, Thomas Singer wrote:
>> Hi,
>>
>> When performing following steps on my old Linux test machine (with
>> slow hard disk):
>>
>> - have a SVN working copy at /home/user/test
>> - sudo apt install nfs-kernel-server
>> - add following line to /etc/exports:
>>    /home/user/test *(rw,sync,no_root_squash)
>> - start the NFS server:
>>    sudo systemctl start nfs-kernel-server.service
>> - mount the NFS share:
>>    sudo mount localhost:/home/user/test /home/user/test.nfs
>>
>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>> JavaHL binaries), adding/removing a file is very slow. It boils down
>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>> (/home/user/test.nfs). First, I thought, it would be caused by the
>> native-Java overhead calling the call-back ~11,000 times for my
>> working copy, but when using the working copy directly
>> (/home/user/test), the method just takes <1s though the ~11,000 times
>> call-back invocations are still there.
>>
>> My working copy has no local modifications, no untracked or ignored
>> files, no changelists.
>>
>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>> on a NFS share even if there are no changelists?
>
> I don't know if it's "expected" but I bet that NFS is killing SQLite
> performance.
>
> https://www.mail-archive.com/sqlite-users@.../msg88989.html
>
> I'm not sure about the reason but the most likely answer, apart from
> slow data rate and latency when compared to a local filesystem (which,
> in your case on loopback, should not be an issue), is that the OS can't
> really use a cache for files on NFS since it has no way to know whether
> or not it's valid. With a lot of random-access reads and writes, that
> can be a HUGE slowdown, as you found.
>
>
> Also this:
> https://sqlite.org/faq.html#q5
>
> In other words, Subversion working copies on NFS are, and have always
> been, a bad idea; not only because of SQLite but also because
> Subversion's code itself relies on atomic renames, which NFS does not
> provide.
>
> -- Brane

What SVN command (on command line) I should test to get a similar result
as from ISVNClient.getChangelists? I've tried "git status" and it just
needs <1s on the NFS working copy.

--
Best regards,
Thomas Singer
=============
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Branko Čibej
On 19.10.2017 08:30, Thomas Singer wrote:

> On 18.10.2017 19:56, Branko Čibej wrote:
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>    /home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>    sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>    sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>> https://www.mail-archive.com/sqlite-users@.../msg88989.html
>>
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
> What SVN command (on command line) I should test to get a similar
> result as from ISVNClient.getChangelists?

If "adding/removing a file" is any indication, then "svn add" or "svn
remove" should be comparable.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Thomas Singer (SyntEvo)
On 2017-10-19 9:05, Branko Čibej wrote:

> On 19.10.2017 08:30, Thomas Singer wrote:
>> On 18.10.2017 19:56, Branko Čibej wrote:
>>> On 18.10.2017 13:31, Thomas Singer wrote:
>>>> Hi,
>>>>
>>>> When performing following steps on my old Linux test machine (with
>>>> slow hard disk):
>>>>
>>>> - have a SVN working copy at /home/user/test
>>>> - sudo apt install nfs-kernel-server
>>>> - add following line to /etc/exports:
>>>>     /home/user/test *(rw,sync,no_root_squash)
>>>> - start the NFS server:
>>>>     sudo systemctl start nfs-kernel-server.service
>>>> - mount the NFS share:
>>>>     sudo mount localhost:/home/user/test /home/user/test.nfs
>>>>
>>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>>> native-Java overhead calling the call-back ~11,000 times for my
>>>> working copy, but when using the working copy directly
>>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>>> call-back invocations are still there.
>>>>
>>>> My working copy has no local modifications, no untracked or ignored
>>>> files, no changelists.
>>>>
>>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>>> on a NFS share even if there are no changelists?
>>>
>>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>>> performance.
>>>
>>> https://www.mail-archive.com/sqlite-users@.../msg88989.html
>>>
>>>
>>> I'm not sure about the reason but the most likely answer, apart from
>>> slow data rate and latency when compared to a local filesystem (which,
>>> in your case on loopback, should not be an issue), is that the OS can't
>>> really use a cache for files on NFS since it has no way to know whether
>>> or not it's valid. With a lot of random-access reads and writes, that
>>> can be a HUGE slowdown, as you found.
>>>
>>>
>>> Also this:
>>> https://sqlite.org/faq.html#q5
>>>
>>> In other words, Subversion working copies on NFS are, and have always
>>> been, a bad idea; not only because of SQLite but also because
>>> Subversion's code itself relies on atomic renames, which NFS does not
>>> provide.
>>>
>>> -- Brane
>>
>> What SVN command (on command line) I should test to get a similar
>> result as from ISVNClient.getChangelists?
>
> If "adding/removing a file" is any indication, then "svn add" or "svn
> remove" should be comparable.

Sorry, this was misleading. Adding/removing a file cause a refresh of
which ISVNClient.getChangelists takes the longest time. With SmartSVN
Foundation ISVNClient.getChangelists is not invoked and hence much
faster when refreshing after a command, e.g. add or remove.

Which is the SVN command line equivalent of ISVNClient.getChangelists?

--
Best regards,
Thomas Singer
=============
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Branko Čibej
On 19.10.2017 10:07, Thomas Singer wrote:

> On 2017-10-19 9:05, Branko Čibej wrote:
>> On 19.10.2017 08:30, Thomas Singer wrote:
>>> On 18.10.2017 19:56, Branko Čibej wrote:
>>>> On 18.10.2017 13:31, Thomas Singer wrote:
>>>>> Hi,
>>>>>
>>>>> When performing following steps on my old Linux test machine (with
>>>>> slow hard disk):
>>>>>
>>>>> - have a SVN working copy at /home/user/test
>>>>> - sudo apt install nfs-kernel-server
>>>>> - add following line to /etc/exports:
>>>>>     /home/user/test *(rw,sync,no_root_squash)
>>>>> - start the NFS server:
>>>>>     sudo systemctl start nfs-kernel-server.service
>>>>> - mount the NFS share:
>>>>>     sudo mount localhost:/home/user/test /home/user/test.nfs
>>>>>
>>>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS
>>>>> share
>>>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>>>> native-Java overhead calling the call-back ~11,000 times for my
>>>>> working copy, but when using the working copy directly
>>>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>>>> call-back invocations are still there.
>>>>>
>>>>> My working copy has no local modifications, no untracked or ignored
>>>>> files, no changelists.
>>>>>
>>>>> Is it expected that this method (ISVNClient.getChangelists) is so
>>>>> slow
>>>>> on a NFS share even if there are no changelists?
>>>>
>>>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>>>> performance.
>>>>
>>>> https://www.mail-archive.com/sqlite-users@.../msg88989.html
>>>>
>>>>
>>>>
>>>> I'm not sure about the reason but the most likely answer, apart from
>>>> slow data rate and latency when compared to a local filesystem (which,
>>>> in your case on loopback, should not be an issue), is that the OS
>>>> can't
>>>> really use a cache for files on NFS since it has no way to know
>>>> whether
>>>> or not it's valid. With a lot of random-access reads and writes, that
>>>> can be a HUGE slowdown, as you found.
>>>>
>>>>
>>>> Also this:
>>>> https://sqlite.org/faq.html#q5
>>>>
>>>> In other words, Subversion working copies on NFS are, and have always
>>>> been, a bad idea; not only because of SQLite but also because
>>>> Subversion's code itself relies on atomic renames, which NFS does not
>>>> provide.
>>>>
>>>> -- Brane
>>>
>>> What SVN command (on command line) I should test to get a similar
>>> result as from ISVNClient.getChangelists?
>>
>> If "adding/removing a file" is any indication, then "svn add" or "svn
>> remove" should be comparable.
>
> Sorry, this was misleading. Adding/removing a file cause a refresh of
> which ISVNClient.getChangelists takes the longest time. With SmartSVN
> Foundation ISVNClient.getChangelists is not invoked and hence much
> faster when refreshing after a command, e.g. add or remove.
>
> Which is the SVN command line equivalent of ISVNClient.getChangelists?

The only one I can find that actually uses the
svn_client_get_changelists() function is 'svn update --changelist', but
of course it also does an update, so the comparison is not entirely valid.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Johan Corveleyn-3
In reply to this post by Thomas Singer (SyntEvo)
On Thu, Oct 19, 2017 at 8:30 AM, Thomas Singer
<[hidden email]> wrote:

> On 18.10.2017 19:56, Branko Čibej wrote:
>>
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>>
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>    /home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>    sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>    sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>>
>> https://www.mail-archive.com/sqlite-users@.../msg88989.html
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
>
> What SVN command (on command line) I should test to get a similar result as
> from ISVNClient.getChangelists? I've tried "git status" and it just needs
> <1s on the NFS working copy.

I assume you meant "svn status", not "git status".

I have no idea what ISVNClient.getChangelists corresponds to compared
with the native 'svn' commandline. But one thing that might be
relevant: in the client-side configuration file 'config' there are two
properties that can have a big influence here:

(see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
[[[
### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn                (the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
]]]

So with the above options you can make all clients (that use that
config file) respect "exclusive-locking" (with the exclusive-locking
flag), or just a subset of specific client applications (with the
exclusive-locking-clients property).

Perhaps your commandline 'svn' invocation uses the standard
~/.subversion/config file, which has the exclusive-locking flag set,
and your SmartSVN application might use another one (it can choose
another "config dir").

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: Slow ISVNClient.getChangelists on Linux/NFS share

Thomas Singer (SyntEvo)
On 2017-10-19 10:44, Johan Corveleyn wrote:

> I have no idea what ISVNClient.getChangelists corresponds to compared
> with the native 'svn' commandline. But one thing that might be
> relevant: in the client-side configuration file 'config' there are two
> properties that can have a big influence here:
>
> (see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
> [[[
> ### Section for configuring working copies.
> [working-copy]
> ### Set to a list of the names of specific clients that should use
> ### exclusive SQLite locking of working copies.  This increases the
> ### performance of the client but prevents concurrent access by
> ### other clients.  Third-party clients may also support this
> ### option.
> ### Possible values:
> ###   svn                (the command line client)
> # exclusive-locking-clients =
> ### Set to true to enable exclusive SQLite locking of working
> ### copies by all clients using the 1.8 APIs.  Enabling this may
> ### cause some clients to fail to work properly. This does not have
> ### to be set for exclusive-locking-clients to work.
> # exclusive-locking = false
> ]]]
>
> So with the above options you can make all clients (that use that
> config file) respect "exclusive-locking" (with the exclusive-locking
> flag), or just a subset of specific client applications (with the
> exclusive-locking-clients property).
>
> Perhaps your commandline 'svn' invocation uses the standard
> ~/.subversion/config file, which has the exclusive-locking flag set,
> and your SmartSVN application might use another one (it can choose
> another "config dir").

Thank you for this suggestion. I've tried with

exclusive-locking = true

but it makes no difference. I don't know what to enter for

exclusive-locking-clients =

How does SVN detect the client name when using JavaHL?

--
Best regards,
Thomas Singer
=============
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog