svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

Thamm, Russell

UNCLASSIFIED

Hi,

 

svn-win32-1.88

 

I am not subscribed to the mailing list and would appreciate being cc:ed in any response.

 

I have a service that uses svn_client_status5 to determine the current state of a subversion working copy. Even though I have several thousand files in the working copy, this normally takes a few seconds.

 

However, on one particular PC, this takes several minutes. This particular PC doesn’t have access to the repository – the working copy is transferred via CD. However, my laptop doesn’t have access to the repository when it’s not connected to the network and it doesn’t exhibit this problem. Furthermore according to WireShark, the software is not trying to access the repository.

 

Using sysinternals procmon, it appears that every single file in the working copy is accessed on the problem PC, but only modified files are accessed normally.

In both cases, the directory tree is scanned (apparently in search of .svn directories) and many accesses are made in the .svn directory (especially ws.db).

 

Consequently procmon reports 416,177 file system events on the problem PC vs 30,068 on another normal PC.

 

My call is:

 

svn_error_t *err = svn_client_status5(NULL, context, path, &rev, svn_depth_infinity, true, false, false, true, false, NULL, statuslist_walk_cb, this, pool);

 

Any idea why svn_client_status5 is behaving so strangely?

 

Cheers

Russell

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

Johan Corveleyn-3
On Fri, Jul 28, 2017 at 6:20 AM, Thamm, Russell
<[hidden email]> wrote:

> UNCLASSIFIED
>
> Hi,
>
>
>
> svn-win32-1.88
>
>
>
> I am not subscribed to the mailing list and would appreciate being cc:ed in
> any response.
>
>
>
> I have a service that uses svn_client_status5 to determine the current state
> of a subversion working copy. Even though I have several thousand files in
> the working copy, this normally takes a few seconds.
>
>
>
> However, on one particular PC, this takes several minutes. This particular
> PC doesn’t have access to the repository – the working copy is transferred
> via CD. However, my laptop doesn’t have access to the repository when it’s
> not connected to the network and it doesn’t exhibit this problem.
> Furthermore according to WireShark, the software is not trying to access the
> repository.
>
>
>
> Using sysinternals procmon, it appears that every single file in the working
> copy is accessed on the problem PC, but only modified files are accessed
> normally.
>
> In both cases, the directory tree is scanned (apparently in search of .svn
> directories) and many accesses are made in the .svn directory (especially
> ws.db).
>
>
>
> Consequently procmon reports 416,177 file system events on the problem PC vs
> 30,068 on another normal PC.
>
>
>
> My call is:
>
>
>
> svn_error_t *err = svn_client_status5(NULL, context, path, &rev,
> svn_depth_infinity, true, false, false, true, false, NULL,
> statuslist_walk_cb, this, pool);
>
>
>
> Any idea why svn_client_status5 is behaving so strangely?

The reason is that the "lastModified" times in the svn metadata
(inside .svn/wc.db) are different from the "lastModified" times of the
files on disk. Probably because of the CD transfer (copying files over
different volumes often resets timestamps on files on the filesystem).

'status' has an optimization to assume the files are unchanged when
both filesize and lastModifiedTime are unchanged. If either filesize
or lastModifiedTime are different, svn has to read the entire file and
checksum it.

You can fix the situation by running 'svn cleanup' on the slow working
copy. This will correct the lastModifiedTime's in the svn metadata to
correspond to the on-disk state. It may also be possible to transfer
the files with precisely keeping the original lastModifiedTimes (maybe
with some copying options), then the working copy will still be
optimized after the transfer. But it depends, it's also possible that
the target filesystem cannot represent the same granularity of
lastModTimes as the original one.

--
Johan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

Philip Martin
In reply to this post by Thamm, Russell
"Thamm, Russell" <[hidden email]> writes:

> Any idea why svn_client_status5 is behaving so strangely?

I suspect you altered all the timestamps on the working files when you
copied the working copy, or perhaps the filesystems have different
timestamp resolution.  Running 'svn cleanup' will probably fix the
problem.

During write operations that modify files Subversion stores
timestamp/filesize for files that are known to be unmodified.  Status
uses this information to avoid doing a full-text comparison on files
where the timestamp/filesize match the stored values.  If you do
something to alter the timestamps of the working files then status will
have do a full-text comparison.

"Broken" timestamps get fixed by various write operations that detect
the unmodified file, in particular 'svn commit'.

--
Philip
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

Thamm, Russell
In reply to this post by Thamm, Russell
UNCLASSIFIED

Thanks very much.

-----Original Message-----
From: Johan Corveleyn [mailto:[hidden email]]
Sent: Friday, 28 July, 2017 7:13 p.m.
To: Thamm, Russell
Cc: [hidden email]
Subject: Re: svn_client_status5 veeeeeeery slow [SEC=UNCLASSIFIED]

On Fri, Jul 28, 2017 at 6:20 AM, Thamm, Russell <[hidden email]> wrote:

> UNCLASSIFIED
>
> Hi,
>
>
>
> svn-win32-1.88
>
>
>
> I am not subscribed to the mailing list and would appreciate being
> cc:ed in any response.
>
>
>
> I have a service that uses svn_client_status5 to determine the current
> state of a subversion working copy. Even though I have several
> thousand files in the working copy, this normally takes a few seconds.
>
>
>
> However, on one particular PC, this takes several minutes. This
> particular PC doesn’t have access to the repository – the working copy
> is transferred via CD. However, my laptop doesn’t have access to the
> repository when it’s not connected to the network and it doesn’t exhibit this problem.
> Furthermore according to WireShark, the software is not trying to
> access the repository.
>
>
>
> Using sysinternals procmon, it appears that every single file in the
> working copy is accessed on the problem PC, but only modified files
> are accessed normally.
>
> In both cases, the directory tree is scanned (apparently in search of
> .svn
> directories) and many accesses are made in the .svn directory
> (especially ws.db).
>
>
>
> Consequently procmon reports 416,177 file system events on the problem
> PC vs
> 30,068 on another normal PC.
>
>
>
> My call is:
>
>
>
> svn_error_t *err = svn_client_status5(NULL, context, path, &rev,
> svn_depth_infinity, true, false, false, true, false, NULL,
> statuslist_walk_cb, this, pool);
>
>
>
> Any idea why svn_client_status5 is behaving so strangely?

The reason is that the "lastModified" times in the svn metadata (inside .svn/wc.db) are different from the "lastModified" times of the files on disk. Probably because of the CD transfer (copying files over different volumes often resets timestamps on files on the filesystem).

'status' has an optimization to assume the files are unchanged when both filesize and lastModifiedTime are unchanged. If either filesize or lastModifiedTime are different, svn has to read the entire file and checksum it.

You can fix the situation by running 'svn cleanup' on the slow working copy. This will correct the lastModifiedTime's in the svn metadata to correspond to the on-disk state. It may also be possible to transfer the files with precisely keeping the original lastModifiedTimes (maybe with some copying options), then the working copy will still be optimized after the transfer. But it depends, it's also possible that the target filesystem cannot represent the same granularity of lastModTimes as the original one.

--
Johan

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.
Loading...