"svn pget svn:externals -r <rev> . -R" dramatically slow

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

"svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
Hello Users,

Original issue: https://issues.apache.org/jira/browse/SVN-4681

Just discovered a really strange case where exactly titled command bring really slow response.

The repository contains more than 1000 revisions. The WC is in the middle, say at rev 1193 (the current), the show log shows 1191 at the last revision, the HEAD revision say 1300.
Trying to cd into WC directory and run the command. Needs about 1 minute  to wait it's response. The Process Hacker shows traffic with the server up to about 2MB.

If try to run command w/o -R flag - command returns immediately.

The content of the externals property without the -R flag is:
https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
^/solutions/project1/sdk proj2-sdk

The content of the externals property with the -R flag is:
https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
^/solutions/project1/sdk proj2-sdk
https://domain.ab/svn/proj2/trunk/proj2-gui -
https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -

I think the 2 last records draws the svn mad and it begin to crawl the server for something for about 1 minute.

I tries variations of the command. For example all these having the same result as above:
svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk" -R --non-interactive
svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive
svn pget svn:externals "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive

Dig a bit further and found this:
svn info https://domain.ab/svn/proj2/trunk/proj2-gui
svn: warning: W170000: URL 'https://domain.ab/svn/proj2/trunk/proj2-gui' non-existent in revision 1300
svn: E200009: Could not display info for all targets because some targets don't exist

Seems the 2 last records reference URL's what does not exist anymore in the HEAD.

These 2 records are left after an upgrade from a previous version of database, but why is the svn runs so slow about it? Anyway, i can't just cleanup the externals because they are from 3dparty repository in which i have no access.

--
Best regards,
 Andry                          mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Johan Corveleyn-3
On Tue, May 16, 2017 at 11:42 PM, Andry <[hidden email]> wrote:

> Hello Users,
>
> Original issue: https://issues.apache.org/jira/browse/SVN-4681
>
> Just discovered a really strange case where exactly titled command bring really slow response.
>
> The repository contains more than 1000 revisions. The WC is in the middle, say at rev 1193 (the current), the show log shows 1191 at the last revision, the HEAD revision say 1300.
> Trying to cd into WC directory and run the command. Needs about 1 minute  to wait it's response. The Process Hacker shows traffic with the server up to about 2MB.
>
> If try to run command w/o -R flag - command returns immediately.
>
> The content of the externals property without the -R flag is:
> https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
> ^/solutions/project1/sdk proj2-sdk
>
> The content of the externals property with the -R flag is:
> https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
> ^/solutions/project1/sdk proj2-sdk
> https://domain.ab/svn/proj2/trunk/proj2-gui -
> https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -
>
> I think the 2 last records draws the svn mad and it begin to crawl the server for something for about 1 minute.
>
> I tries variations of the command. For example all these having the same result as above:
> svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk" -R --non-interactive
> svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive
> svn pget svn:externals "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive
>
> Dig a bit further and found this:
> svn info https://domain.ab/svn/proj2/trunk/proj2-gui
> svn: warning: W170000: URL 'https://domain.ab/svn/proj2/trunk/proj2-gui' non-existent in revision 1300
> svn: E200009: Could not display info for all targets because some targets don't exist
>
> Seems the 2 last records reference URL's what does not exist anymore in the HEAD.
>
> These 2 records are left after an upgrade from a previous version of database, but why is the svn runs so slow about it? Anyway, i can't just cleanup the externals because they are from 3dparty repository in which i have no access.

Thanks for bringing the issue here to the list.

First a couple questions to get our context right: what version of svn
on the client and on the server? Is there a slow network between
client and server, or are they on the same LAN?

Just as background: svn:externals definitions can also optionally take
a peg revision (the @REV notation) [1], [2]. Maybe in the future you
can use the peg revision syntax to avoid this problem of the "external
source" disappearing at some point in history.

[1] http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html
[2] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html

--
Johan
Reply | Threaded
Open this post in threaded view
|

RE: "svn pget svn:externals -r <rev> . -R" dramatically slow

Bert Huijben-5


> -----Original Message-----
> From: Johan Corveleyn [mailto:[hidden email]]
> Sent: woensdag 17 mei 2017 19:57
> To: Andry <[hidden email]>
> Cc: [hidden email]
> Subject: Re: "svn pget svn:externals -r <rev> . -R" dramatically slow
>
> On Tue, May 16, 2017 at 11:42 PM, Andry <[hidden email]> wrote:
> > Hello Users,
> >
> > Original issue: https://issues.apache.org/jira/browse/SVN-4681
> >
> > Just discovered a really strange case where exactly titled command bring
> really slow response.
> >
> > The repository contains more than 1000 revisions. The WC is in the middle,
> say at rev 1193 (the current), the show log shows 1191 at the last revision,
> the HEAD revision say 1300.
> > Trying to cd into WC directory and run the command. Needs about 1
> minute  to wait it's response. The Process Hacker shows traffic with the
> server up to about 2MB.
> >
> > If try to run command w/o -R flag - command returns immediately.
> >
> > The content of the externals property without the -R flag is:
> > https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
> > ^/solutions/project1/sdk proj2-sdk
> >
> > The content of the externals property with the -R flag is:
> > https://domain.ab/svn/proj2/trunk - ^/../proj3/trunk/cmake cmake_proj3
> > ^/solutions/project1/sdk proj2-sdk
> > https://domain.ab/svn/proj2/trunk/proj2-gui -
> > https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -
> >
> > I think the 2 last records draws the svn mad and it begin to crawl the server
> for something for about 1 minute.
> >
> > I tries variations of the command. For example all these having the same
> result as above:
> > svn pget svn:externals -r "1193" "https://domain.ab/svn/proj2/trunk" -R --
> non-interactive
> > svn pget svn:externals -r "1193"
> "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive
> > svn pget svn:externals "https://domain.ab/svn/proj2/trunk@1193" -R --
> non-interactive
> >
> > Dig a bit further and found this:
> > svn info https://domain.ab/svn/proj2/trunk/proj2-gui
> > svn: warning: W170000: URL 'https://domain.ab/svn/proj2/trunk/proj2-gui'
> non-existent in revision 1300
> > svn: E200009: Could not display info for all targets because some targets
> don't exist
> >
> > Seems the 2 last records reference URL's what does not exist anymore in
> the HEAD.
> >
> > These 2 records are left after an upgrade from a previous version of
> database, but why is the svn runs so slow about it? Anyway, i can't just
> cleanup the externals because they are from 3dparty repository in which i
> have no access.
>
> Thanks for bringing the issue here to the list.

There is no optimized code path for retrieving properties recursively directly from the server.

The implementation of this specific command is like running 'svn ls' on every directory + fetching the properties on every file and every directory. (It is slightly more optimized than that, but not much)

        Bert


Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
Bert Huijben <[hidden email]> писал(а) в своём письме Thu, 18 May 2017  
13:09:07 +0300:

> There is no optimized code path for retrieving properties recursively  
> directly from the server.
>
> The implementation of this specific command is like running 'svn ls' on  
> every directory + fetching the properties on every file and every  
> directory. (It is slightly more optimized than that, but not much)
Why is that important when the links are empty?
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Andry
>> As you see the file1.txt is missed in second status output as
>> unversioned and so can be missed from add before a commit.
> It's not unversioned, it's in the "deleted" state. You can't have both,
> since you can revert the deletion.
If i'll revert it then i'll LOSE CHANGES because the svn will remove  
another file w/o warning in this case. Just because it had the same name.

However, I don't want to revert anything, i am talking about possibility  
of forget to add files because they are obscured by the deletion state in  
the status.

PS: Please send me a mail copy (CC) next time. You know is hard to put  
everything in the answer from a raw mail in the mailing list archive.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Stefan Sperling
In reply to this post by Andry
On Thu, May 18, 2017 at 02:17:26PM +0300, Andrey wrote:

> Bert Huijben <[hidden email]> писал(а) в своём письме Thu, 18 May 2017
> 13:09:07 +0300:
>
> > There is no optimized code path for retrieving properties recursively
> > directly from the server.
> >
> > The implementation of this specific command is like running 'svn ls' on
> > every directory + fetching the properties on every file and every
> > directory. (It is slightly more optimized than that, but not much)
> Why is that important when the links are empty?

External references are stored in properties called svn:externals.
I think Bert is referring to the retrieval of those properties.

There is no concept of a "link", as you called, it in Subversion.
So I am not sure what an "empty link" would mean.
If you wish to brush up your working knowledge of his part of
the system, please read:
http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html
Then we could have a conversion where both sides use the same terminology.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Andry
> First a couple questions to get our context right: what version of svn
> on the client and on the server?

> Is there a slow network between client and server, or are they on the  
> same LAN?
svn --version
```
svn, version 1.9.5 (r1770682)
    compiled Nov 26 2016, 14:22:31 on x86-microsoft-windows
...
The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network  
protocol.
   - with Cyrus SASL authentication
   - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
   - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using  
serf.
   - using serf 1.3.9 (compiled with 1.3.9)
   - handles 'http' scheme
   - handles 'https' scheme

The following authentication credential caches are available:

* Wincrypt cache in C:\Users\Andry\AppData\Roaming\Subversion
```

svnadmin --version
```
$ svnadmin --version
svnadmin, version 1.9.5 (r1770682)
    compiled Dec  1 2016, 14:48:33 on x86_64-unknown-linux-gnu
...
The following repository back-end (FS) modules are available:
* fs_fs : Module for working with a plain file (FSFS) repository.
* fs_x : Module for working with an experimental (FSX) repository.
* fs_base : Module for working with a Berkeley DB repository.
```

> Just as background: svn:externals definitions can also optionally take
> a peg revision (the @REV notation) [1], [2]. Maybe in the future you
> can use the peg revision syntax to avoid this problem of the "external
> source" disappearing at some point in history.
It's not mine repo to avoid such cases. Basically people use HEAD  
revisions to
take last changes by one external reference at once (as a solution of  
multiple projects).
I see nothing wrong with what.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Branko Čibej
In reply to this post by Andry
On 18.05.2017 13:27, Andrey wrote:

>>> As you see the file1.txt is missed in second status output as
>>> unversioned and so can be missed from add before a commit.
>> It's not unversioned, it's in the "deleted" state. You can't have both,
>> since you can revert the deletion.
> If i'll revert it then i'll LOSE CHANGES because the svn will remove
> another file w/o warning in this case. Just because it had the same name.
>
> However, I don't want to revert anything, i am talking about
> possibility of forget to add files because they are obscured by the
> deletion state in the status.

So what do you suggest we do instead?

There's a file named 'foo' that was deleted with 'svn rm' but not
committed, and also a file named 'foo' that you created on disk before
committing the deletion (or rename). What do you propose  'svn status'
should show?

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
Branko Čibej <[hidden email]> писал(а) в своём письме Thu, 18 May 2017  
14:41:17 +0300:

>> However, I don't want to revert anything, i am talking about
>> possibility of forget to add files because they are obscured by the
>> deletion state in the status.
>
> So what do you suggest we do instead?
>
> There's a file named 'foo' that was deleted with 'svn rm' but not
> committed, and also a file named 'foo' that you created on disk before
> committing the deletion (or rename). What do you propose  'svn status'
> should show?
may be add file content hash to represent 2 statuses of the same file  
paths? At least it will protect file from accidental remove and miss of  
add to commit?
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Andry
>> Why is that important when the links are empty?
> External references are stored in properties called svn:externals.
> I think Bert is referring to the retrieval of those properties.
I meant an external reference of cause. By the link i mean peace of string
in the output from the "svn pget svn:externals" command.

There is 2 records which are empty:
```
https://domain.ab/svn/proj2/trunk/proj2-gui -
https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -
```
Why just not to ignore them? Anyway they are don't have any mapping to a  
local directory.

PS: Please add me to a mail copy next time.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Andry
> First a couple questions to get our context right: what version of svn
> on the client and on the server?

> Is there a slow network between client and server, or are they on the  
> same LAN?
svn --version
```
svn, version 1.9.5 (r1770682)
      compiled Nov 26 2016, 14:22:31 on x86-microsoft-windows
...
The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network
protocol.
     - with Cyrus SASL authentication
     - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
     - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
     - using serf 1.3.9 (compiled with 1.3.9)
     - handles 'http' scheme
     - handles 'https' scheme

The following authentication credential caches are available:

* Wincrypt cache in C:\Users\Andry\AppData\Roaming\Subversion
```

svnadmin --version
```
$ svnadmin --version
svnadmin, version 1.9.5 (r1770682)
      compiled Dec  1 2016, 14:48:33 on x86_64-unknown-linux-gnu
...
The following repository back-end (FS) modules are available:
* fs_fs : Module for working with a plain file (FSFS) repository.
* fs_x : Module for working with an experimental (FSX) repository.
* fs_base : Module for working with a Berkeley DB repository.
```

> Just as background: svn:externals definitions can also optionally take
> a peg revision (the @REV notation) [1], [2]. Maybe in the future you
> can use the peg revision syntax to avoid this problem of the "external
> source" disappearing at some point in history.
It's not mine repo to avoid such cases. Basically people use HEAD
revisions to
take last changes by one external reference at once (as a solution of
multiple projects).
I see nothing wrong with what.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Andry
> First a couple questions to get our context right: what version of svn
> on the client and on the server?

> Is there a slow network between client and server, or are they on the  
> same LAN?
svn --version
```
svn, version 1.9.5 (r1770682)
      compiled Nov 26 2016, 14:22:31 on x86-microsoft-windows
...
The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network
protocol.
     - with Cyrus SASL authentication
     - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
     - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
     - using serf 1.3.9 (compiled with 1.3.9)
     - handles 'http' scheme
     - handles 'https' scheme

The following authentication credential caches are available:

* Wincrypt cache in C:\Users\Andry\AppData\Roaming\Subversion
```

svnadmin --version
```
$ svnadmin --version
svnadmin, version 1.9.5 (r1770682)
      compiled Dec  1 2016, 14:48:33 on x86_64-unknown-linux-gnu
...
The following repository back-end (FS) modules are available:
* fs_fs : Module for working with a plain file (FSFS) repository.
* fs_x : Module for working with an experimental (FSX) repository.
* fs_base : Module for working with a Berkeley DB repository.
```

> Just as background: svn:externals definitions can also optionally take
> a peg revision (the @REV notation) [1], [2]. Maybe in the future you
> can use the peg revision syntax to avoid this problem of the "external
> source" disappearing at some point in history.
It's not mine repo to avoid such cases. Basically people use HEAD
revisions to
take last changes by one external reference at once (as a solution of
multiple projects).
I see nothing wrong with what.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Stefan Sperling
In reply to this post by Andry
On Thu, May 18, 2017 at 02:58:51PM +0300, Andrey wrote:

> > > Why is that important when the links are empty?
> > External references are stored in properties called svn:externals.
> > I think Bert is referring to the retrieval of those properties.
> I meant an external reference of cause. By the link i mean peace of string
> in the output from the "svn pget svn:externals" command.
>
> There is 2 records which are empty:
> ```
> https://domain.ab/svn/proj2/trunk/proj2-gui -
> https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -
> ```
> Why just not to ignore them? Anyway they are don't have any mapping to a
> local directory.

My impression is that you misunderstand how the SVN system was designed.

'svn propget' is showing property contents and is oblivious to the
semantics of that content. There are different kinds of properties
(not just svn:externals) and 'propget' is a generic tool that
works for all of these properties.

If you need a tool which specifically interprets svn:externals
property content in some way, then you would need to write one.
Such as a wrapper script which does post-processing on the content.

BTW, using svn:externals to excessively link between source code files
and modules has downsides. svn:externals were *not* designed to be
a dependency management or linking system. Though admittedly many
people end up using it as such, and then sometimes get confused by
the inherent limitations of the user interface. For example, merging
between branches can become unnecessarily complicated when externals
are used excessively in a project. Simply put, I always recommend
that if there is another way to solve a problem then don't use
svn:externals to solve the problem.

svn:externals are just a convenience wrapper around functionality
provided by 'svn checkout'. If you keep this in mind, and look at
how Subversion's properties work (again, please refer to the
svnbook I linked in my previous mail), then you should see why
the propget command behaves as it does.

> PS: Please add me to a mail copy next time.

Your address has been (and is now) in the To: header of email I send to you.
Is that not enough?
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Johan Corveleyn-3
In reply to this post by Andry
On Thu, May 18, 2017 at 1:58 PM, Andrey <[hidden email]> wrote:

>>> Why is that important when the links are empty?
>>
>> External references are stored in properties called svn:externals.
>> I think Bert is referring to the retrieval of those properties.
>
> I meant an external reference of cause. By the link i mean peace of string
> in the output from the "svn pget svn:externals" command.
>
> There is 2 records which are empty:
> ```
> https://domain.ab/svn/proj2/trunk/proj2-gui -
> https://domain.ab/svn/proj2/trunk/proj2-gui/lib/Resource/Files -
> ```
> Why just not to ignore them? Anyway they are don't have any mapping to a
> local directory.

In general I think it's fine for a directory to have an empty
svn:externals property (have not tested it, but I guess it's fine). So
the property is there, but has no content. That's not the problem.
When you request "svn propget svn:externals -R $URL" the client and
server still have to do the work to retrieve that property from all
nodes in the entire subtree -- it doesn't matter whether it's empty or
not at this point (it doesn't even matter whether it's defined or not,
they still have to look it up).

To rule out any relation to "svn:externals" in itself, can you test if
you get the same "propget -R" performance if you retrieve another
property with exactly the same command?

For instance, try:
svn pget fooprop "https://domain.ab/svn/proj2/trunk@1193" -R --non-interactive

On my svn repository that command takes also a minute or 2 for a
reasonably large tree.

So even without any property defined anywhere, your "propget -R"
request still makes the client (and/or server) crawl the entire
subtree and requesting the property at every node, and this seems to
take a lot of time (like Bert said, "There is no optimized code path
for retrieving properties recursively directly from the server.")

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Stefan Sperling
On Thu, May 18, 2017 at 02:51:01PM +0200, Johan Corveleyn wrote:
> (like Bert said, "There is no optimized code path
> for retrieving properties recursively directly from the server.")

And this, we have to admit, can be a problem for some use cases.
It would be nice to have an optimized way of doing this,
and I expect we would be open to suggestions and patches.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Branko Čibej
In reply to this post by Andry
On 18.05.2017 13:51, Andrey wrote:

> Branko Čibej <[hidden email]> писал(а) в своём письме Thu, 18 May
> 2017 14:41:17 +0300:
>
>>> However, I don't want to revert anything, i am talking about
>>> possibility of forget to add files because they are obscured by the
>>> deletion state in the status.
>>
>> So what do you suggest we do instead?
>>
>> There's a file named 'foo' that was deleted with 'svn rm' but not
>> committed, and also a file named 'foo' that you created on disk before
>> committing the deletion (or rename). What do you propose  'svn status'
>> should show?
> may be add file content hash to represent 2 statuses of the same file
> paths? At least it will protect file from accidental remove and miss
> of add to commit?

There will be no accidental remove; when you commit, your new file will
remain unversioned in the working copy:

brane@zulu:~/test/wc$ svn mv foo bar
A         bar
D         foo
brane@zulu:~/test/wc$ echo foo > foo
brane@zulu:~/test/wc$ svn st
A  +    bar
        > moved from foo
D       foo
        > moved to bar
brane@zulu:~/test/wc$ svn ci -mm
Adding         bar
Deleting       foo
Committing transaction...
Committed revision 2.
brane@zulu:~/test/wc$ svn st
?       foo

So far, this is what you reported. Now see what happens when you
actually run 'svn add' to replace the deleted file:

brane@zulu:~/test/wc$ touch qux
brane@zulu:~/test/wc$ svn add qux
A         qux
brane@zulu:~/test/wc$ svn ci -mm
Adding         qux
Transmitting file data .done
Committing transaction...
Committed revision 3.
brane@zulu:~/test/wc$ svn mv qux baz
A         baz
D         qux
brane@zulu:~/test/wc$ echo qux > qux
brane@zulu:~/test/wc$ svn add qux
A         qux
brane@zulu:~/test/wc$ svn st
A  +    baz
        > moved from qux
?       foo
R       qux
        > moved to baz
brane@zulu:~/test/wc$ svn ci -mm
Adding         baz
Replacing      qux
Transmitting file data .done
Committing transaction...
Committed revision 4.
brane@zulu:~/test/wc$ svn st
?       foo


In the first case, I suppose we could display something like the following:

D  +    foo
        > moved to bar

or:

D  ?    foo
        > moved to bar


to indicate that the file was deleted, but still exists on disk.

The more interesting question is how, if at all, this would affect the
Subversion API.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Johan Corveleyn-3
Johan Corveleyn <[hidden email]> писал(а) в своём письме Thu, 18 May  
2017 15:51:01 +0300:

>> Why just not to ignore them? Anyway they are don't have any mapping to a
>> local directory.
>
> In general I think it's fine for a directory to have an empty
> svn:externals property (have not tested it, but I guess it's fine). So
> the property is there, but has no content. That's not the problem.
> When you request "svn propget svn:externals -R $URL" the client and
> server still have to do the work to retrieve that property from all
> nodes in the entire subtree -- it doesn't matter whether it's empty or
> not at this point (it doesn't even matter whether it's defined or not,
> they still have to look it up).
>
> To rule out any relation to "svn:externals" in itself, can you test if
> you get the same "propget -R" performance if you retrieve another
> property with exactly the same command?
>
> For instance, try:
> svn pget fooprop "https://domain.ab/svn/proj2/trunk@1193" -R  
> --non-interactive
Ok, i see your point. Yes, the svn:ignore works the same way and nothing  
to do with these 2 "external records".

> On my svn repository that command takes also a minute or 2 for a
> reasonably large tree.
>
> So even without any property defined anywhere, your "propget -R"
> request still makes the client (and/or server) crawl the entire
> subtree and requesting the property at every node, and this seems to
> take a lot of time (like Bert said, "There is no optimized code path
> for retrieving properties recursively directly from the server.")
Ok. May be a --depth option for the "svn co" command like "--depth  
directories"
to workaround this for script writers?

I am writing script to collect these externals and seems 1-2 minute of  
waiting one directory of about (i've opened it in the Repository Browser  
and took a look on the depth of directories there, it took 5 seconds) ~100  
subdirectories and ~400 files is not worth to wait so long. So i suggest  
some command to take all these
externals on the drive as is w/o files, for example, in to the TEMP  
directory to traverse it offline.

I don't know, but why is not? At least it could reduce server disk work or  
something.
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Branko Čibej
Branko Čibej <[hidden email]> писал(а) в своём письме Thu, 18 May 2017  
16:19:23 +0300:

>> may be add file content hash to represent 2 statuses of the same file
>> paths? At least it will protect file from accidental remove and miss
>> of add to commit?
>
> There will be no accidental remove; when you commit, your new file will
> remain unversioned in the working copy:
I meant delete revert. It silently replaces unversioned one.

> In the first case, I suppose we could display something like the  
> following:
>
> D  +    foo
>         > moved to bar
>
> or:
>
> D  ?    foo
>         > moved to bar
>
>
> to indicate that the file was deleted, but still exists on disk.
>
> The more interesting question is how, if at all, this would affect the
> Subversion API.
That would be good. Thx!
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Branko Čibej
On 18.05.2017 15:51, Andrey wrote:

> Branko Čibej <[hidden email]> писал(а) в своём письме Thu, 18 May
> 2017 16:19:23 +0300:
>
>>> may be add file content hash to represent 2 statuses of the same file
>>> paths? At least it will protect file from accidental remove and miss
>>> of add to commit?
>>
>> There will be no accidental remove; when you commit, your new file will
>> remain unversioned in the working copy:
> I meant delete revert. It silently replaces unversioned one.


Ah. We probably shouldn't be doing that; silently losing data is bad.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: "svn pget svn:externals -r <rev> . -R" dramatically slow

Andry
In reply to this post by Stefan Sperling
Stefan Sperling <[hidden email]> писал(а) в своём письме Thu, 18 May 2017  
15:55:36 +0300:

> And this, we have to admit, can be a problem for some use cases.
> It would be nice to have an optimized way of doing this,
> and I expect we would be open to suggestions and patches.
I tested it a bit deeper.
I checked out entire directory for the 1193 revision on the drive (took ~6  
seconds) and recall the command w/o "-r <rev>" (took ~1 second).
I don't think this is about optimization because (i checked it) the server  
is not so busy to hangs about 1 minute. Is it much like excessive search  
somethere?
12