Problems with using a symbolic link for .svn folder on TSVN

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with using a symbolic link for .svn folder on TSVN

Oscar Lee
Hi,

I was told to post my issue here from a TSVN dev. 

My company uses TortoiseSVN internally to keep our files updated. The .svn folder for the project I have is massive (250GB) and as such I had to move it off to an external HDD. I created a symbolic link to the new location so that TortoiseSVN 'should' still continue to work.

I managed to run a clean-up, but when I tried to revert a file, it gave me an error 'Failed to run the WC DB work queue associated with (file)" and "Can't move (tmp file) to ... (original file): The system cannot move the file to a different disk drive". 

I found that this error is caused by Windows not letting a file be renamed while it is being moved (https://docs.microsoft.com/en-us/previous-versions/ms837428(v=msdn.10)). Does anyone know a solution to this? Why is this an issue that only occurs with a symbolic link setup?

Regards,
Oscar
Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Paul Hammant-3
The .svn folder on the client side?

If you do a fresh checkout, how big is it then?
Reply | Threaded
Open this post in threaded view
|

RE: Problems with using a symbolic link for .svn folder on TSVN

Oscar Lee

(Sorry forgot to reply all)

 

Yes, it’s on my computer when I do a checkout if that’s what you mean. If I do a fresh checkout, the .svn is the same size (over 250GB).

 

From: Paul Hammant <[hidden email]>
Sent: Thursday, 10 January 2019 1:27 PM
To: Oscar Lee <[hidden email]>
Cc: Subversion <[hidden email]>
Subject: Re: Problems with using a symbolic link for .svn folder on TSVN

 

The .svn folder on the client side?

 

If you do a fresh checkout, how big is it then?

Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Daniel Shahaf-2
In reply to this post by Oscar Lee
Oscar Lee wrote on Wed, 09 Jan 2019 19:10 +0100:
> My company uses TortoiseSVN internally to keep our files updated. The .svn
> folder for the project I have is massive (250GB) and as such I had to move
> it off to an external HDD. I created a symbolic link to the new location so
> that TortoiseSVN 'should' still continue to work.

How large are the working copy files not under the .svn/ directory?

If they're substantially smaller than 250GB, you might be running into this:
https://subversion.apache.org/docs/release-notes/1.7#wc-pristines

> I managed to run a clean-up, but when I tried to revert a file, it gave me
> an error 'Failed to run the WC DB work queue associated with (file)" and
> "Can't move (tmp file) to ... (original file): The system cannot move the
> file to a different disk drive".
>
> I found that this error is caused by Windows not letting a file be renamed
> while it is being moved (
> https://docs.microsoft.com/en-us/previous-versions/ms837428(v=msdn.10)).
> Does anyone know a solution to this?

> Why is this an issue that only occurs with a symbolic link setup?

Subversion assumes that it is possible to atomically move a file from
the .svn directory to the working copy's checked out files.  That would
not possible when the .svn directory is on a different drive / filesystem.
Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Paul Hammant-3
In reply to this post by Oscar Lee
Alternative solution: do a Sparse Checkout - excising things in that tree that you don't really need.  Google's monorepo is north of 100TB of history with over nine million source files at HEAD revision. Individual Googler's day to day checkout tens of megabytes only: https://trunkbaseddevelopment.com/expanding-contracting-monorepos/

Though they made tooling for that expanding/contracting stuff, most normal companies could do the same to share within the team with a single bat/sh script for checkout. After that, 'svn up', 'svn commit' work as they always have. Indeed, if that bat file using the command line svn exe did it stuff, TortoiseSvn afte that will work as you expect for update/commit.

The sym-link for something that Subversion needs to refer to a lot is a bad idea.

- Paul
Reply | Threaded
Open this post in threaded view
|

RE: Problems with using a symbolic link for .svn folder on TSVN

Oscar Lee

I cannot do a sparse checkout as I require every part of the information to get the software running (a game).

 

From: Paul Hammant <[hidden email]>
Sent: Thursday, 10 January 2019 1:36 PM
To: Oscar Lee <[hidden email]>
Cc: Subversion <[hidden email]>
Subject: Re: Problems with using a symbolic link for .svn folder on TSVN

 

Alternative solution: do a Sparse Checkout - excising things in that tree that you don't really need.  Google's monorepo is north of 100TB of history with over nine million source files at HEAD revision. Individual Googler's day to day checkout tens of megabytes only: https://trunkbaseddevelopment.com/expanding-contracting-monorepos/

 

Though they made tooling for that expanding/contracting stuff, most normal companies could do the same to share within the team with a single bat/sh script for checkout. After that, 'svn up', 'svn commit' work as they always have. Indeed, if that bat file using the command line svn exe did it stuff, TortoiseSvn afte that will work as you expect for update/commit.

 

The sym-link for something that Subversion needs to refer to a lot is a bad idea.

 

- Paul

Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Branko Čibej
In reply to this post by Oscar Lee
On 09.01.2019 19:10, Oscar Lee wrote:
> Hi,
>
> I was told to post my issue here from a TSVN dev.
>
> My company uses TortoiseSVN internally to keep our files updated. The .svn
> folder for the project I have is massive (250GB) and as such I had to move
> it off to an external HDD. I created a symbolic link to the new location so
> that TortoiseSVN 'should' still continue to work.

Well, it "should" not, see below.

But if your .svn/ directory is much bigger than the rest of your working
copy, then 'svn cleanup --vacuum-pristines' will probably reduce its
size. In TSVN you'll have to select the checkbox "Vacuum pristine copies".

> I managed to run a clean-up, but when I tried to revert a file, it gave me
> an error 'Failed to run the WC DB work queue associated with (file)" and
> "Can't move (tmp file) to ... (original file): The system cannot move the
> file to a different disk drive".
>
> I found that this error is caused by Windows not letting a file be renamed
> while it is being moved (
> https://docs.microsoft.com/en-us/previous-versions/ms837428(v=msdn.10)).
> Does anyone know a solution to this? Why is this an issue that only occurs
> with a symbolic link setup?

This has nothing to do with symbolic links but with the fact that
Subversion, during normal operations, has to atomically rename (and
move) a file from somewhere in the .svn/ directory to its expected
location in the working copy. Windows can't do that if the source and
target of the rename are on different volumes, that's what the error
message is telling you. The explanation your link points to is
misleading, to put it mildly ... it's the move to a different volume
that fails, not the renaming of the file.

-- Brane

Reply | Threaded
Open this post in threaded view
|

RE: Problems with using a symbolic link for .svn folder on TSVN

Oscar Lee
In reply to this post by Daniel Shahaf-2
I see. Is it possible to change the implementation slightly as outlined by Boost SVN? (https://svn.boost.org/trac10/ticket/6809). I believe this is the same issue.

-----Original Message-----
From: Daniel Shahaf <[hidden email]>
Sent: Thursday, 10 January 2019 1:36 PM
To: Oscar Lee <[hidden email]>; [hidden email]
Subject: Re: Problems with using a symbolic link for .svn folder on TSVN

Oscar Lee wrote on Wed, 09 Jan 2019 19:10 +0100:
> My company uses TortoiseSVN internally to keep our files updated. The
> .svn folder for the project I have is massive (250GB) and as such I
> had to move it off to an external HDD. I created a symbolic link to
> the new location so that TortoiseSVN 'should' still continue to work.

How large are the working copy files not under the .svn/ directory?

If they're substantially smaller than 250GB, you might be running into this:
https://subversion.apache.org/docs/release-notes/1.7#wc-pristines

> I managed to run a clean-up, but when I tried to revert a file, it
> gave me an error 'Failed to run the WC DB work queue associated with
> (file)" and "Can't move (tmp file) to ... (original file): The system
> cannot move the file to a different disk drive".
>
> I found that this error is caused by Windows not letting a file be
> renamed while it is being moved (
> https://docs.microsoft.com/en-us/previous-versions/ms837428(v=msdn.10)).
> Does anyone know a solution to this?

> Why is this an issue that only occurs with a symbolic link setup?

Subversion assumes that it is possible to atomically move a file from the .svn directory to the working copy's checked out files.  That would not possible when the .svn directory is on a different drive / filesystem.

Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Branko Čibej
On 10.01.2019 13:40, [hidden email] wrote:
> I see. Is it possible to change the implementation slightly as outlined by Boost SVN? (https://svn.boost.org/trac10/ticket/6809). I believe this is the same issue.


It is the same issue and no it is not possible to change the
implementation "slightly" because the integrity of the working copy
relies on atomic renames. That is why Subversion doesn't use the system-
(or user-) specific temporary directory (defined by %TMP% and %TEMP% on
Windows) for such files but creates them in .svn/, to make it as likely
as possible that they'll be on the same volume as the rest of the
working copy.

The only thing we could do would be to not create temporary files in
.svn/ but in the same directory as the target file, but then we'd have
problems with potential name collisions and with such temporary files
cluttering the working copy tree after aborted operations.

It's possible, but it's not a "slight" change like adding another flag
to MoveFileEx.

-- Brane


> -----Original Message-----
> From: Daniel Shahaf <[hidden email]>
> Sent: Thursday, 10 January 2019 1:36 PM
> To: Oscar Lee <[hidden email]>; [hidden email]
> Subject: Re: Problems with using a symbolic link for .svn folder on TSVN
>
> Oscar Lee wrote on Wed, 09 Jan 2019 19:10 +0100:
>> My company uses TortoiseSVN internally to keep our files updated. The
>> .svn folder for the project I have is massive (250GB) and as such I
>> had to move it off to an external HDD. I created a symbolic link to
>> the new location so that TortoiseSVN 'should' still continue to work.
> How large are the working copy files not under the .svn/ directory?
>
> If they're substantially smaller than 250GB, you might be running into this:
> https://subversion.apache.org/docs/release-notes/1.7#wc-pristines
>
>> I managed to run a clean-up, but when I tried to revert a file, it
>> gave me an error 'Failed to run the WC DB work queue associated with
>> (file)" and "Can't move (tmp file) to ... (original file): The system
>> cannot move the file to a different disk drive".
>>
>> I found that this error is caused by Windows not letting a file be
>> renamed while it is being moved (
>> https://docs.microsoft.com/en-us/previous-versions/ms837428(v=msdn.10)).
>> Does anyone know a solution to this?
>> Why is this an issue that only occurs with a symbolic link setup?
> Subversion assumes that it is possible to atomically move a file from the .svn directory to the working copy's checked out files.  That would not possible when the .svn directory is on a different drive / filesystem.
>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Nathan Hartman
In reply to this post by Oscar Lee
On Thu, Jan 10, 2019 at 7:39 AM <[hidden email]> wrote:

I cannot do a sparse checkout as I require every part of the information to get the software running (a game)


At risk of saying the obvious, may I make a suggestion: Work on the (larger) external hard drive. In other words, have your entire working copy, .svn directory and all, on the external drive.

Reply | Threaded
Open this post in threaded view
|

Re: Problems with using a symbolic link for .svn folder on TSVN

Nathan Hartman
On Thu, Jan 10, 2019 at 10:48 AM Oscar Lee <[hidden email]> wrote:
The external drive is actually a company samba server that I connect over VPN. As such, it's super slow to read and even worse to write (maxes at 3MB/s). I don't care if the cleanup process or updating of the pristine files is slow, but I can't run the game off such a slow hard drive.

In that case, my suggestion is to invest <$200 in a USB3 5TB external hard drive and work off of that.
 
The answer to Life, The Universe and Everything is 42

How many roads must one take? :-p