Moving CVS to SVN - multiple repositories with different permissions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
Planning to move from CVS to SVN, but I am not sure how to handle the
transfer....
1) We have used CVSNT since 2001, but we have not used any specific
CVSNT additions that I know of. THe CVSNT version is 2.5.03.2382

2) The server handles 6 different repositories organized as
subdirectories below E:\CVSREPOS

3) The permissions are different for these repositories and are
handled through Windows file system access permissions via AD groups

4) Some projects (especially in the PC repository) are defined in the
CVSROOT/modules file as a combination of files/folders from different
physical trees.

5) The repositories total 32787 files with a combined size of 7.9 GB.


Now I wonder how I should go about migrating to SVN? What I have read
indicates that the cvs2svn script does not use the information within
the CVSROOT admin directory of each repository, so I guess I have to
re-create something similar in SVN after the files have been
converted?

Does SVN have a concept like the modules file such that one can create
virtual modules for checkout where some files/folders are shared
between projects (common code)?

Can I use Windows Server 2016 as the server for the SVN repository(s)?


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Stefan Sperling
On Mon, Nov 13, 2017 at 12:16:34PM -0500, Bo Berglund wrote:

> Planning to move from CVS to SVN, but I am not sure how to handle the
> transfer....
> 1) We have used CVSNT since 2001, but we have not used any specific
> CVSNT additions that I know of. THe CVSNT version is 2.5.03.2382
>
> 2) The server handles 6 different repositories organized as
> subdirectories below E:\CVSREPOS
>
> 3) The permissions are different for these repositories and are
> handled through Windows file system access permissions via AD groups
>
> 4) Some projects (especially in the PC repository) are defined in the
> CVSROOT/modules file as a combination of files/folders from different
> physical trees.
>
> 5) The repositories total 32787 files with a combined size of 7.9 GB.
>
>
> Now I wonder how I should go about migrating to SVN? What I have read
> indicates that the cvs2svn script does not use the information within
> the CVSROOT admin directory of each repository, so I guess I have to
> re-create something similar in SVN after the files have been
> converted?

cvs2svn worked fine for me during a CVSNT->SVN migration some years ago.

Subversion stores access permissions in a special configuration file
on the server, so you'll have to re-create users, groups, and their
access rights in the SVN server's configuration:
http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html

Generally, I would recommend scripting the entire data migration process such
that it is repeatable. Then you can iterate the conversion process and allow
users to test the new system well before it goes live. Keep iterating until
you get a result that is acceptable for production use, then switch.

> Does SVN have a concept like the modules file such that one can create
> virtual modules for checkout where some files/folders are shared
> between projects (common code)?

You could use externals for this purpose:
http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html

These can be configured post-migration in SVN, and/or inside SVN as a
post-processing step during scripted data migration. Externals basically
configure a 'repository URL' to 'local directory' mapping which the
SVN client follows when it performs a 'checkout' or 'update'.

As noted in the SVN book chapter I linked above, externals should
always use 'relative' URLs rather than absolute URLs, where possible.
Relative URLs are resilient to potential future changes of most URL
components, such as the SVN server's hostname.

I would discourage use of externals to handle complicated dependency chains.
Externals are good enough if the dependency relationships are of a simple
variety. But if you find yourself using more than a handful of externals,
you're probably better off using a dependency management tool that works
above the version control system (e.g. in the build system).

> Can I use Windows Server 2016 as the server for the SVN repository(s)?

Yes. See http://subversion.apache.org/packages.html#windows for a
couple of options.
Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
On Mon, 13 Nov 2017 19:03:07 +0100, Stefan Sperling <[hidden email]>
wrote:

>
>Subversion stores access permissions in a special configuration file
>on the server, so you'll have to re-create users, groups, and their
>access rights in the SVN server's configuration:
>http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html
>

Thanks I will look into that more closely.
One question, though, that might be a terminology misunderstanding on
my part:
In CVS a repository is the container for a lot of different "modules",
which are top level directories below the root of the repository (or
else defined in CVSROOT/modules).

When reading your link above I found I became hesitant as to what SVN
means by "repository", it does not look like it is a 1-1 compared to
CVS...

I have as I said 6 CVS repositories served by CVSNT in order to
isolate files that are to be kept separate. For example the hardware
designers use repo HW for their firmware developments, Windows
application developers use repository PC and marketing uses Marketing
for their documents etc.
So in essence I have 6 different repositories in CVS language.
How does this map into SVN?
The SVN path-based authorization seems to be dealing with repositories
(modules?) within the repository (singular), all below the SVN root...

Did I misunderstand the way SVN works?


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Stefan Sperling
On Mon, Nov 13, 2017 at 02:11:08PM -0500, Bo Berglund wrote:

> On Mon, 13 Nov 2017 19:03:07 +0100, Stefan Sperling <[hidden email]>
> wrote:
>
> >
> >Subversion stores access permissions in a special configuration file
> >on the server, so you'll have to re-create users, groups, and their
> >access rights in the SVN server's configuration:
> >http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html
> >
>
> Thanks I will look into that more closely.
> One question, though, that might be a terminology misunderstanding on
> my part:
> In CVS a repository is the container for a lot of different "modules",
> which are top level directories below the root of the repository (or
> else defined in CVSROOT/modules).
>
> When reading your link above I found I became hesitant as to what SVN
> means by "repository", it does not look like it is a 1-1 compared to
> CVS...
>
> I have as I said 6 CVS repositories served by CVSNT in order to
> isolate files that are to be kept separate. For example the hardware
> designers use repo HW for their firmware developments, Windows
> application developers use repository PC and marketing uses Marketing
> for their documents etc.
> So in essence I have 6 different repositories in CVS language.
> How does this map into SVN?
> The SVN path-based authorization seems to be dealing with repositories
> (modules?) within the repository (singular), all below the SVN root...
>
> Did I misunderstand the way SVN works?

SVN is more flexible than CVS in this regard. Because directories, rather
than just files, are versioned, subdirectories of a repository can act as
a home for an entire project and thus become a 'module' in CVS terms.
See for instance the Apache Software Foundation's SVN repository, where
each top-level directory corresponds to an apache.org project:
https://svn.apache.org/viewvc

Such a designation of a 'project directory' or 'module' is simply a
convention put in place by users. SVN by itself won't treat any directories
special in any way (this goes further; for instance, project branches and
tags are also modeled as directories).

So you could migrate all 6 CVS repositories into one SVN repository with
one top-level directory per project, or you could create 6 distinct SVN
repositories. cvs2svn supports either approach.
See http://cvs2svn.tigris.org/faq.html#oneatatime

If your 6 projects could ever be expected to merge code from one another,
it makes sense to put them all in a single SVN repository because merge
tracking occurs on a per-repository basis (because SVN is not a distributed
version control system). Also, if you wish to use 'file externals' between
projects, they will have to be in the same repository (this is an unfortunate
implementation quirk).

On the other hand, there is an administrative concern which does not concern
end users: Backup and restore should occur on a per-repository basis.
So if you split projects across repositories there will be less overall
impact in case one of them has to be restored from backup for some unfortunate
reason (such events are very rare, but should not be ruled out in a risk
assessment). But if your CVS repositories aren't very large yet, then this
should not be a concern.
Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
On Mon, 13 Nov 2017 23:10:45 +0100, Stefan Sperling <[hidden email]>
wrote:
>If your 6 projects could ever be expected to merge code from one another,
>it makes sense to put them all in a single SVN repository because merge
>tracking occurs on a per-repository basis (because SVN is not a distributed
>version control system). Also, if you wish to use 'file externals' between
>projects, they will have to be in the same repository (this is an unfortunate
>implementation quirk).
>

OK, I will probably just make one SVN repository and make the current
repositories top level directories in that repo.
Then I can make the path-dependent permissions as described earlier.

But now I have to deal with 6 different CVSROOT directories too...
Is cvs2svn using the CVSROOT directory during conversion or can they
be removed from the source so as not confusing the operation?
Or do I need to use the cvsnt cvs.exe during conversion and does that
in such a case need these admin files in CVSROOT?

I will make a copy of the active repo for use during conversion of
course so I can take out the CVSROOT directories if they are not used.


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
On Mon, 13 Nov 2017 18:54:51 -0500, Bo Berglund
<[hidden email]> wrote:

>But now I have to deal with 6 different CVSROOT directories too...
>Is cvs2svn using the CVSROOT directory during conversion or can they
>be removed from the source so as not confusing the operation?
>Or do I need to use the cvsnt cvs.exe during conversion and does that
>in such a case need these admin files in CVSROOT?

I found a post on the net stating that CVSROOT directory *is*
necessary but that cvs2svn does not use any info therein during
conversion.
See:
http://tigris-scm.10930.n7.nabble.com/Is-CVSROOT-dir-mandatory-when-using-cvs2svn-td57397.html

However, I believe that I need to use the flag --use-cvs in order to
parse the CVSNT repository properly and this might mean that cvs
itself will use CVSROOT for something.
And then I have the problem of stuffing projects from the 6 different
repositories into SVN: Which CVSROOT should be used during conversion?
Or can all of them be there at the same time?

CVSNT specific stuff:
---------------------
I realized that I have used one CVSNT specific item and this is the
keyword -kbx, which means that some binary files have been marked as
binary with the exclusive editing flag. Typically this involves
Microsoft Office documents and PDF:s, which we did not want any two
persons simultaneously editing since they could not be merged.

Can cvs2svn deal with the -kbx keyword, for example by treating it as
-kb?
Or will these files be in any way corrupted by the conversion?


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Stefan Sperling
On Tue, Nov 14, 2017 at 08:16:21AM -0500, Bo Berglund wrote:

> On Mon, 13 Nov 2017 18:54:51 -0500, Bo Berglund
> <[hidden email]> wrote:
>
> >But now I have to deal with 6 different CVSROOT directories too...
> >Is cvs2svn using the CVSROOT directory during conversion or can they
> >be removed from the source so as not confusing the operation?
> >Or do I need to use the cvsnt cvs.exe during conversion and does that
> >in such a case need these admin files in CVSROOT?
>
> I found a post on the net stating that CVSROOT directory *is*
> necessary but that cvs2svn does not use any info therein during
> conversion.
> See:
> http://tigris-scm.10930.n7.nabble.com/Is-CVSROOT-dir-mandatory-when-using-cvs2svn-td57397.html
>
> However, I believe that I need to use the flag --use-cvs in order to
> parse the CVSNT repository properly and this might mean that cvs
> itself will use CVSROOT for something.
> And then I have the problem of stuffing projects from the 6 different
> repositories into SVN: Which CVSROOT should be used during conversion?
> Or can all of them be there at the same time?

In this case, an initial cvs2svn conversion which creates 6 distinct
SVN repositories might be easier.

Note that you could later on merge these repositories with SVN tooling:
http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.migrate

> CVSNT specific stuff:
> ---------------------
> I realized that I have used one CVSNT specific item and this is the
> keyword -kbx, which means that some binary files have been marked as
> binary with the exclusive editing flag. Typically this involves
> Microsoft Office documents and PDF:s, which we did not want any two
> persons simultaneously editing since they could not be merged.

The SVN feature corresponding to this is the svn:needs-lock property.
See http://svnbook.red-bean.com/nightly/en/svn.advanced.locking.html
 
> Can cvs2svn deal with the -kbx keyword, for example by treating it as
> -kb?
> Or will these files be in any way corrupted by the conversion?

Subversion does not need any special options for binaries.
Keywords in files have to be explicitly configured on a per-file basis:
http://svnbook.red-bean.com/nightly/en/svn.advanced.props.special.keywords.html
File content is never modified by default -- what goes in comes back out.
So I expect this problem will turn out to be irrelevant.
Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
Thanks!

Next issue is about Python:
I have used Python in conjunction with WinCVS before (never programmed
Python myself) and I did so by installing from the ActiveState Python
distribution.
Now I see that ActiveState only offers the x64 version of the
installer, whereas python.org has both.

Can cvs2svn use either one (32 or 64 bit)?
And is any of them to be preferred (ActiveState or Python.org, 32 or
64 bit)?

Furthermore the cvs2svn docs say:
Requirements:
....
- Python 2, version 2.4 or later. See http://www.python.org/. (cvs2svn
does not work with Python 3.x.)
- A compatible database library, usually gdbm, and the corresponding
Python bindings. Neither dumbdbm nor standard dbm is sufficient.

Is there any way to find out if the needed database library is
included or not? And if not how does one go about getting it?

I have to install all tools on a "clean" Windows 2016 server x64
machine for the conversion.

Oh! That brings up yet another point:
On Windows Server 2016 it seems like Microsoft has included their web
server (IIS), but I think that Apache is needed for SVN.
How can one deal with that?
Or is SVN a server all by itself?


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Nico Kadel-Garcia-2
On Tue, Nov 14, 2017 at 10:07 AM, Bo Berglund <[hidden email]> wrote:

> Thanks!
>
> Next issue is about Python:
> I have used Python in conjunction with WinCVS before (never programmed
> Python myself) and I did so by installing from the ActiveState Python
> distribution.
> Now I see that ActiveState only offers the x64 version of the
> installer, whereas python.org has both.
>
> Can cvs2svn use either one (32 or 64 bit)?
> And is any of them to be preferred (ActiveState or Python.org, 32 or
> 64 bit)?
>
> Furthermore the cvs2svn docs say:
> Requirements:
> ....
> - Python 2, version 2.4 or later. See http://www.python.org/. (cvs2svn
> does not work with Python 3.x.)
> - A compatible database library, usually gdbm, and the corresponding
> Python bindings. Neither dumbdbm nor standard dbm is sufficient.
>
> Is there any way to find out if the needed database library is
> included or not? And if not how does one go about getting it?

I think you're going to hurt yourself if you try to assemble cvs2svn
from scratch with individual components, installed separately and
built into a Windows environment. I *urge* you to save yourself a lot
of work and use 64-bit CygWin in a Windows environment, which can
contain up-to-date python, up-to-date command line Subversion tools
and a server, and a more reliably consistent scripting environment.

> I have to install all tools on a "clean" Windows 2016 server x64
> machine for the conversion.

See above.

> Oh! That brings up yet another point:
> On Windows Server 2016 it seems like Microsoft has included their web
> server (IIS), but I think that Apache is needed for SVN.
> How can one deal with that?
> Or is SVN a server all by itself?

You've a set of options, very well documented in the "Red Book" at
http://svnbook.red-bean.com/. Apache, or httpd as version 2.x is
called, with mod_svn, is a common approach and well supported. Apache
can run alongside IIS, or IIS ignored, as long as they do not run on
the same network ports. There is also "svnserve", the built-in server,
though it's not perhaps as flexible as httpd nor built into port 443
firewalls as commonly HTTPS is commonly supported. And there is also
"svn+ssh", which allows an SSH daemon with tuned credentials to allow
"svnserve" local access. I personally find svn+ssh more secure for
various reasons, especially because the Subversion command line tool
stores httpd credentials in plain text in a user's home directory, by
default, but folks on this list have previously expressed their
irritation at me for bringing that up.

I'd encourage you to use the simplest, most integrated tools you can
find for the server, and spend your development time on activating
time on reliable backup, and your user education time on getting users
accustomed to the new workflow.
Reply | Threaded
Open this post in threaded view
|

Re: Moving CVS to SVN - multiple repositories with different permissions

Bo Berglund
On Wed, 15 Nov 2017 08:55:27 -0500, Nico Kadel-Garcia
<[hidden email]> wrote:

>I think you're going to hurt yourself if you try to assemble cvs2svn
>from scratch with individual components, installed separately and
>built into a Windows environment. I *urge* you to save yourself a lot
>of work and use 64-bit CygWin in a Windows environment, which can
>contain up-to-date python, up-to-date command line Subversion tools
>and a server, and a more reliably consistent scripting environment.
>
>
>> Oh! That brings up yet another point:
>> On Windows Server 2016 it seems like Microsoft has included their web
>> server (IIS), but I think that Apache is needed for SVN.
>> How can one deal with that?
>> Or is SVN a server all by itself?
>
>You've a set of options, very well documented in the "Red Book" at
>http://svnbook.red-bean.com/. Apache, or httpd as version 2.x is
>called, with mod_svn, is a common approach and well supported. Apache
>can run alongside IIS, or IIS ignored, as long as they do not run on
>the same network ports. There is also "svnserve", the built-in server,
>though it's not perhaps as flexible as httpd nor built into port 443
>firewalls as commonly HTTPS is commonly supported. And there is also
>"svn+ssh", which allows an SSH daemon with tuned credentials to allow
>"svnserve" local access. I personally find svn+ssh more secure for
>various reasons, especially because the Subversion command line tool
>stores httpd credentials in plain text in a user's home directory, by
>default, but folks on this list have previously expressed their
>irritation at me for bringing that up.

I guess that VisualSVN server installation can deal with that?
It will be a server side issue anyway, not really affecting he CVS
data migration.
I googled svn port and found that it uses 3690, so that would not
interfere with http port 80, I guess.

>I'd encourage you to use the simplest, most integrated tools you can
>find for the server, and spend your development time on activating
>time on reliable backup, and your user education time on getting users
>accustomed to the new workflow.

Well, what I had in mind was this:
Server installation
-------------------
I would use VisualSVN since that seems to be a complete package (not
so many alternatives around for Windows really).
This would be done on the new Windows 2016 server.

CVS(NT) migration
-----------------
I thought I could do that on my Windows 7 X64 PC by using a copy of
the actual repository files and using cvs2svn as the tool.
I already have ActiveState Python 2.7.1 installed.
So I would create one dump file per CVS repository and then later
import those into the SVN server.

Now I have read up a lot on the svn help pages and found that I need
to use the config file option for cvs2svn in order to specify all the
different modules in the CVS repositories I need to convert.
It also seems like in order to include the authors of all the
revisions of the files I really need the option file so I can map the
CVS users to the svn users. And I also need the --use-cvs option.

But unfortunately that brought me to a full stop becuase when I looked
inside the config file example it turned out that the command line
options I had imagined would be listed really are not there, the
config file uses completely different options it looks like (or at
least different syntax for the same options)...



--
Bo Berglund
Developer in Sweden