Supporting multiple WC formats

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Supporting multiple WC formats

Branko Čibej
As you may have noticed, I started implementing support for multiple WC
formats on the better-pristines branch, as a prelude to adding support
for compressed pristines without forcing users to upgrade all their
working copies. There's an overview in the branch readme file, but the
basic idea is this:

  * the client will support more than one WC format
  * when working copies are created or upgraded, the default is always
    the latest supported format
  * 'svn checkout' and 'svn upgrade' accept a --compatible-version
    option that says which format of WC to create/upgrade to (the option
    name and semantics are the same as svnadmin's)
  * when update/switch create new externals working copies, they will
    use the parent's format.

The implementation of 'svn upgrade' is fairly straight-forward, but I've
still run into a bit of a conundrum; the question is this: since 'svn
upgrade' can theoretically be performed on an external WC, should we
support working copies with different formats in the top level and the
externals? I haven't checked if we currently do, though obviously, as
things stand now, upgrading only a subtree doesn't make much sense,
since the top-level format would still not be supported.

The other question that's popped up is more technical: how to
communicate the WC format down the library stack (for creating externals
working copies) during checkout/upgrade/switch. The only structure that
currently holds the format number is svn_wc__wcroot_t, which is strictly
private to libsvn_wc and not exposed anywhere else. The obvious solution
of recording the format in svn_wc_context_t seems like a bit of a
layering violation to me, but on the other hand it's the only bit of
context that's available to libsvn_client.

Any ideas will be most welcome.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

Johan Corveleyn-3
On Thu, Sep 14, 2017 at 6:01 AM, Branko Čibej <[hidden email]> wrote:

> As you may have noticed, I started implementing support for multiple WC
> formats on the better-pristines branch, as a prelude to adding support
> for compressed pristines without forcing users to upgrade all their
> working copies. There's an overview in the branch readme file, but the
> basic idea is this:
>
>   * the client will support more than one WC format
>   * when working copies are created or upgraded, the default is always
>     the latest supported format
>   * 'svn checkout' and 'svn upgrade' accept a --compatible-version
>     option that says which format of WC to create/upgrade to (the option
>     name and semantics are the same as svnadmin's)
>   * when update/switch create new externals working copies, they will
>     use the parent's format.

Woohoo! Those are huge steps forward IMHO (both "multiple wc formats"
and "compressed pristines"). I hope these features might still make it
into 1.10 :-).

I remember asking for supporting "older wc formats" before [1], and I
very much look forward to this extra flexibility. For example it makes
it much easier to organize the upgrade of an svn client on a big build
server with hundreds of working copies; or to get your users to
upgrade their clients without jumping through too many hoops at the
same time. Excellent!

(BTW: in that old mailthread I also asked about a "downgrading"
possibility (which used to be possible with a python script in /tools
[2] before 1.7) -- could be handy in case of accidental or premature
upgrades -- but I'm guessing that's out-of-scope for your current
work).

Also, it will be great to finally see compressed pristines become a
reality :-). That buildserver with hundreds of working copies,
totalling 200 GB's of diskspace, will be very glad I think :-).

> The implementation of 'svn upgrade' is fairly straight-forward, but I've
> still run into a bit of a conundrum; the question is this: since 'svn
> upgrade' can theoretically be performed on an external WC, should we
> support working copies with different formats in the top level and the
> externals? I haven't checked if we currently do, though obviously, as
> things stand now, upgrading only a subtree doesn't make much sense,
> since the top-level format would still not be supported.

My opinion: don't allow upgrading individual externals-sub-wc's. It's
all or nothing IMHO.

Wasn't there some vague plan to, one day, migrate away from "dir
externals are embedded wc's loosely integrated into the parent wc" to
"dir externals are fully managed by the root wc (no embedded wc with
sugar on top)"? In that case it might be important to view the
"embedded wc technique" as an implementation detail of directory
externals ...

(maybe Bert has some idea about the long-term vision if there ever was one?)

> The other question that's popped up is more technical: how to
> communicate the WC format down the library stack (for creating externals
> working copies) during checkout/upgrade/switch. The only structure that
> currently holds the format number is svn_wc__wcroot_t, which is strictly
> private to libsvn_wc and not exposed anywhere else. The obvious solution
> of recording the format in svn_wc_context_t seems like a bit of a
> layering violation to me, but on the other hand it's the only bit of
> context that's available to libsvn_client.

Sorry, can't comment on that ...

[1] https://svn.haxx.se/dev/archive-2012-04/0468.shtml
[2] http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/change-svn-wc-format.py

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

C. Michael Pilato
In reply to this post by Branko Čibej
On 09/14/2017 12:01 AM, Branko Čibej wrote:
> As you may have noticed, I started implementing support for multiple WC
> formats on the better-pristines branch, as a prelude to adding support
> for compressed pristines without forcing users to upgrade all their
> working copies.

Sweet!

> The other question that's popped up is more technical: how to
> communicate the WC format down the library stack (for creating externals
> working copies) during checkout/upgrade/switch. The only structure that
> currently holds the format number is svn_wc__wcroot_t, which is strictly
> private to libsvn_wc and not exposed anywhere else. The obvious solution
> of recording the format in svn_wc_context_t seems like a bit of a
> layering violation to me, but on the other hand it's the only bit of
> context that's available to libsvn_client.

(Small correction:  I _think_ you mean svn_wc__db_wcroot_t, not
svn_wc__wcroot_t.)

Maybe I'm missing something (like, oh, most of the technical context
required to answer this in detail, plus any degree of Subversion
development momentum), but that which is private can be made effectively
not so with public getter/setter functions, right?

Some other questions (partly out of curiousity, partly out of a hope
that by thinking through them your path becomes more clear):

Are you looking to expose mechanisms all the way up in the client binary
for declaring the version compatibility of a newly checked out working
copy (rather like the way 'svnadmin create' allows for the repository
flavor of this same idea)?

Will 'svn upgrade' someday be able to upgrade to anything other than the
most-current supported format?
Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

Branko Čibej
In reply to this post by Johan Corveleyn-3
On 14.09.2017 11:48, Johan Corveleyn wrote:

> On Thu, Sep 14, 2017 at 6:01 AM, Branko Čibej <[hidden email]> wrote:
>> As you may have noticed, I started implementing support for multiple WC
>> formats on the better-pristines branch, as a prelude to adding support
>> for compressed pristines without forcing users to upgrade all their
>> working copies. There's an overview in the branch readme file, but the
>> basic idea is this:
>>
>>   * the client will support more than one WC format
>>   * when working copies are created or upgraded, the default is always
>>     the latest supported format
>>   * 'svn checkout' and 'svn upgrade' accept a --compatible-version
>>     option that says which format of WC to create/upgrade to (the option
>>     name and semantics are the same as svnadmin's)
>>   * when update/switch create new externals working copies, they will
>>     use the parent's format.
> Woohoo! Those are huge steps forward IMHO (both "multiple wc formats"
> and "compressed pristines"). I hope these features might still make it
> into 1.10 :-).
>
> I remember asking for supporting "older wc formats" before [1], and I
> very much look forward to this extra flexibility. For example it makes
> it much easier to organize the upgrade of an svn client on a big build
> server with hundreds of working copies; or to get your users to
> upgrade their clients without jumping through too many hoops at the
> same time. Excellent!
>
> (BTW: in that old mailthread I also asked about a "downgrading"
> possibility (which used to be possible with a python script in /tools
> [2] before 1.7) -- could be handy in case of accidental or premature
> upgrades -- but I'm guessing that's out-of-scope for your current
> work).

Very much out of scope, though not impossible, given how I've decided to
go about supporting multiple formats. Downgrading for small changes
should be easy ... not so much for large ones, but it's really just a
matter of throwing enough code at the problem.

> Also, it will be great to finally see compressed pristines become a
> reality :-). That buildserver with hundreds of working copies,
> totalling 200 GB's of diskspace, will be very glad I think :-).

Some of the necessary code has been in place for years ... but the
addition of LZ4 compression finally pushed me towards doing something
about it. I have a nasty suspicion that using gzip compression would
have been disappointing in terms of performance. It's just a hunch, however.

>> The implementation of 'svn upgrade' is fairly straight-forward, but I've
>> still run into a bit of a conundrum; the question is this: since 'svn
>> upgrade' can theoretically be performed on an external WC, should we
>> support working copies with different formats in the top level and the
>> externals? I haven't checked if we currently do, though obviously, as
>> things stand now, upgrading only a subtree doesn't make much sense,
>> since the top-level format would still not be supported.
> My opinion: don't allow upgrading individual externals-sub-wc's. It's
> all or nothing IMHO.

That's the way I'm leaning, too. I do have to figure out if the code for
that is already in place, or whether I need some more wc-fu to invent it. :)

> Wasn't there some vague plan to, one day, migrate away from "dir
> externals are embedded wc's loosely integrated into the parent wc" to
> "dir externals are fully managed by the root wc (no embedded wc with
> sugar on top)"? In that case it might be important to view the
> "embedded wc technique" as an implementation detail of directory
> externals ...

There was such a vague plan. And, hehe, this is the kind of WC format
change that I would not want to write downgrade code for.

-- Brane

Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

Branko Čibej
In reply to this post by C. Michael Pilato
On 14.09.2017 14:46, C. Michael Pilato wrote:

> On 09/14/2017 12:01 AM, Branko Čibej wrote:
>> As you may have noticed, I started implementing support for multiple WC
>> formats on the better-pristines branch, as a prelude to adding support
>> for compressed pristines without forcing users to upgrade all their
>> working copies.
> Sweet!
>
>> The other question that's popped up is more technical: how to
>> communicate the WC format down the library stack (for creating externals
>> working copies) during checkout/upgrade/switch. The only structure that
>> currently holds the format number is svn_wc__wcroot_t, which is strictly
>> private to libsvn_wc and not exposed anywhere else. The obvious solution
>> of recording the format in svn_wc_context_t seems like a bit of a
>> layering violation to me, but on the other hand it's the only bit of
>> context that's available to libsvn_client.
> (Small correction:  I _think_ you mean svn_wc__db_wcroot_t, not
> svn_wc__wcroot_t.)

Yes.

> Maybe I'm missing something (like, oh, most of the technical context
> required to answer this in detail, plus any degree of Subversion
> development momentum), but that which is private can be made effectively
> not so with public getter/setter functions, right?

Indeed, however the wcroot struct is not part of any context or baton
that gets passed around; it's created on demand by functions within
libsvn_wc. It would take moderately drastic surgery to change that.

One possibly sane change would be to take the format _out_ of
svn_wc__db_wcroot_t and put it into svn_wc__db_t, with appropriate
getters and setters; that one's part of svn_wc_context_t. If I do that,
the answer to my first question would be set in concrete, more or less.


> Some other questions (partly out of curiousity, partly out of a hope
> that by thinking through them your path becomes more clear):
>
> Are you looking to expose mechanisms all the way up in the client binary
> for declaring the version compatibility of a newly checked out working
> copy (rather like the way 'svnadmin create' allows for the repository
> flavor of this same idea)?

Yes. That in fact already done; it was the easy part. :)

The multiple-format support feature would be a lot less usable if users
couldn't create new working copies in a not-latest format, for
compatibility with other tools.

> Will 'svn upgrade' someday be able to upgrade to anything other than the
> most-current supported format?

It can already do so on the branch. Of course there aren't that many
formats to choose from right now.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

Branko Čibej
In reply to this post by Branko Čibej
On 14.09.2017 14:47, Branko Čibej wrote:
> On 14.09.2017 11:48, Johan Corveleyn wrote:
>> My opinion: don't allow upgrading individual externals-sub-wc's. It's
>> all or nothing IMHO.
> That's the way I'm leaning, too. I do have to figure out if the code for
> that is already in place, or whether I need some more wc-fu to invent it. :)

$ svn st
X       foo

Performing status on external item at 'foo':
svn: E155021: This client is too old to work with the working copy at
'/Users/brane/src/svn/test/bar/foo' (format 32).
You need to get a newer Subversion client. For more details, see
  http://subversion.apache.org/faq.html#working-copy-format-change


The upgrade of the foo external working copy (using the client from the
better-pristines branch) worked without a hitch. So we currently do not
have any logic in the code to prevent partial upgrades or, in other
words, to prevent mixed-format working copies.

This needs fixing ... preferably on trunk.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Supporting multiple WC formats

Branko Čibej
On 14.09.2017 17:03, Branko Čibej wrote:

> On 14.09.2017 14:47, Branko Čibej wrote:
>> On 14.09.2017 11:48, Johan Corveleyn wrote:
>>> My opinion: don't allow upgrading individual externals-sub-wc's. It's
>>> all or nothing IMHO.
>> That's the way I'm leaning, too. I do have to figure out if the code for
>> that is already in place, or whether I need some more wc-fu to invent it. :)
> $ svn st
> X       foo
>
> Performing status on external item at 'foo':
> svn: E155021: This client is too old to work with the working copy at
> '/Users/brane/src/svn/test/bar/foo' (format 32).
> You need to get a newer Subversion client. For more details, see
>   http://subversion.apache.org/faq.html#working-copy-format-change
>
>
> The upgrade of the foo external working copy (using the client from the
> better-pristines branch) worked without a hitch. So we currently do not
> have any logic in the code to prevent partial upgrades or, in other
> words, to prevent mixed-format working copies.
>
> This needs fixing ... preferably on trunk.


However ... didn't we have a published use case for upgrading very large
working copies by upgrading subtrees first? Or was that just for
upgrades from entries-based to wc.db-based working copies?

-- Brane