[RFC] Shelving and Checkpointing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[RFC] Shelving and Checkpointing

Julian Foad-7
Dear Subversion Developers,

I am delighted to announce that I am working with Assembla to develop
shelving and checkpointing functionality in Subversion. These have
been on the wish list for many years, and are becoming ever more in
demand since the popularity of git is making more users appreciate the
benefits of local operations.

Based on our many discussions in the past, along with a fresh look at
how other VCS's implement these features, I have written down the
requirements and started working on a design in the document
Shelving-Checkpointing Dev [1]. I will follow up with an HTML email
containing a copy of today's version. I invite you to leave comments
directly in Google Docs, or send them by reply here, as you prefer.

I will be the project lead on this within Assembla. I will regularly
demonstrate small increments of functionality, which will also act as
calls for feedback. We want to keep the scope quite small to get it
done as quickly as possible.

Please read through and if you are able to contribute with any
suggestions or practical help, that would be wonderful. Thanks!

- Julian


References:
   [1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-7

Shelving and Checkpointing Dev.

Local History Operations for Subversion

The Problem

Shelving

Checkpointing

The Solution

Shelving

Checkpointing

Feature/Benefit Matrix; Comparison with Existing Options

Role Models

The Design

Shelving

Extensions / Not Supported Initially

Integrate Shelving with Changelists

Checkpointing

Checkpointing Option 1

Checkpointing Option 2

Checkpointing Option 3

Extensions / Not Supported Initially

Rebasing Checkpoints

Integrating Checkpoints with Shelving

GUI Considerations

Complete WC State Shelving

Existing Issues to Overcome


References:

  1. Shelving-Checkpointing Dev doc. (J Foad)

  2. Shelving-Checkpointing UI doc. (J Foad)

  3. issue SVN-3625 "Commit shelving"

  4. issue SVN-3626 "Commit checkpointing"

  5. Svn Wiki page "Design: SavePoints"

The Problem

This is the problem we are looking to solve. We will use this to determine the scope of design and implementation.

Shelving

While developing one change, you need to stop and work on an urgent fix or you want to commit a quick and easy change or try a different approach in the same code. You need to store the unfinished work and return to it later. Existing options: server-based branches, patch files, extra Working Copies.


Priorities: fast, offline.

Checkpointing

While developing a change, you reach a state which you want to save. The next thing you do might break it, or you want to treat this work as good and probably ready to publish as a first step and to be looking at your next work as diff against this first step.


In other words, you want successive local commits, and then be able to:


  • discard the entire work, or

  • roll back to an intermediate state, or

  • commit as a whole, or

  • commit all the separate steps.


Priorities: fast, offline.

The Solution

Shelving

One-click patch management on top of 'svn diff' and 'svn patch'.


  • similar to "git stash"

  • backward-compatible extension to WC format

    • (old clients still work, just not seeing the shelved changes)


Functions:


  • shelve/stash: save a WC diff as a patch and revert it from WC

  • unshelve/stash-pop: apply specified patch to WC and delete the patch

  • list shelved patches and the files that they affect

  • delete shelved patches

  • shelving supports exactly the same kinds of change as 'svn patch' and 'svn diff'


Compared to manual patch management:


  • one-click simplicity

    • don't need to choose a filename or remember where it is

  • still can't save & restore WC base state (in v1)

Checkpointing

Options:


  1. further patch management built on a series of shelved changes

  2. local commits tightly integrated

  3. checkpoints are commits in a local repository


Option 1 looks and feels like an extension -- clunky, new commands, existing features (such as diff) have minimal support for acting on checkpoints.


  • key benefit: roll back to a checkpoint

  • key drawback: limited support for viewing current work against last checkpoint

  • backward-compatible extension to WC format

    • (old clients still work, just not seeing the checkpoints)


Option 2 looks and feels like a Subversion version of DVCS local commits. A key differentiator is the WC base state is the most recent checkpoint instead of the original base revision. That affects, for example, which lines a Subversion-aware editor would highlight as changed.


  • key benefit: view the current work against the last checkpoint

  • key drawback: not very feasible to implement in current state of Subversion

  • incompatible change of WC format


Option 3 looks and feels like using native Subversion on a local branch, with the restriction that the connection between this branch and the original repository is weak. (For example, "svn log" and "blame" might see only the checkpoints, not the prior history in the original repository, unless/until we extend their functionality.)


  • key benefit: works exactly like normal svn commits in many ways

  • key drawback: a simple implementation will increase WC size by 50% and on making the first checkpoint in a series will take time to perform a full local commit of the WC base state

  • backward-compatible extension to WC format

    • (when checkpoints are in use, an old client would see only the checkpoints repository and know nothing about the original repository)


Feature/Benefit Matrix; Comparison with Existing Options

Shelving

(& Checkpoints)

server branch

patch files

extra WCs

Shelving (patches)

Chkpt 1

Chkpt 2

Chkpt 3

works offline

no

yes

yes

yes

yes

yes

yes

one-click simplicity

no

no

no

yes

yes

yes

yes

fast

no

yes

medium

yes

yes

yes

yes

low disk space requirement

yes

yes

no

yes

yes

yes

yes

incremental re-build

yes

yes

no

yes

yes

yes

yes

save & restore WC base state

no

no

yes

no

no

yes

yes

save & restore uncommittable state (conflicts, etc.)

no

no

yes

no

n/a

n/a

n/a

can discard the entire work

no

yes

yes

yes

yes

yes

yes

back-compatible WC format

n/a

n/a

n/a

yes

yes

no

yes


Checkpoints

(additional)

server branch

patch files

extra WCs

Checkpt 1 (patches)

Checkpt 2 (special)

Checkpt 3 (local repo)

roll back to any checkpoint

yes

yes

yes

yes

yes

yes

commit as a whole

yes

yes

yes

yes

yes

yes

commit all the separate steps

yes

awkward

awkward

yes

yes

yes

incremental view in WC

yes

no

no

no

yes

yes

automatic checkpoints to enable rolling back a conflicting update

no

no

no

no

can do

maybe


Note: "Automatic checkpoint" is listed as a separate feature, under the name "Destructive command rollback", in "Design: SavePoints" [5].

Role Models

With Shelving, we are catching up with a feature that most current VCSs already provide. We should make it easy for users to apply what they have learnt in those systems, and avoid needless differences. We should also support Subversion's unique characteristics such as directories being versioned objects.


SHELVING

git stashing

hg shelving

bzr shelving

p4 shelving

IntelliJ

NetBeans

general





well integrated

simple patch

stored where?

local repo

WC/.hg/

WC/.bzr/

the depot (shareable)

local (project dir by default)

~/.netbeans/

stored how?

special commit

git patch

special commit?

special commit (changelist)

svn+git patch

patch + MIME binary support

changelist integration

n/a



tight: a shelf is a p4 change

cl <==> shelf strong association

no

pop vs. apply

pop or keep

backup or keep

pop or keep

pop

backup or keep

pop or keep

access in other cmds

yes



diff,diff2,files,print

(no)

(no)

merge & conflicts

regular merge

temp. commit + merge + continue

regular merge





The Design

For UI design, see Shelving-Checkpointing UI doc. [2].

Shelving

Implementation in libsvn_client with 'svn' command-line UI.


Main functionality:


  • shelve/stash [name] [paths/changelist]: save a WC diff

    • if no name given, automatically generate a name

    • reverts the successfully shelved changes from the WC

  • unshelve/stash-pop: apply specified patch to WC

    • optionally removes the patch

    • can apply to any revision or branch

  • list/delete shelved patches

  • stores patches in '.svn' dir, e.g. in '.svn/patches/'

  • standard svn patch file format; paths relative to WC root

Extensions / Not Supported Initially

Kinds of change that can be shelved:

WC State or Change

starting point

fair usability

good usability

committable changes

file text, file delete/add, most properties

yes

yes

yes

mergeinfo changes

no

yes

yes

copies and moves

no

yes

yes

directories (mkdir/rmdir/...)

no

yes

yes

binary files & properties

no

yes

yes

uncommittable state

unresolved conflicts

no

no

yes

changelist assignment

no

no

yes

unversioned items (git stash -u/-a)

no

no

no

inconsistent WC states (missing/obstructed)

no

no

no

WC base state (rev, URL, switched, depth)

no

no

no


Other shelving extensions to consider:


  • remember the order; 'unshelve' takes newest patch by default

  • allow showing a shelf's content -- like 'svn log' options

  • keep a backup when unshelving

  • allow a description (log message) in shelves and changelists

    • show it when listing shelves/changelists

    • use it when committing from a changelist that has a log msg

  • handle conflicts better, using a proper merge rather than patching

Integrate Shelving with Changelists

I hate to add complexity without simplifying something at the same time. Considering the relationship between shelving and changelists:


  • a changelist is a named set of file-paths in the WC; each path may be locally modified

  • a shelved change is a named set of file-patches that is not currently applied to the WC


Integrate them like this:


  • shelved changes and changelists share the same namespace

  • shelving means making a separate patch for each changelist (and one for the not-in-a-changelist paths)

  • unshelving means converting shelved changes to changelists


Benefits:


  • clear and simple relationship with no overlap

  • changelists remain backwards-compatible


Main issues:


  • a changelist can include unmodified paths:

    • should a shelved patch also include unmodified paths?

  • a path can appear in multiple shelves, but only in one changelist:

    • what to do when unshelving if a path clashes?

  • any extensions should apply uniformly to both shelving and changelists


Role models:


Checkpointing

Checkpointing Option 1

Further patch management built on a series of shelved changes. The working copy largely does not see the checkpoints, it only see the current working state against the original base; new commands provide limited operations on checkpoints.


Take design ideas from Mercurial Queues?


Functionality:


  • all existing svn WC commands operate against original base revision

    • status, diff, revert, commit, etc.

  • some new commands operate against last (or any) checkpoint

    • status, diff, etc.

    • limited subset and not exactly equivalent functionality

  • new commands for new operations such as commit-all-separately and rollback

  • 'svn update' disallowed, as it implies rebasing the checkpoints

Checkpointing Option 2

Local commits tightly integrated into svn WC functionality & command syntax.


Functionality:


  • all svn WC commands operate against latest checkpoint by default

    • status, diff, revert, etc.

  • all svn WC commands can operate against any intermediate checkpoint

    • using extended revision-selector syntax

  • 'svn update' disallowed, as it implies rebasing the checkpoints


Main design tasks:


  • pseudo-revision syntax to address local commits, in all WC and higher APIs

  • WC storage for multiple versions

  • WC able to switch the 'current base version' to an arbitrary stored version

  • internal ability to:

    • diff & merge arbitrary versions (local as well as repository versions)

    • send merge output to any WC layer (base or working or checkpoint)

  • UI and high level implementation of the new checkpointing features


No feasible implementation plan for this option.

Checkpointing Option 3

Checkpoints are commits in a local repository embedded in the WC.


Functionality:


  • all svn WC commands operate against latest checkpoint by default

    • status, diff, revert, etc.

  • all svn WC commands can operate against any intermediate checkpoint

    • using standard svn repository-access syntax

  • all svn repository operations by default only see the checkpoints repository

    • e.g. 'svn update' moves only from one checkpoint to another


Design points:


  • each series of checkpoints is stored in a new local repository

  • committing a first checkpoint creates a new local repository, stores the WC base as r1 and the WC working layer as r2 in it, and points the WC at r2 in this repository, updated to r2.

  • convenient access to the original repository is limited to specific new functionality -- e.g. 'log' and 'blame' might visit the checkpoints first and then the original repository

Extensions / Not Supported Initially

  • rebasing checkpoints

Rebasing Checkpoints

It is desirable to be able to rebase a series of checkpoints, when pulling in new changes from the repository. This involves several steps of merging, once into each checkpoint, and the possibility of conflicts at each step.


Rebasing checkpoints is more important in trunk-based, less important in branch-based work flows.

Integrating Checkpoints with Shelving

How are shelving and checkpoints related conceptually?


In other words, what do we expect to happen if we start working, make some checkpoints, and then shelve "the work" to work on something else? Do we usually want to shelve the whole caboodle, all the work checkpointed... or just the latest (uncheckpointed) working state and start new work against the latest checkpoint?


  1. one series of checkpoints, plus a set of stashed working states

    1. each stashed working state is relative to a particular checkpoint

    2. each stashed working state is relative to the original base

  2. multiple shelves, each holding a series of checkpoints plus a working state

    1. all shelves are based on the same "original" base state


(A checkpoint is a committable change; a working state is a committable change and/or uncommittable WC state such as conflicts, unversioned files, etc.)


And is there also a degenerate option where we simply don't track what each shelved change was relative to, it's just a context-diff patch?


Needs design.

GUI Considerations

GUI priorities tend to be a little different from a CLI.

  • APIs that do the job simply and reliably

  • progress indication and cancel, for any long-running ops

    • hopefully we won't have any long-running ops

  • ability to undo

    • we don't have to implement undo, just provide enough hooks so caller can

Complete WC State Shelving

A key concern will be to have APIs that do the job simply and reliably. Such as: ability to shelve a WC state whatever state it's in (all the odd states like unresolved conflicts, etc.) and restore to exactly the same state.


We should have two complementary APIs:


  • a "shelve" API should visit the WC reporting absolutely everything about the state, in such a way that the caller can capture that state;

    • a "diff" API should report the subset of this that is committable;

  • an "unshelve" API should be able to recreate any such state, using input entirely provided by the caller

    • a "patch" API should accept the subset of this that is committable.


To what extent do such APIs exist?


We should write automatic round-trip testing for diff-&-patch and for status-&-modify.


For states that we can't shelve, a GUI wants to know in advance that this is the case (to grey-out the button), rather than the operation to error out part way through. So we want:


  • an API to efficiently tell whether the WC state can be shelved (or why not)

Existing Issues to Overcome

Bugs and deficiencies:

  • svn patch to handle mergeinfo: SVN-3747

  • svn diff/patch to handle binary files (is partly done -- in "svn diff --git" anyway)

  • svn diff/patch to handle copies and moves: SVN-1056

  • svn diff/patch to handle directories (mkdir, rm, cp, mv)



Julian Foad, Assembla, 2017


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Nathan Hartman
In reply to this post by Julian Foad-7
On Jul 10, 2017, at 8:59 AM, Julian Foad <[hidden email]> wrote:
>
> Dear Subversion Developers,
>
> I am delighted to announce that I am working with Assembla to develop
> shelving and checkpointing functionality in Subversion. These have
> been on the wish list for many years, and are becoming ever more in
> demand since the popularity of git is making more users appreciate the
> benefits of local operations.

Hi all,

I just wanted to chime in and say that I'm very happy to hear that these local features are planned for Subversion. After ten years of using Subversion, I can say that it has been a joy to use from day one, owing in large part to its simplicity and reliability.

It occurred to me recently that in the distant past Subversion's purpose was to be a better CVS, that that goal has been achieved long ago, and that perhaps it's time for a Subversion 2.0, which could take it to the next level. I don't know much about future plans, such as what FSX is intended to achieve, but it occurred to me that perhaps Subversion could gain the ability to do local commits, which could be an optional step before doing the real commit back to the repository. As I thought about that, I realized that it might not have to require too much new machinery to achieve that.

I had this idea: When you checkout a working copy, Subversion creates the hidden .svn directory which contains the pristine copies among other things. What if, instead of just a pristine copy, it actually created a private local repository. Revision 1 of this repository would be the pristine copy. The user need not know that anything has changed and can continue working as before, committing back to the server. But, if you type some other command instead of commit, or maybe prepend the word "local" or something, the commit would go into the local repository rather than the remote one. Now this makes it possible to do locally anything you could do remotely such as creating multiple local feature branches. It also means that when you do an update, there could be an implied local commit first, which allows you to undo an update which perhaps messed something up. Shelving, stashing, etc., all become local operations against this local souped up pristine copy replacement.

Since you're only checking out one revision just as before, as opposed to cloning a whole repository, this idea comes with some big advantages that you don't get with git or hg, such as that you need not clone an entire gigantic repository. This is one of the big reasons that those systems are not a good choice for housing lots of big files that change infrequently, as every user would have to clone potentially gigabytes of data they're not interested in. In other words a svn checkout still does what it always did before, except that you have the option to work locally if you want to.

I didn't mention any of this before because:

(1) there are obviously a lot of holes in this idea, such as, what happens to this private local repo when you do a svn switch? Or the pesky issue of how to add such a major new feature without inheriting a ton of obscure commands a la git?

And (2) although I am a developer and very fond of Subversion, I cannot take this upon myself because of other obligations now and in the foreseeable future. So I felt it would be rude to dump a crazy idea like this on everyone else. Julian's post prompted me to break my silence, not knowing if what I've suggested is remotely realistic.

I realize that this suggests a major backwards-incompatible change to the working copy format and indeed the entire client side. But this was a request for comment. :-) Hopefully someone finds my input valuable.

Kind regards,
Nathan

> References:
>   [1] Shelving-Checkpointing Dev doc. (J Foad)
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
Thanks for your suggestion, Nathan.

Nathan Hartman wrote:
> [...] What if, instead of just a pristine copy, it actually created a private local repository. Revision 1 of this repository would be the pristine copy. [...] if you type some other command instead of commit, or maybe prepend the word "local" or something, the commit would go into the local repository [...] Shelving, stashing, etc., all become local operations against this local souped up pristine copy replacement.

That is exactly what I was thinking about when I wrote "Option 3...
Checkpoints are commits in a local repository embedded in the WC"
towards the end of that document. My feeling is that both the amount of
effort required just to get that far, and the explosion of further
possibilities it would open up, make it too heavy-weight for my current
project, but I would definitely like to explore that possibility further.

- Julian

>> References:
>>    [1] Shelving-Checkpointing Dev doc. (J Foad)
>> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Daniel Shahaf-5
In reply to this post by Julian Foad-7
On Mon, Jul 10, 2017 at 02:03:41PM +0100, Julian Foad wrote:

> Checkpointing
>
> Options:
>
>
>    1.
>
>    further patch management built on a series of shelved changes
>    2.
>
>    local commits tightly integrated
>    3.
>
>    checkpoints are commits in a local repository

Can you explain these three options in more words?  AIUI #1 is
"syntactic sugar to manage a patch series", like quilt(1), but I'm not
sure I understand #2 and #3.

> SHELVING
>
> git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>
>

For git, 'stashing' can also be implemented by a temporary branch («git
checkout -b foo && git commit -amm && git checkout master»), which
changes some of the values of the table.  I assume the same is true for hg.

Unfortunately the text/plain rendering of that table is quite lossy.
The part I quoted are the column headings; the row headings are
"general" "stored where?" "stored how?" "changelist integration" "pop
vs. apply" "access in other cmds" "merge & conflicts".

Cheers,

Daniel

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
Daniel Shahaf wrote:

> Julian Foad wrote:
>> Checkpointing
>>
>> Options:
>>
>>     1. further patch management built on a series of shelved changes
>>     2. local commits tightly integrated
>>     3. checkpoints are commits in a local repository
>
> Can you explain these three options in more words?  AIUI #1 is
> "syntactic sugar to manage a patch series", like quilt(1), but I'm not
> sure I understand #2 and #3.

2. What I was thinking there is to rewrite as much of our libs as needed
to implement deeply integrated local branching in Svn client. The full
extent of what that might entail or look like is unknown.

3. To store a series of checkpoints, create a temporary repos inside
.svn/ and "relocate" the WC base to it. Then use it for all operations
until the user comes to the point they want to commit to the real repo.


>> SHELVING
>>
>> git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>
>
> For git, 'stashing' can also be implemented by a temporary branch («git
> checkout -b foo && git commit -amm && git checkout master»), which
> changes some of the values of the table.  I assume the same is true for hg.

Yes, I should note that mode of working as well. Thanks.

- Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Daniel Shahaf-5
Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:

> Daniel Shahaf wrote:
> > Julian Foad wrote:
> >> Checkpointing
> >>
> >> Options:
> >>
> >>     1. further patch management built on a series of shelved changes
> >>     2. local commits tightly integrated
> >>     3. checkpoints are commits in a local repository
> >
> > Can you explain these three options in more words?  AIUI #1 is
> > "syntactic sugar to manage a patch series", like quilt(1), but I'm not
> > sure I understand #2 and #3.
>
> 2. What I was thinking there is to rewrite as much of our libs as needed
> to implement deeply integrated local branching in Svn client. The full
> extent of what that might entail or look like is unknown.

I don't understand the distinction apparent here between 'checkpointing'
and 'local branching'.

> 3. To store a series of checkpoints, create a temporary repos inside
> .svn/ and "relocate" the WC base to it. Then use it for all operations
> until the user comes to the point they want to commit to the real repo.

I think when wc-ng was invented, the thinking was that a 'checkpoint'
would simply be another tree, alongside BASE and WORKING.  None


>
> >> SHELVING
> >>
> >> git stashing <https://git-scm.com/book/en/v1/Git-Tools-Stashing>
> >
> > For git, 'stashing' can also be implemented by a temporary branch («git
> > checkout -b foo && git commit -amm && git checkout master»), which
> > changes some of the values of the table.  I assume the same is true for hg.
>
> Yes, I should note that mode of working as well. Thanks.
>
> - Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Nathan Hartman
In reply to this post by Julian Foad-5
On Tue, Jul 11, 2017 at 4:53 AM, Julian Foad <[hidden email]> wrote:
Nathan Hartman wrote:
[...] What if, instead of just a pristine copy, it actually created a private local repository. Revision 1 of this repository would be the pristine copy. [...]

That is exactly what I was thinking about when I wrote "Option 3... Checkpoints are commits in a local repository embedded in the WC" towards the end of that document. My feeling is that both the amount of effort required just to get that far, and the explosion of further possibilities it would open up, make it too heavy-weight for my current project, but I would definitely like to explore that possibility further.

The explosion of possibilities, to borrow your words, is IMO a disadvantage of Option 3. I'm saying this despite having suggested this same idea.

I'd like to emphasize that I don't advocate turning Subversion into a DVCS. Proponents of DVCSs seem to think that Subversion's centralized nature is a disadvantage. I think the exact opposite. With centralized, you get Single Source of Truth, a system that is easy to learn, easy to teach, programmers and non programmers can use it successfully. I like that history is immutable, never rewritten, because I think the primary responsibility is to never lose information. You only have two places to know about: the repository and your working copy.

What I do like is the idea of remaining centralized while having some local capabilities. The stashing / checkpointing wishlist items are valid. I mentioned the ability to roll back a svn update if something unexpected happens (by doing an implied local commit of your current state as the first step of svn update). Local feature branches could be useful for one-off experiments, short-lived work done by one person, etc. In any case, these features should all be optional. They'd be there if you want them, but you never have to know about them if you're not interested.

In any event, I wouldn't go full blown DVCS where the concept of repository vs. working copy goes away. I wouldn't turn a checkout into a full clone, or a working copy into a repository.

Other than the addition of new features in the client, all else would remain the same.

Why put a hidden repository in the working copy, then? The reason is that while the client could use a different "lighter weight" method to manage various versions of its working copy, that would reinvent the wheel. You'd end up with a second incompatible version control system within Subversion. If I understand correctly, the client already has server code in it, because when you use the file:// URL scheme, the client is the server. So why reinvent a new version control system within Subversion when it already has everything it needs?

But here's the key: The difference between a normal repository and the hidden one in the working copy is that the latter is a hidden implementation detail, not something the user can access or control. For example you can't checkout a working copy of it. Everything that concerns that hidden repo, its creation, the directory and file layout within it... all of that is controlled by Subversion itself. It's more of an internal client-side filing system than an actual repository, although it uses a repository as the underlying implementation. The user gets some new local features, but not the dreaded "explosion of possibilities."

Kind regards,
Nathan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Daniel Shahaf-5
In reply to this post by Daniel Shahaf-5
Sorry, pressed [Send] too early.

Thanks for the explanations.  More below.

Daniel Shahaf wrote on Tue, 11 Jul 2017 22:02 +0000:

> Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:
> > Daniel Shahaf wrote:
> > > Julian Foad wrote:
> > >> Checkpointing
> > >>
> > >> Options:
> > >>
> > >>     1. further patch management built on a series of shelved changes
> > >>     2. local commits tightly integrated
> > >>     3. checkpoints are commits in a local repository
> > >
> > > Can you explain these three options in more words?  AIUI #1 is
> > > "syntactic sugar to manage a patch series", like quilt(1), but I'm not
> > > sure I understand #2 and #3.
> >
> > 2. What I was thinking there is to rewrite as much of our libs as needed
> > to implement deeply integrated local branching in Svn client. The full
> > extent of what that might entail or look like is unknown.

I don't understand the distinction apparent here between 'checkpointing'
and 'local branching'.  Are these two terms not synonymous?

> > 3. To store a series of checkpoints, create a temporary repos inside
> > .svn/ and "relocate" the WC base to it. Then use it for all operations
> > until the user comes to the point they want to commit to the real repo.

I think when wc-ng was invented, the idea was that a 'checkpoint' would
simply be another tree, alongside BASE and WORKING.  I suppose this
is option #4.

In general, we have at least five ways of representing a tree: as a
dumpstream, as an editor stream, as a FSFS revision file, as a wc.db
(+ pristine store), as a diff against an empty tree.  The 'temporary
repository' could store the changes in any of these forms.  I suppose the
question is which representation is best amenable to being replayed and to
resolving conflicts (when rebasing a checkpoint onto a younger start
revision).

Cheers,

Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Branko Čibej
In reply to this post by Julian Foad-5
On 11.07.2017 10:53, Julian Foad wrote:

> Thanks for your suggestion, Nathan.
>
> Nathan Hartman wrote:
>> [...] What if, instead of just a pristine copy, it actually created a
>> private local repository. Revision 1 of this repository would be the
>> pristine copy. [...] if you type some other command instead of
>> commit, or maybe prepend the word "local" or something, the commit
>> would go into the local repository [...] Shelving, stashing, etc.,
>> all become local operations against this local souped up pristine
>> copy replacement.
>
> That is exactly what I was thinking about when I wrote "Option 3...
> Checkpoints are commits in a local repository embedded in the WC"
> towards the end of that document. My feeling is that both the amount
> of effort required just to get that far, and the explosion of further
> possibilities it would open up, make it too heavy-weight for my
> current project, but I would definitely like to explore that
> possibility further.

Ben (Reser) and I discussed this to death at one of the hackathons. It
turns out that checkpoints / local commits are, while not a
pre-requisite, definitely a logical first step on the road to stashes
and local branches. We even had a pretty good architectural model and UI
feature list written down ... I wonder where that's gone now.

The problem we encountered at the time is that the current working copy
schema is too limited to implement them effectively; it needs another
dimension of change tracking.

-- Brane

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
In reply to this post by Daniel Shahaf-5
Daniel Shahaf wrote:
> Julian Foad wrote on Tue, 11 Jul 2017 21:53 +0100:
>> 2. What I was thinking there is to rewrite as much of our libs as needed
>> to implement deeply integrated local branching in Svn client. The full
>> extent of what that might entail or look like is unknown.
>
> I don't understand the distinction apparent here between 'checkpointing'
> and 'local branching'.  Are these two terms not synonymous?

Sorry, I didn't mean to make a distinction. "... implement deeply
integrated <whatever it takes>". Such as extra trees in the WC DB, as
envisioned in Reference 5. Svn Wiki page "Design: SavePoints"
<https://wiki.apache.org/subversion/SavePoints>, for example.

- Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
In reply to this post by Julian Foad-7
I committed an initial prototype for shelving on the "shelve-checkpoint"
branch.

Here is the help output:
[[[
$ svn shelve --help
shelve: Shelve changes.
usage: 1. shelve NAME PATH...
        2. shelve --delete NAME
        3. shelve --list

   1. Shelve as NAME the local changes in the given PATHs.
   2. Delete the shelved patch NAME.
   3. List shelved patches.
[...]

$ svn unshelve --help
unshelve: Unshelve changes.
usage: unshelve [--keep] NAME

]]]

Here is an example session:
[[[
$ # create and commit some files

$ mkdir doc; echo hello > doc/hello.txt; echo config > config

$ svn add --force .; svn ci -q -m ""
A         config
A         doc
A         doc/hello.txt

$ # start making some changes to docs

$ echo new > doc/new.txt; svn add doc/new.txt
A         doc/new.txt

$ echo more >> config

$ svn st
M       config
A       doc/new.txt

$ # shelve these documentation changes

$ svn shelve docs .
U         config
D         doc/new.txt
shelved 'docs'

$ svn st

$ svn shelve --list
docs.patch

$ # work on a quick fix and commit it

$ echo Hello > doc/hello.txt

$ svn ci -q -m "Fix capitalization."

$ # unshelve to continue work on docs

$ svn unshelve docs
U         config
A         doc/new.txt
unshelved 'docs'

$ svn st
M       config
A       doc/new.txt

$ svn shelve --list

]]]

If you have a chance to try it yourself, I'd love to hear your first
impressions.

- Julian


> References:
>     [1] Shelving-Checkpointing Dev doc. (J Foad)
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Johan Corveleyn-3
In reply to this post by Julian Foad-7
On Mon, Jul 10, 2017 at 2:59 PM, Julian Foad <[hidden email]> wrote:

> Dear Subversion Developers,
>
> I am delighted to announce that I am working with Assembla to develop
> shelving and checkpointing functionality in Subversion. These have
> been on the wish list for many years, and are becoming ever more in
> demand since the popularity of git is making more users appreciate the
> benefits of local operations.
>
> Based on our many discussions in the past, along with a fresh look at
> how other VCS's implement these features, I have written down the
> requirements and started working on a design in the document
> Shelving-Checkpointing Dev [1]. I will follow up with an HTML email
> containing a copy of today's version. I invite you to leave comments
> directly in Google Docs, or send them by reply here, as you prefer.
>
> I will be the project lead on this within Assembla. I will regularly
> demonstrate small increments of functionality, which will also act as
> calls for feedback. We want to keep the scope quite small to get it
> done as quickly as possible.
>
> Please read through and if you are able to contribute with any
> suggestions or practical help, that would be wonderful. Thanks!
>
> - Julian
>
>
> References:
>    [1] Shelving-Checkpointing Dev doc. (J Foad)
> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#

Hi Julian,

That's great news! I hope this will give Subversion some much needed
extra momentum (together with the tree conflict resolution
improvements of 1.10). Indeed, these "local / offline" features have
been on many people's wish list for a long time.

I've quickly scanned your google docs, but have to go through them in
some more detail. I'll try to dig into them and give some feedback if
I can.

One thing that crossed my mind: a nice additional feature that
probably could be built on top of shelving would be "partial commit"
(committing only some hunks out of a modified file). I've recently
looked up past discussions about this, in response to a post on
users@. See my response with a lot of references here:

    https://svn.haxx.se/users/archive-2017-06/0004.shtml
(For the "partial commit" feature, start reading below "Is there an
option to inspect each file further line-by-line for lines that have
changed to either be selected or excluded from the commit?")

As I explained in that post, TortoiseSVN has implemented this in some
way with what they call "restore after commit" [1].

So I imagine we could do the same by (1) making the user select some
lines he wants to commit; (2) shelve the modifications that were not
selected; (3) commit; (4) unshelve. We even have a command line UI
(sort of) for selecting lines out of a diff, namely the UI stsp
created for handling text conflicts interactively.

Just a thought ... Of course fleshing out the fundamentals of the
Shelving and Checkpointing features in the first place will be most
important.

[1] https://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-dug-commit.html#tsvn-dug-commit-restore

--
Johan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Mark Phippard-3
In reply to this post by Julian Foad-5
On Wed, Jul 12, 2017 at 4:18 AM, Julian Foad <[hidden email]> wrote:
I committed an initial prototype for shelving on the "shelve-checkpoint" branch.

Here is the help output:
[[[
$ svn shelve --help
shelve: Shelve changes.
usage: 1. shelve NAME PATH...
       2. shelve --delete NAME
       3. shelve --list

  1. Shelve as NAME the local changes in the given PATHs.
  2. Delete the shelved patch NAME.
  3. List shelved patches.
[...]

$ svn unshelve --help
unshelve: Unshelve changes.
usage: unshelve [--keep] NAME

]]]

Here is an example session:
[[[
$ # create and commit some files

$ mkdir doc; echo hello > doc/hello.txt; echo config > config

$ svn add --force .; svn ci -q -m ""
A         config
A         doc
A         doc/hello.txt

$ # start making some changes to docs

$ echo new > doc/new.txt; svn add doc/new.txt
A         doc/new.txt

$ echo more >> config

$ svn st
M       config
A       doc/new.txt

$ # shelve these documentation changes

$ svn shelve docs .
U         config
D         doc/new.txt
shelved 'docs'

$ svn st

$ svn shelve --list
docs.patch

$ # work on a quick fix and commit it

$ echo Hello > doc/hello.txt

$ svn ci -q -m "Fix capitalization."

$ # unshelve to continue work on docs

$ svn unshelve docs
U         config
A         doc/new.txt
unshelved 'docs'

$ svn st
M       config
A       doc/new.txt

$ svn shelve --list

]]]

If you have a chance to try it yourself, I'd love to hear your first impressions.




Nice to see you have gotten this far.

I am not even sure how this behaves with git stash but what happens if the patch cannot be applied cleanly?  Is the path to "fixing things" relatively clear?  Like does patch just create conflicts that you resolve like anything else?

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
In reply to this post by Johan Corveleyn-3
Johan Corveleyn wrote:
> I've quickly scanned your google docs, but have to go through them in
> some more detail. I'll try to dig into them and give some feedback if
> I can.

Thanks you Johan! I look forward to hearing your comments when you have
a chance.

> One thing that crossed my mind: a nice additional feature that
> probably could be built on top of shelving would be "partial commit"
> (committing only some hunks out of a modified file). I've recently
> looked up past discussions about this, in response to a post on
> users@. See my response with a lot of references here:
>
>      https://svn.haxx.se/users/archive-2017-06/0004.shtml

Yes, I would find that useful myself. A good future extension.

That's what I mean by the 'patch hunks' feature listed in the 'Role
Models' table in the 'UI' doc which is linked from the 'Dev' doc. Bazaar
'bzr shelve' does that interactively by default, and both Mercurial and
Git do it with 'hg shelve --interactive' and 'git stash --patch'.

- Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
In reply to this post by Mark Phippard-3
Mark Phippard wrote:
> Nice to see you have gotten this far.
>
> I am not even sure how this behaves with git stash but what happens if
> the patch cannot be applied cleanly?  Is the path to "fixing things"
> relatively clear?  Like does patch just create conflicts that you
> resolve like anything else?

It is 'svn patch' -- so it raises conflicts.

- Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [RFC] Shelving and Checkpointing

Bert Huijben-5


> -----Original Message-----
> From: Julian Foad [mailto:[hidden email]]
> Sent: woensdag 12 juli 2017 16:38
> To: Mark Phippard <[hidden email]>
> Cc: Subversion Developers <[hidden email]>
> Subject: Re: [RFC] Shelving and Checkpointing
>
> Mark Phippard wrote:
> > Nice to see you have gotten this far.
> >
> > I am not even sure how this behaves with git stash but what happens if
> > the patch cannot be applied cleanly?  Is the path to "fixing things"
> > relatively clear?  Like does patch just create conflicts that you
> > resolve like anything else?
>
> It is 'svn patch' -- so it raises conflicts.

Svn patch creates reject files. It doesn't create conflicts (yet).

        Bert

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
Bert Huijben wrote:
> Julian Foad wrote:
>> Mark Phippard wrote:
>>> [...]  does patch just create conflicts that you
>>> resolve like anything else?
>>
>> It is 'svn patch' -- so it raises conflicts.
>
> Svn patch creates reject files. It doesn't create conflicts (yet).

Gosh, so it does! How primitive! :-)

I will expand on that aspect.

- Julian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Julian Foad-5
In reply to this post by Julian Foad-5
I committed an initial prototype for checkpointing on the
"shelve-checkpoint" branch.  http://svn.apache.org/r1801942

This prototype is of the "store snapshot patches" ("option 1") design.

Here is the help output:
[[[
$ svn checkpoint --help
checkpoint: Checkpoint the local changes.
usage: 1. checkpoint save
        2. checkpoint revert
        3. checkpoint rollback NUMBER
        4. checkpoint list|--list

   1. Save the working state as a new checkpoint.
   2. Revert the working state to the current checkpoint.
   3. Roll back the working state to checkpoint NUMBER.
   4. List all checkpoints. A synonym for 'svn checkpoints'.
[...]
]]]

Please try it and let me know your thoughts on the directions we should
take it in, if you have a chance.

I think the next most important things it needs are:

   * recognize when there are no further changes relative to the last
checkpoint, and do nothing in that case;
   * see the status and diff of the WC against the current checkpoint;
   * see a summary and diff of each checkpoint against the previous one.

Diffing checkpoints is quite hard (well, needs a patch-arithmetic
library) in this design. Even just determining whether two patches are
"identical". That might be a good reason for me to try the alternative
design "store checkpoints in a local repository" ("option 3") next.

- Julian


On 12/07/17, Julian Foad wrote:

> I committed an initial prototype for shelving on the "shelve-checkpoint"
> branch.
>
> Here is the help output:
> [[[
> $ svn shelve --help
> shelve: Shelve changes.
> usage: 1. shelve NAME PATH...
>         2. shelve --delete NAME
>         3. shelve --list
>
>    1. Shelve as NAME the local changes in the given PATHs.
>    2. Delete the shelved patch NAME.
>    3. List shelved patches.
> [...]
>
> $ svn unshelve --help
> unshelve: Unshelve changes.
> usage: unshelve [--keep] NAME
>
> ]]]
>
> Here is an example session:
> [[[
> $ # create and commit some files
>
> $ mkdir doc; echo hello > doc/hello.txt; echo config > config
>
> $ svn add --force .; svn ci -q -m ""
> A         config
> A         doc
> A         doc/hello.txt
>
> $ # start making some changes to docs
>
> $ echo new > doc/new.txt; svn add doc/new.txt
> A         doc/new.txt
>
> $ echo more >> config
>
> $ svn st
> M       config
> A       doc/new.txt
>
> $ # shelve these documentation changes
>
> $ svn shelve docs .
> U         config
> D         doc/new.txt
> shelved 'docs'
>
> $ svn st
>
> $ svn shelve --list
> docs.patch
>
> $ # work on a quick fix and commit it
>
> $ echo Hello > doc/hello.txt
>
> $ svn ci -q -m "Fix capitalization."
>
> $ # unshelve to continue work on docs
>
> $ svn unshelve docs
> U         config
> A         doc/new.txt
> unshelved 'docs'
>
> $ svn st
> M       config
> A       doc/new.txt
>
> $ svn shelve --list
>
> ]]]
>
> If you have a chance to try it yourself, I'd love to hear your first
> impressions.
>
> - Julian
>
>
>> References:
>>     [1] Shelving-Checkpointing Dev doc. (J Foad)
>> https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit# 
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [RFC] Shelving and Checkpointing

Nathan Hartman
On Fri, Jul 14, 2017 at 7:56 AM, Julian Foad <[hidden email]> wrote:
Diffing checkpoints is quite hard (well, needs a patch-arithmetic library) in this design. Even just determining whether two patches are "identical". That might be a good reason for me to try the alternative design "store checkpoints in a local repository" ("option 3") next.

My thinking is that options 1 or 2 will eventually grow into a redundant implementation of a VCS within svn to handle various revisions of the working copy. That will most likely be faster to get off the ground, but results in redundant code, or refactoring existing code to do the patch arithmetic, etc. Option 3 requires reinventing the client outright but gives more flexibility in the long run.

I was thinking that svn switch needs to be handled, or chaos would ensue if someone does some work, makes some shelves and checkpoints, and then does svn switch to a different path. A solution could be to remember the checkout path and revision with each shelve/checkpoint. If the user performs svn switch and later wants to reinstate a shelved working copy, the client will notice that it can't be unshelved until the workspace is switched back to the correct path and error out with a descriptive message. The user will then svn switch to the correct branch and reissue the unshelve command. The normal rules apply, which mean that if there are uncommitted changes in the working copy, the user will have to either commit them or shelve them before performing the svn switch. A --force option could allow the user to unshelve the work into the wrong path, possibly with conflicts (but maybe this is intentional).


References:
   [1] Shelving-Checkpointing Dev doc. (J Foad)
https://docs.google.com/document/d/1PVgw0BdPF7v67oxIK7B_Yjmr3p28ojabP5N1PfZTsHk/edit#
12
Loading...