Re: PMCs: any Hackathon requests? (deadline 11 October)

Next Topic
 
classic Classic list List threaded Threaded
153 messages Options
1234 ... 8
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Julian Foad-5
Moving this thread to dev@ as it should be public [1].

See my responses inline.

> Sally Khudairi <[hidden email]> wrote:
>> We received an email from hackCBS 2.0, a hackathon that will be taking place 19-20 October at the University of Delhi with 700+ students.
>>
>> They are interested in our participation by providing a list of tasks from various Apache projects for them to work on.
>>
>> If you would like to submit a list of work areas, please let me know your interest and forward your list(s) no later than 11 October.
>
> Johan Corveleyn wrote:
>> Nathan Hartman wrote:
>>> We should totally take advantage of this hackathon request, if at all feasible.
>>
>> I'm not sure if we will be able to get enough attention, but there's
>> no harm in trying. Some ideas are listed on
>> http://subversion.apache.org/ideas.html.
>> But of course there are many other possible suggestions (perhaps some
>> more fresh ideas, appealing to University students, ...).
>
> I propose to reply to Sally's message with something along the
> following lines, pending thoughts / suggestions / criticisms...
> Opportunities for new devs must not slip through our fingers!
>
> [[[
> [...]
> The Apache Subversion PMC is always interested in attracting new
> developers to the project.
>
> Between our 'ideas' page and issue tracker, we have some opportunities
> [...]
> ]]]

Johan Corveleyn wrote:

> Nathan Hartman wrote:
>> Daniel Shahaf wrote:
>>> I read Sally's request as asking for a (reviewed/edited) list of tasks,
>>> not just asking each project to tell her where their issue tracker is.  Seeing
>>> as the duration of the hackathon is two days, perhaps linking to the
>>> "bite-sized" label could work, though?
>>
>> I agree.
>>
>> Can we come up with 2 or 3 suggested tasks? I'll go first: I nominate
>> https://issues.apache.org/jira/browse/SVN-2858.
>
> Ah, that one :-). Note that Neels Hofmeyr did work on a design for
> that back in 2011 (see
> https://svn.apache.org/repos/asf/subversion/trunk/notes/hold). You can
> also find several discussion threads about it in 2011 if you search
> for "svn:hold".

That's a bad choice because it's a new feature that isn't clearly agreed
and defined, so it requires proposing and debating a design and all its
interactions with existing behaviour.

We're in a period where we should be working instead toward stability
and availability.

> Another potential task might be: overhaul of our website, for which I
> think you had some ideas, right, Nathan? Maybe someone with web design
> skills might be able to do part of the work (under your guidance)?

A subtask of that might be suitable.

> Another suggestion:
> https://issues.apache.org/jira/browse/SVN-4753 (SVN viewspec feature)
> We now have "svn info --x-viewspec classic|svn11". We need a
> corresponding way to "consume" viewspecs (e.g. "svn update
> --apply-viewspec some.spec" [...]

I would consider that more of a missing feature, and so potentially
something I'd like us to do, but unfortunately it requires deep
understanding and careful changing of the core APIs.

- Julian


[1] This discussion should have been public from the start.  Generally
one would need to ask all participants before making it public, but in
this case I'm confident we'll agree and it's just that unfortunately
Sally sent it to private@ by default, as an easy way of contacting all
the PMCs.  See a long thread about the problem, with nobody agreeing to
do anything about it, on [hidden email] 2018-08-23 "Keeping
PMC communications public when possible":
https://lists.apache.org/thread.html/9faff8ffc1e5017b241c47a4e62bf55a5b8b2e9365bb3f48f9700ec4@%3Cdev.community.apache.org%3E
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Nathan Hartman
On Mon, Oct 7, 2019 at 5:13 AM Julian Foad <[hidden email]> wrote:
> Moving this thread to dev@ as it should be public [1].

Sally Khudairi <[hidden email]> wrote:

> We received an email from hackCBS 2.0, a hackathon that will be
> taking place 19-20 October at the University of Delhi with 700+
> students.
>
> They are interested in our participation by providing a list of
> tasks from various Apache projects for them to work on.
>
> If you would like to submit a list of work areas, please let me
> know your interest and forward your list(s) no later than 11
> October.

We should take advantage of this hackathon request. We should do the
same for any opportunity to increase participation!

Since there will be 700+ university participants, there may be a
variety of different skill sets, but we don't know what they are.

Let's make a list of 2 or 3 items, each from a different skill area.

Perhaps:
* 1 "byte size" code quality issue from the issue tracker
* 1 website task
* 1 Py3 support task

Byte size issue tracker task: Some suggestions were already made but
are not a good fit for several reasons. Let's suggest other ones. I'll
go first again: https://issues.apache.org/jira/browse/SVN-1722 "svn
diff may missreport a revision as the working copy."

Website task: Johan suggests the website overhaul and Julian suggests
a subtask of that. I think that's a good idea. The task I suggest is
to incorporate normalize.css and main.css from html5boilerplate.com
and improve the navigation bar to make the static pages mobile
friendly.

Python 3 support task: I think something Python-related should be on
the list. How about https://issues.apache.org/jira/browse/SVN-2079
"utf8_tests.py should be made non-iso8859-1 specific"

Thoughts? Other items we should consider?

Nathan
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Julian Foad-5
Nathan Hartman wrote:
> Perhaps:
> * 1 "byte size" code quality issue from the issue tracker
> * 1 website task
> * 1 Py3 support task

Generally, sure, sounds good.  Go for it.

> https://issues.apache.org/jira/browse/SVN-1722 "svn
> diff may missreport a revision as the working copy."

Errm... last reviewed 9 years ago.  Is it really what it says it is?

Sadly, we have hundreds of unreviewed (or not recently reviewed) open
issues.  Some are misleadingly labeled "bite-sized" but are really
symptoms of deeper problems.  Can we take this opportunity to do
something about that?  Like, maybe start bringing one old issue to the
dev list for review each week?

- Julian
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Nathan Hartman
In reply to this post by Nathan Hartman
On Mon, Oct 7, 2019 at 10:49 AM Nathan Hartman <[hidden email]> wrote:
> Since there will be 700+ university participants, there may be a
> variety of different skill sets, but we don't know what they are.
>
> Let's make a list of 2 or 3 items, each from a different skill area.
>
> Perhaps:
> * 1 "byte size" code quality issue from the issue tracker
> * 1 website task
> * 1 Py3 support task

I think we have our website task:
> to incorporate normalize.css and main.css from html5boilerplate.com
> and improve the navigation bar to make the static pages mobile
> friendly.

Now can we please have some more suggestions for a "byte size"
issue and a Python-related task?

Reminder: We must reply to the hackathon request this week!
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Daniel Shahaf-2
Nathan Hartman wrote on Tue, 08 Oct 2019 15:09 +00:00:
> Now can we please have some more suggestions for a "byte size"
> issue and a Python-related task?

SVN-4781 - expose svn_fs_change_rev_prop2() to swig.

SVN-4568 - expose svn_fs_set_warning_func() to swig.

SVN-4343 - "svnadmin verify" should verify changed-paths list.  (FSFS
backend work; not hard, but need to make sure the solution scales)

SVN-4605 - "svnadmin verify" doesn't verify locks.

SVN-3914 - INSTALL: document how to compile with libmagic on Windows.
(Is this also a Python task, to _implement_ linking with libmagic on
windows in the build scripts?)

SVN-4065 - server should enforce LF normalization for svn:eol-
style=native files. (Not just a patch is needed, but also a FAQ entry
that explains how to handle existing repositories that have this
problem, once the patch is released. If it'll be like when
svn_repos__validate_prop() was added, then dump/load will still work,
but svnsync not.)

> Reminder: We must reply to the hackathon request this week!
>
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Julian Foad-5
Daniel Shahaf wrote:
> SVN-4065 - server should enforce LF normalization for svn:eol-
> style=native files. (Not just a patch is needed, but also a FAQ entry
> that explains how to handle existing repositories that have this
> problem, once the patch is released. If it'll be like when
> svn_repos__validate_prop() was added, then dump/load will still work,
> but svnsync not.)

That issue appears to need review.  It's not clear to me whether the
server should really enforce LF normalization.  I rather think not.
There's a discussion thread linked from it, with no final conclusion.

- Julian

Reply | Threaded
Open this post in threaded view
|

SVN-4065 - server should enforce LF normalization for svn:eol-style=native (was: PMCs: any Hackathon requests? (deadline 11 October))

Johan Corveleyn-3
On Tue, Oct 8, 2019 at 6:13 PM Julian Foad <[hidden email]> wrote:

> Daniel Shahaf wrote:
> > SVN-4065 - server should enforce LF normalization for svn:eol-
> > style=native files. (Not just a patch is needed, but also a FAQ entry
> > that explains how to handle existing repositories that have this
> > problem, once the patch is released. If it'll be like when
> > svn_repos__validate_prop() was added, then dump/load will still work,
> > but svnsync not.)
>
> That issue appears to need review.  It's not clear to me whether the
> server should really enforce LF normalization.  I rather think not.
> There's a discussion thread linked from it, with no final conclusion.

My last comment in the issue [1] summarizes the discussion(s) as follows:

[[[
Some recent discussion on dev@:
    http://svn.haxx.se/dev/archive-2012-12/0179.shtml

The current thinking from that thread is that this should (or can only) be
enforced through a pre-commit hook. It was suggested that this would ideally be
some standard, supported (and efficient) pre-commit hook. In a follow-up thread,
Ivan suggested to create a new program "svnhooks" for standardizing such hooks
(and go together with a configuration file to enable/disable certain behaviors):
    http://svn.haxx.se/dev/archive-2012-12/0217.shtml
]]]

I think that was the conclusion from those threads. I.e. it would be
best if we developed a standard "svnhooks" program that can be invoked
from the pre-commit hook (and not try to implement this directly in
the repos layer). At least, after those svnhooks suggestions no-one
objected, so I assumed silent consensus about that way forward :-) ...

Not sure if this is a good bite-sized task for interested hackers though ...
Though it's quite well-contained (develop a separate (small) program
with a configuration file, depending on existing server-side API's),
and we have a clear use case to start with.

[1] https://issues.apache.org/jira/browse/SVN-4065?focusedCommentId=14931167&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14931167

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Julian Foad-5
Johan Corveleyn wrote:
> I think that was the conclusion from those threads. I.e. it would be
> best if we developed a standard "svnhooks" program that can be invoked
> from the pre-commit hook (and not try to implement this directly in
> the repos layer). At least, after those svnhooks suggestions no-one
> objected, so I assumed silent consensus about that way forward :-) ...

Silence meant "errm... what?"

> Not sure if this is a good bite-sized task for interested hackers though ...
> Though it's quite well-contained (develop a separate (small) program
> with a configuration file, depending on existing server-side API's),
> and we have a clear use case to start with.

Glad we're reviewing this old issue.

What do you envision the purpose/scope/role of this program should be?
It's easy to suggest a facetious answer: "a collecting place for random
bug fixes".  Of course not.

Seriously, the question that needs addressing for this issue is at what
level the LF normalization is to be enforced in a Subversion system
(deployment) in general.  So far it has been client-side, with the
implication that different clients are expected but not forcibly
required to co-operate.

The first implication of that is that clients should handle gracefully
the case where repository data is not in fact normalized how they
expected it.  That's one actionable conclusion.

If we want to change this to a repository-level rule, then that implies
we are changing the repository semantics in a backward-incompatible way
and we would need to address that (using a format number bump, and an
upgrade/migration path).  We could discuss that path but I don't think
anyone's currently pushing for that and I don't see good reason to do
so, so let's leave that aside.

It seems we want to keep the design where this normalization is a
client-side rule, but now we want to provide additional server-side
enforcement of the client-side rule.  We must still in odd cases allow
server tools (like dump/load) to continue working with repository
content that doesn't obey that rule.

That means we have to design it in such a way that only "client commits"
are affected, and "server tools" such as sump/load are not, because we
can't have them suddenly start failing to load existing repository data.

Are "svnrdump" and "svnsync" client-layer or repos-layer tools, for
these purposes?  We don't yet have a formal way to distinguish and use
"repos-layer" tools through our client-server (RA) interface.  So at the
moment I suppose we might say that ("de facto") all interactions through
the RA interface are deemed to be client-layer interactions.  We could
consider an enhancement to enable "repos-layer" interactions to be
performed over RA with suitable authorization.  We probably ought to
file that as a future enhancement issue.

Currently we have "svn commit" and other client-layer operations, vs.
"svnadmin load" as the main repos-layer (server side) interaction.

"svnadmin load" has these options:
   --use-pre-commit-hook
   --use-post-commit-hook
   --normalize-props
   --bypass-prop-validation

To me, this looks like a crude attempt to allow it to support both
client-layer and repos-layer modes, but with no overall mode switch so
the user has to think about the implications of each individual sub-switch.

We never spelled out the role of commit hooks.  Therefore presumably
some commit hooks are used for client-side purposes (enforcing rules
like LF normalization and log message formats, etc.) and some for
server-side purposes (logging, backups, mirroring to svnsync, etc.).
The option to enable or disable all commit hooks seems misguided:
instead it would seem more appropriate to categorize hooks into client
and server roles, and have "svnadmin load" apply only those in the
server role.

Is two roles of hooks something it makes sense to introduce?  Or could
we clarify the current situation, document it?

It looks to me like "enforcement of the standard svn client's rules" is
an option we would ideally like to provide to server operators.  How
would this be defined more precisely?  How should we design it to cope
with different client versions, whose rules have changed a few times?

- Julian
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Julian Foad-5
Julian Foad wrote:
> If we want to change this to a repository-level rule, then that implies
> we are changing the repository semantics in a backward-incompatible way
> and we would need to address that (using a format number bump, and an
> upgrade/migration path).  We could discuss that path but I don't think
> anyone's currently pushing for that and I don't see good reason to do
> so, so let's leave that aside.

I want to add something stronger.  It would be a mistake to push
validation of file content against property values into the repos layer
as a new rule about the allowed semantics of repository contents.

It's important to have different levels in the architecture with
strongly defined characteristics, generally with lower layers having
simpler semantics.  Currently the repos layer does not interfere with
file content at all.  That is Good.

The repos layer to a large extent transparent to properties and their
values, though not so much: it has some validation and even
"normalization" of "svn:" property names and values.  I feel this is
generally Bad; there is some room for repos-layer knowledge of
properties but we should have separated the concerns better.

- Julian
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Branko Čibej
On 09.10.2019 11:50, Julian Foad wrote:

> Julian Foad wrote:
>> If we want to change this to a repository-level rule, then that
>> implies we are changing the repository semantics in a
>> backward-incompatible way and we would need to address that (using a
>> format number bump, and an upgrade/migration path).  We could discuss
>> that path but I don't think anyone's currently pushing for that and I
>> don't see good reason to do so, so let's leave that aside.
>
> I want to add something stronger.  It would be a mistake to push
> validation of file content against property values into the repos
> layer as a new rule about the allowed semantics of repository contents.
>
> It's important to have different levels in the architecture with
> strongly defined characteristics, generally with lower layers having
> simpler semantics.  Currently the repos layer does not interfere with
> file content at all.  That is Good.

Couldn't agree more. There's a rather large can of really ugly worms
hiding in there and we do not want to look for it, let alone open it.


> The repos layer to a large extent transparent to properties and their
> values, though not so much: it has some validation and even
> "normalization" of "svn:" property names and values.  I feel this is
> generally Bad; there is some room for repos-layer knowledge of
> properties but we should have separated the concerns better.

Does the repos layer actually normalize svn: properties? I know
'svnadmin load' can, but I don't believe the repos API does that?

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Johan Corveleyn-3
In reply to this post by Julian Foad-5
On Wed, Oct 9, 2019 at 11:41 AM Julian Foad <[hidden email]> wrote:
>
> Johan Corveleyn wrote:
> > I think that was the conclusion from those threads. I.e. it would be
> > best if we developed a standard "svnhooks" program that can be invoked
> > from the pre-commit hook (and not try to implement this directly in
> > the repos layer). At least, after those svnhooks suggestions no-one
> > objected, so I assumed silent consensus about that way forward :-) ...
>
> Silence meant "errm... what?"

I disagree. On this list silence means "no objections". If someone
wanted to say "errm... what?", they should have mailed it.

In that very thread, 7 years ago, it seemed quite clear that consensus
was gravitating towards "don't solve it in the repos layer, but in a
pre-commit hook":
* Daniel first posted a simple script for pre-commit hook validation:
https://svn.haxx.se/dev/archive-2012-12/0180.shtml
* Brane said the only sane place to do it was in pre-commit hook:
https://svn.haxx.se/dev/archive-2012-12/0182.shtml
* Brane also suggested to write it in C:
https://svn.haxx.se/dev/archive-2012-12/0191.shtml
* Ivan suggested to create a separate program for it "svnhooks", with
this concrete "rule" as a first use case for a more general tool:
https://svn.haxx.se/dev/archive-2012-12/0202.shtml
* Ivan specified his idea a bit further here:
https://svn.haxx.se/dev/archive-2012-12/0217.shtml

During that entire discussion thread the only objections raised were
about "enforcing it in the repos layer / server directly". No-one
objected to the proposal(s) to solve the issue via pre-commit hooks.

Sound pretty consensus-y to me.

...
> That means we have to design it in such a way that only "client commits"
> are affected, and "server tools" such as sump/load are not, because we
> can't have them suddenly start failing to load existing repository data.

Yes, and hooks can do that. You're arriving at the same conclusion as
the thread 7 years ago.

> Are "svnrdump" and "svnsync" client-layer or repos-layer tools, for
> these purposes?  We don't yet have a formal way to distinguish and use
> "repos-layer" tools through our client-server (RA) interface.  So at the
> moment I suppose we might say that ("de facto") all interactions through
> the RA interface are deemed to be client-layer interactions.  We could
> consider an enhancement to enable "repos-layer" interactions to be
> performed over RA with suitable authorization.  We probably ought to
> file that as a future enhancement issue.
>
> Currently we have "svn commit" and other client-layer operations, vs.
> "svnadmin load" as the main repos-layer (server side) interaction.
>
> "svnadmin load" has these options:
>    --use-pre-commit-hook
>    --use-post-commit-hook
>    --normalize-props
>    --bypass-prop-validation
>
> To me, this looks like a crude attempt to allow it to support both
> client-layer and repos-layer modes, but with no overall mode switch so
> the user has to think about the implications of each individual sub-switch.
>
> We never spelled out the role of commit hooks.  Therefore presumably
> some commit hooks are used for client-side purposes (enforcing rules
> like LF normalization and log message formats, etc.) and some for
> server-side purposes (logging, backups, mirroring to svnsync, etc.).
> The option to enable or disable all commit hooks seems misguided:
> instead it would seem more appropriate to categorize hooks into client
> and server roles, and have "svnadmin load" apply only those in the
> server role.
>
> Is two roles of hooks something it makes sense to introduce?  Or could
> we clarify the current situation, document it?
>
> It looks to me like "enforcement of the standard svn client's rules" is
> an option we would ideally like to provide to server operators.  How
> would this be defined more precisely?  How should we design it to cope
> with different client versions, whose rules have changed a few times?

You're pulling this way out of scope. Issue SVN-4065 is very concrete
and specific, and a concrete proposal was made on how best to solve
it.

You're using it to open a much broader architectural discussion. Which
is fine of course, and I think quite interesting. But that shouldn't
block progress for SVN-4065 in the way which was proposed 7 years ago,
and which still seems the best way (via a utility program that can be
invoked from hooks, where our hooks still have the same semantics /
confusions / shortcomings / ... as today).

That being said, maybe that svnhooks utility program (Ivan's proposal)
could be used to introduce a way to separate those different roles
that hooks serve. Ivan hinted a bit in that direction in his post
there. From https://svn.haxx.se/dev/archive-2012-12/0217.shtml:

> svnhooks configuration file has separate section for each policy. For example:
> [[[
> [check-eol-normalization]
> enable = yes
>
> [rev-propchange]
> enable = yes
> users = svnsync
>
> [edit-log-message]
> enable = yes
> users = admin, @author
> ]]]

I agree this should be thought through, with a good design (a "users"
field might not cut it -- perhaps something else). It's clear that the
proposal needs further design work.

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Julian Foad-5
In reply to this post by Branko Čibej
Branko Čibej wrote:
>> The repos layer to a large extent transparent to properties and their
>> values, though not so much: it has some validation and even
>> "normalization" of "svn:" property names and values.  I feel this is
>> generally Bad; there is some room for repos-layer knowledge of
>> properties but we should have separated the concerns better.
>
> Does the repos layer actually normalize svn: properties? I know
> 'svnadmin load' can, but I don't believe the repos API does that?

AFAICT the RA parts of the libsvn_repos API never modify node props,
only revision props.

The normalization of node props done by 'svnadmin load' is implemented
in libsvn_repos through svn_repos_get_fs_build_parser6():

http://svn.apache.org/viewvc/subversion/tags/1.12.0/subversion/include/svn_repos.h?view=markup#l3863

That is yet another case where it's implemented at the wrong level.
Here, "svnrdump load" can't share it.  The dumpstream loader should be
refactored so svnadmin and svnrdump share code:
https://subversion.apache.org/issue/4780 "Factor out the dumpstream
loader editor driver".

- Julian
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Branko Čibej
In reply to this post by Johan Corveleyn-3
On 09.10.2019 15:00, Johan Corveleyn wrote:
> During that entire discussion thread the only objections raised were
> about "enforcing it in the repos layer / server directly". No-one
> objected to the proposal(s) to solve the issue via pre-commit hooks.

Validating contents (such as line endings based on svn:eol-style) is a
perfect fit for hooks. Not normalising of course. :)

-- Brane

Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Julian Foad-5
In reply to this post by Johan Corveleyn-3
Johan Corveleyn wrote:

> Julian Foad wrote:
>> Johan Corveleyn wrote:
>>> I think that was the conclusion from those threads. I.e. it would be
>>> best if we developed a standard "svnhooks" program that can be invoked
>>> from the pre-commit hook (and not try to implement this directly in
>>> the repos layer). At least, after those svnhooks suggestions no-one
>>> objected, so I assumed silent consensus about that way forward :-) ...
>>
>> Silence meant "errm... what?"
>
> I disagree. On this list silence means "no objections". If someone
> wanted to say "errm... what?", they should have mailed it.

Sorry, I was too harsh there.

Nevertheless, there are hundreds of issues and mail threads that are
incomplete but don't end with someone saying so.

My reading of the issue ended with the impression that there should be a
standard supplied "hooks program", but with no definition of what its
scope/purpose/role should be, so unclear what we should add to it, in
what contexts the svn server would execute it (like, svnadmin load?),
why it should be an external program, which server operators would
reasonably want to replace this program with a different one in what
kinds of situation, etc.

> In that very thread, 7 years ago, it seemed quite clear that consensus
> was gravitating towards "don't solve it in the repos layer, but in a
> pre-commit hook":
> * Daniel first posted a simple script for pre-commit hook validation:
> https://svn.haxx.se/dev/archive-2012-12/0180.shtml
[...]

> * Ivan suggested to create a separate program for it "svnhooks", with
> this concrete "rule" as a first use case for a more general tool:
> https://svn.haxx.se/dev/archive-2012-12/0202.shtml
> * Ivan specified his idea a bit further here:
> https://svn.haxx.se/dev/archive-2012-12/0217.shtml
>
> During that entire discussion thread the only objections raised were
> about "enforcing it in the repos layer / server directly". No-one
> objected to the proposal(s) to solve the issue via pre-commit hooks.
>
> Sound pretty consensus-y to me.

Thanks for summarizing the thread.  I had only skimmed through bits of
it.  In particular I hadn't seen where Ivan said the program would
implement hooks for "standard recommended polices", and I hadn't picked
up on it looking so consensus-y.

>> That means we have to design it in such a way that only "client commits"
>> are affected, and "server tools" such as sump/load are not, because we
>> can't have them suddenly start failing to load existing repository data.
>
> Yes, and hooks can do that. You're arriving at the same conclusion as
> the thread 7 years ago.
>
>> Are "svnrdump" and "svnsync" client-layer or repos-layer tools, for
>> these purposes?  We don't yet have a formal way to distinguish and use
>> "repos-layer" tools through our client-server (RA) interface.  So at the
>> moment I suppose we might say that ("de facto") all interactions through
>> the RA interface are deemed to be client-layer interactions.  We could
>> consider an enhancement to enable "repos-layer" interactions to be
>> performed over RA with suitable authorization.  We probably ought to
>> file that as a future enhancement issue.
>>
>> Currently we have "svn commit" and other client-layer operations, vs.
>> "svnadmin load" as the main repos-layer (server side) interaction.
>>
>> "svnadmin load" has these options:
>>     --use-pre-commit-hook
>>     --use-post-commit-hook
>>     --normalize-props
>>     --bypass-prop-validation
>>
>> To me, this looks like a crude attempt to allow it to support both
>> client-layer and repos-layer modes, but with no overall mode switch so
>> the user has to think about the implications of each individual sub-switch.
>>
>> We never spelled out the role of commit hooks.  Therefore presumably
>> some commit hooks are used for client-side purposes (enforcing rules
>> like LF normalization and log message formats, etc.) and some for
>> server-side purposes (logging, backups, mirroring to svnsync, etc.).
>> The option to enable or disable all commit hooks seems misguided:
>> instead it would seem more appropriate to categorize hooks into client
>> and server roles, and have "svnadmin load" apply only those in the
>> server role.
>>
>> Is two roles of hooks something it makes sense to introduce?  Or could
>> we clarify the current situation, document it?
>>
>> It looks to me like "enforcement of the standard svn client's rules" is
>> an option we would ideally like to provide to server operators.  How
>> would this be defined more precisely?  How should we design it to cope
>> with different client versions, whose rules have changed a few times?
>
> You're pulling this way out of scope.

I'm unpicking a load of unspoken assumptions.

> Issue SVN-4065 is very concrete
> and specific, and a concrete proposal was made on how best to solve
> it.
>
> You're using it to open a much broader architectural discussion. Which
> is fine of course, and I think quite interesting. But that shouldn't
> block progress for SVN-4065 in the way which was proposed 7 years ago,
> and which still seems the best way (via a utility program that can be
> invoked from hooks, where our hooks still have the same semantics /
> confusions / shortcomings / ... as today).

OK, so a valid option is to treat this as "just another pre-commit
hook".  If the suggestion was just to write and supply another hook
script, it would be reasonable to ignore all that: we don't need to
think about the system semantics, as it's just like any other pre-commit
hook.

The talk about building the required hook functionality into a new
compiled program led to me wondering more widely about the design.  If
we're going to combine multiple "standard" hooks into one configurable
program, that sort of thing does need to be discussed.

It seems to me that combining multiple hooks into one configurable
program is orthogonal to the actual issue.  The only reason it came up,
AFAICT, is because of possibly premature concern about efficiency.


> That being said, maybe that svnhooks utility program (Ivan's proposal)
> could be used to introduce a way to separate those different roles
> that hooks serve. Ivan hinted a bit in that direction in his post
> there. From https://svn.haxx.se/dev/archive-2012-12/0217.shtml:
>
>> svnhooks configuration file has separate section for each policy. For example:
>> [[[
>> [check-eol-normalization]
>> enable = yes
>>
>> [rev-propchange]
>> enable = yes
>> users = svnsync
>>
>> [edit-log-message]
>> enable = yes
>> users = admin, @author
>> ]]]
>
> I agree this should be thought through, with a good design (a "users"
> field might not cut it -- perhaps something else). It's clear that the
> proposal needs further design work.

Thanks.

I didn't mean to prevent a simple task from being done.

- Julian


Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Julian Foad-5
I suggest there are three issues mixed up here:

* [SVN-4065] Redefine as:
   "Write a pre-commit hook to reject non-conforming eol-style=native."
   Put it in 'tools/hook-scripts/'.

* [new issue]
   Review/organize the set of recommended/suggested hooks.
   See 'tools/hook-scripts/'.  See what categories there are, e.g.
   enforcing standard svn client behaviours (such as eol-style=native).
   What can we do to make them easier to choose, configure, deploy?
   Should some be withdrawn?

* [new issue]
   Reconsider the idea of a configurable multi-hooks program.
   I think that is not as obvious as it might look in that
   old email thread and needs wider discussion.

The redefined SVN-4065 could be a suitable hackathon task.

- Julian
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Daniel Shahaf-2
Julian Foad wrote on Wed, 09 Oct 2019 15:13 +00:00:
> * [SVN-4065] Redefine as:
>    "Write a pre-commit hook to reject non-conforming eol-style=native."
>    Put it in 'tools/hook-scripts/'.
...
> The redefined SVN-4065 could be a suitable hackathon task.

Isn't writing `svnlook changed -t $TXN | cut -c 5- | while read -r line;
do if svnlook propget $REPOS svn:eol-style $line | grep -qx native &&
svnlook cat -t $TXN $REPOS $line | xxd -p -c 1 | grep -qxi 0d; then
exit 1; fi; done` is too small a task for a hackathon?
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Daniel Shahaf-2
In reply to this post by Branko Čibej
Branko Čibej wrote on Wed, Oct 09, 2019 at 15:45:32 +0200:
> On 09.10.2019 15:00, Johan Corveleyn wrote:
> > During that entire discussion thread the only objections raised were
> > about "enforcing it in the repos layer / server directly". No-one
> > objected to the proposal(s) to solve the issue via pre-commit hooks.
>
> Validating contents (such as line endings based on svn:eol-style) is a
> perfect fit for hooks. Not normalising of course. :)

The repos layer validates svn:* revprops in svn_repos__validate_prop().

The FS layer doesn't do that validation.

The result of that is that old repositories with invalid svn:* properties can
be dump/load'd but can't be svnsync'.d

---

As to separation of concerns, that's a valid point, but we should be consistent
about what concerns belong where.  The validation of svn:* props and of
contents of files with svn:eol-style set belongs in the same layer, doesn't it?

Therefore, I think there are two options:

- These validation belongs in the repos layer.  The simpler/lower layer that
  doesn't do such validation is the FS layer.  svn:eol-style validation belongs
  in the repos layer.  (And if svnsync needs to bypass it, we'll design a way
  for it to do so.)

- These validations belong $elsewhere.  svn:eol-style validation will be added
  $elsewhere and svn_repos__validate_prop() will be moved $elsewhere.

My interpretation: I don't have an opinion yet, except that if we move the
validation out of the repos layer, I won't be quite as sure any more what the
difference between the FS layer and the repos layer _is_.  The FS layer exposes
a filesystem that's a 0-indexed array of trees [implemented by the DAG layer].
The repos layer does… what, exactly, on top of that?

Cheers,

Daniel
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Branko Čibej
On 09.10.2019 19:14, Daniel Shahaf wrote:
> The FS layer exposes
> a filesystem that's a 0-indexed array of trees [implemented by the DAG layer].
> The repos layer does… what, exactly, on top of that?

I've been asking that from day one. :) All that API duplication, which
is incomplete to boot, is confusing.

Well, the repos layer is responsible for authorization checks, too.

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: SVN-4065 - server should enforce LF normalization for svn:eol-style=native

Johan Corveleyn-3
In reply to this post by Daniel Shahaf-2
On Wed, Oct 9, 2019 at 7:14 PM Daniel Shahaf <[hidden email]> wrote:

>
> Branko Čibej wrote on Wed, Oct 09, 2019 at 15:45:32 +0200:
> > On 09.10.2019 15:00, Johan Corveleyn wrote:
> > > During that entire discussion thread the only objections raised were
> > > about "enforcing it in the repos layer / server directly". No-one
> > > objected to the proposal(s) to solve the issue via pre-commit hooks.
> >
> > Validating contents (such as line endings based on svn:eol-style) is a
> > perfect fit for hooks. Not normalising of course. :)
>
> The repos layer validates svn:* revprops in svn_repos__validate_prop().
>
> The FS layer doesn't do that validation.
>
> The result of that is that old repositories with invalid svn:* properties can
> be dump/load'd but can't be svnsync'.d
>
> ---
>
> As to separation of concerns, that's a valid point, but we should be consistent
> about what concerns belong where.  The validation of svn:* props and of
> contents of files with svn:eol-style set belongs in the same layer, doesn't it?
>
> Therefore, I think there are two options:
>
> - These validation belongs in the repos layer.  The simpler/lower layer that
>   doesn't do such validation is the FS layer.  svn:eol-style validation belongs
>   in the repos layer.  (And if svnsync needs to bypass it, we'll design a way
>   for it to do so.)
>
> - These validations belong $elsewhere.  svn:eol-style validation will be added
>   $elsewhere and svn_repos__validate_prop() will be moved $elsewhere.
>
> My interpretation: I don't have an opinion yet, except that if we move the
> validation out of the repos layer, I won't be quite as sure any more what the
> difference between the FS layer and the repos layer _is_.  The FS layer exposes
> a filesystem that's a 0-indexed array of trees [implemented by the DAG layer].
> The repos layer does… what, exactly, on top of that?

Thinking about it again some more, I guess I agree that this really
belongs in the same place as svn_repos__validate_prop(), and really
should be in the repos layer (or in any case, more widely /
automatically / standardly available than just a hook script, where it
depends too much on the individual sysadmin).

In fact, back in 2012 I wasn't happy with the consensus gravitating
towards "solve this in a pre-commit hook" (except as a stop-gap
measure). But the "standard pre-commit utility" seemed to be the best
attainable idea that would receive support from most devs, at the
time.

Earlier in the current thread Brane said:
"Validating contents (such as line endings based on svn:eol-style) is
a perfect fit for hooks."

But I think this kind of content validation is different from
"application-level content validation", like "follow coding style X",
or "commit message should contain an issue key", or "I want to enforce
that .c files have svn:eol-style=native". Such application-level
content rules have no effect whatsoever on the workings of SVN itself.
SVN clients don't care about those, only the applications (IDE, users,
...) do.

In contrast, having a "non-LF-normalized with svn:eol-style=native" in
the repository breaks "svn diff" (all lines shown as different) and
"svn status" (if timestamps mismatch, contents are checksummed, and
the file shows up as modified while it isn't -- causing confusion and
even worse, spurious tree conflicts).

So in my eyes this is more of an obligatory validation, to preserve
the sanity of your "svn ecosystem". A bit like the sha1 collision
protection (yes, the repository can store sha1 collisions perfectly
fine, but they wreak havoc amongst your clients / working copies if
you let them enter your repository). The "havoc" caused by an SVN-4065
file is orders of magnitude less severe than what a sha1 collision
causes (entire working copy hosed), but still, it's not nice.

(Julian: sorry for my frustrated reaction earlier, and thanks for
staying constructive, as always ... you were right, the discussion was
clearly not over yet :-))

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: PMCs: any Hackathon requests? (deadline 11 October)

Nathan Hartman
In reply to this post by Daniel Shahaf-2
On Wed, Oct 2, 2019 at 5:43 PM Sally Khudairi <[hidden email]> wrote:

> Hello PMCs --I hope you are well.
>
> We received an email from hackCBS 2.0, a hackathon that will be taking place 19-20 October at the University of Delhi with 700+ students.
>
> They are interested in our participation by providing a list of tasks from various Apache projects for them to work on.
>
> If you would like to submit a list of work areas, please let me know your interest and forward your list(s) no later than 11 October.
>
> Many thanks,
> Sally
>
> - - -
> Vice President Marketing & Publicity
> Vice President Sponsor Relations
> The Apache Software Foundation
>
> Tel +1 617 921 8656 | [hidden email]

We are basically out of time.

Unless there are any reasonable objections, I propose to reply to
Sally's email with the following.

I've taken into account everything that was discussed previously,
incorporated Johan's and Daniel's suggested tasks, removed the ones
that Julian pointed out are a bad idea, need review, or are pending a
design, etc.

Please speak up with any improvements as soon as possible!

If I hear nothing to the contrary, I'll assume silence = agreement
and reply to Sally (with CC to this list) at 12:00 AM GMT.

Proposed message follows:

[[[

Dear Sally,

We have opportunities for the 2-day hackathon across several skill set
areas including: web design, documentation, swig bindings, Python, and
C programming:


* Web design on https://subversion.apache.org:

  - Incorporate normalize.css and main.css from html5boilerplate.com.

  - Improve the navigation bar to make the static pages mobile
    friendly, while keeping the site in a form that can be
    hand-edited.

* Documentation:

  - SVN-3914 - INSTALL: document how to compile with libmagic on
    Windows.
    https://issues.apache.org/jira/browse/SVN-3914?issueNumber=3914

* Python programming:

  - Implement linking with libmagic on Windows in the build scripts.

* Swig bindings:

  - SVN-4781 - expose svn_fs_change_rev_prop2() to swig.
    https://issues.apache.org/jira/browse/SVN-4781?issueNumber=4781

  - SVN-4568 - expose svn_fs_set_warning_func() to swig.
    https://issues.apache.org/jira/browse/SVN-4568?issueNumber=4568

* C programming:

  - SVN-4343 - FSFS backend work: "svnadmin verify" should verify
    changed-paths list.
    https://issues.apache.org/jira/browse/SVN-4343?issueNumber=4343

  - SVN-4605 - "svnadmin verify" doesn't verify locks.
    https://issues.apache.org/jira/browse/SVN-4605?issueNumber=4605


Hackathon participants should join our 'dev' mailing list and introduce
themselves. To join, email [hidden email] -- see
https://subversion.apache.org/mailing-lists.html for details.

We'll be happy to provide whatever guidance we can, as well as review
and (hopefully) apply some quality patches!

If you have any questions, please let us know by emailing
[hidden email]!

Kind regards,
The Apache Subversion PMC

]]]
1234 ... 8