Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Paul Hammant-3
OK, so most of ye know I am on this quest.

Context: I've a working merkle-ish client tracking changes in a Svn server over DAV, and noting when files directories change and replicating that change on the client side. For files it gets the SHA1 for the file, stores it, and updates local file copies when the SHA1 changes. For directories there's not SHA1 (yet), and it tracks directory revisions. They'll change too fast, so I need to server-side create the SHA1s for directories and cache them appropriately.  

I know how to calculate the (presently missing) SHA1s for directories in Svn. I even know how to do that for different auth settings for different users safely.

Intention:

I want to mount (over DAV only) '.indexSHA1s.csv' (or something similar) in every directory, but I don't want those checked in (race condition avoidance, and commit noise).  Thus, while those files are visible over DAV, subversion does not know about them, even if they do follow the ACLs apparent in the authz files. Specifically, if you are not permitted to read <root>/foo/bar/ then you cannot read <root>/foo/bar/.indexSHA1s.csv either.

Question:

What Apache2 technique to need to read about, and find docker-style examples for to facilitate this? There's a huge amount of Apache modules, and I can't glean whether Mod_filter or any Apache module is the right tool for the job.  Or even if this isn't delivered in modules at all (build in).

More info: I've prototyped my Merkle hash calc (and recalc) logic in Rust, with tests of course. I may end up writing an apache module for this (and donating back) but right now I'm happy with any Apache2-centric technique (regardless of speed or hackyness) that can acclimate me to augmenting the DAV available Svn representation with files that are hot-calculated (and LRU cached).

Can someone point me in the right direction?

- Paul
Reply | Threaded
Open this post in threaded view
|

Re: Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Paul Hammant-3
"Two docroots" might be what I'm looking for:  http://madbean.com/2007/two-docroots/ 

The author, Matt Quail, has been Jira lead for a decade, I think.
Reply | Threaded
Open this post in threaded view
|

Re: Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Thomas Åkesson-2
Yes, I was also about to suggest mod_rewrite. Should be possible to match on your secret hidden filename and return content from a “shadow” directory structure that only contains those files.

You could generate those files in the shadow structure on the post-commit event.

I have rewrites within the mod_dav_svn location in order to provide additional REST functionality on svn paths.

Merkle-tree sounds interesting, let us know about your progress.

/Thomas Å.


Reply | Threaded
Open this post in threaded view
|

Re: Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Paul Hammant-3
Yup. I'm able to go down a rabbit hole on things, though. I've just spent an hour trying to find a docker image to start with so that I was able to ship something that's easily run as an an example for this group. Who knew there's a few thousand ways of configuring Apache in docker images, heh?
Reply | Threaded
Open this post in threaded view
|

Re: Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Paul Hammant-3
Well the code from that blog entry works as reported for <directory> declarations but not for <location> ones where mod_dav_svn is the handler.  I recreate both of those in a single server implementation if I mount the a svn/ folder inside the canonical docroot (as is common), then play with URLs that should be rewritten in the browser (or Wget) that are inside and outside of that svn/ sub directory.

For extra fun I moved the four rewrite lines before, inside, and after the Svn <location>, but it didn't make any difference.

I'll try a straight handler next, in the hope that it can wholly intercept requests that would otherwise be routed to mod_dav_svn.

- Paul

 
Reply | Threaded
Open this post in threaded view
|

Re: Augmenting the WebDAV side of Subversion with merkle hashes for directories.

Paul Hammant-3
OK, not a straight handler, but 'ScriptAliasMatch'. Like so:

ScriptAliasMatch \.foo*$ "/path/to/helloworld.py$1"

I should be able to hit /a/b/d/e/f.foo in a browser and have helloworld.py execute, right?

Well, it works just fine when declared at root level (not in a <directory> or <location> directive) and for when the request URL is anything outside the Subversion mounted dir/location:

But when I change the URL to be inside the Svn location, then mod_dav_svn intercepts the request and reports 404 - the wildcard on .foo suffixed URLs is ignored.

Specifically:

   http://myserver/svn/anthing.foo  --> does not work 

Unless there's a way through this (other nesting techniques), I'll have to mount the Merkle directory summaries on a parallel directory tree:

   http://myserver/svn/path/to/imminent_layoffs.doc      --> Svn via DAV 

   http://myserver/meta/path/to/.indexSHA1s.csv  --> a GET-centric app
   http://myserver/meta/path/.indexSHA1s.csv      --> a GET-centric app 
   http://myserver/meta/.indexSHA1s.csv              --> a GET-centric app

Regards,

- Paul