[PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

Evgeny Kotkov
Branko Čibej <[hidden email]> writes:

> Still missing is a mechanism for the libsvn_wc (and possibly
> libsvn_client) to determine the capabilities of the working copy at
> runtime (this will be needed for deciding whether to use compressed
> pristines).

FWIW, I tried the idea of using LZ4 to compress the pristines and storing small
pristines as blobs in the `PRISTINE` table.  I was particularly interested in
how such change would affect the performance and what kind of obstacles
would have to be dealt with.

In the attachment you will find a more or less functional implementation of
this idea that might be useful to some extent.  The patch is a proof of
concept: it doesn't include the WC compatibility bits and most certainly
doesn't have everything necessary in place.  But in the meanwhile, I think
that is might give a good approximation of what can be expected from the
approach.

The patch applies to the `better-pristines` branch.

A couple of observations:

 - As expected, the combined size of the pristines is halved when the data
   itself is compressible, thus making the working copy 25% smaller.

 - A variety of the callers currently access the pristine contents by reading
   the corresponding files.  That doesn't work in case of compressed pristines
   or pristines stored as BLOBs.

   I think that ideally we would want to use streams as much as possible, and
   only spill the uncompressed pristine contents to temporary files when we
   need to pass them to external tools, etc.; and that temporary files need
   to be backed by a work queue to avoid leaving them in place in case of an
   application crash.

   The patch does that kind of plumbing to some extent, but that part of the
   work is not complete.  The starting point is around wc_db_pristine.c:
   svn_wc__db_pristine_get_path().

 - Using BLOBs to store the pristine contents didn't have a measurable impact
   on the speed of the WC operations such as checkout in my experiments on
   Windows.  These experiments were not comprehensive, and also I didn't run
   the tests on *nix.

 - There's also the deprecated svn_wc_get_pristine_copy_path() public API that
   would require plumbing to maintain compatibility; the patch performs it by
   spilling the pristine contents result into a temporary file whose lifetime
   is attached to the `result_pool`.

 (I probably won't be able to continue the work on this patch in the nearby
 future; posting this in case it might be useful.)


Thanks,
Evgeny Kotkov

proof-of-concept-pristines-lz4+blob-v1.patch.txt (81K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

Branko Čibej
On 22.10.2018 22:14, Evgeny Kotkov wrote:

> Branko Čibej <[hidden email]> writes:
>
>> Still missing is a mechanism for the libsvn_wc (and possibly
>> libsvn_client) to determine the capabilities of the working copy at
>> runtime (this will be needed for deciding whether to use compressed
>> pristines).
> FWIW, I tried the idea of using LZ4 to compress the pristines and storing small
> pristines as blobs in the `PRISTINE` table.  I was particularly interested in
> how such change would affect the performance and what kind of obstacles
> would have to be dealt with.

Nice! I did some simpler tests by compressing exported trees, but this
is definitely better.

> In the attachment you will find a more or less functional implementation of
> this idea that might be useful to some extent.  The patch is a proof of
> concept: it doesn't include the WC compatibility bits and most certainly
> doesn't have everything necessary in place.  But in the meanwhile, I think
> that is might give a good approximation of what can be expected from the
> approach.
>
> The patch applies to the `better-pristines` branch.
>
> A couple of observations:
>
>  - As expected, the combined size of the pristines is halved when the data
>    itself is compressible, thus making the working copy 25% smaller.

Yes, that was my observation as well. In fact, though, storing small
BLOBs in the database itself should have even better effects, since the
space on disk actually used by a file is rounded up to the nearest
cluster size, but SQLite's blocks are typically much smaller than that.


>  - A variety of the callers currently access the pristine contents by reading
>    the corresponding files.  That doesn't work in case of compressed pristines
>    or pristines stored as BLOBs.
>
>    I think that ideally we would want to use streams as much as possible, and
>    only spill the uncompressed pristine contents to temporary files when we
>    need to pass them to external tools, etc.; and that temporary files need
>    to be backed by a work queue to avoid leaving them in place in case of an
>    application crash.

Yes and yes. Keeping those temporary spilled files on disk could turn
out to be a problem, finding a reasonable time to delete them without
having to run cleanup will be rather important, I think.


>    The patch does that kind of plumbing to some extent, but that part of the
>    work is not complete.  The starting point is around wc_db_pristine.c:
>    svn_wc__db_pristine_get_path().
>
>  - Using BLOBs to store the pristine contents didn't have a measurable impact
>    on the speed of the WC operations such as checkout in my experiments on
>    Windows.  These experiments were not comprehensive, and also I didn't run
>    the tests on *nix.

I wouldn't expect much change in performance but would expect better use
of the disk, as explained above.

>  - There's also the deprecated svn_wc_get_pristine_copy_path() public API that
>    would require plumbing to maintain compatibility; the patch performs it by
>    spilling the pristine contents result into a temporary file whose lifetime
>    is attached to the `result_pool`.

Ack; that's one reasonable definition of "lifetime." But I suspect that
any users of that function expect the pristine file to survive at least
to the next WC cleanup.

>  (I probably won't be able to continue the work on this patch in the nearby
>  future; posting this in case it might be useful.)

Thanks, it definitely is useful!

-- Brane

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

Bert Huijben-5
On Windows' NTFS implementation very small files (probably something like < 256 bytes, but this is not documented/strictly stable) are stored in the directory table and so don't use 'a whole cluster'.

Nice work on all the research!

    Bert

On Tue, Oct 23, 2018 at 6:12 PM, Branko Čibej <[hidden email]> wrote:
On 22.10.2018 22:14, Evgeny Kotkov wrote:
> Branko Čibej <[hidden email]> writes:
>
>> Still missing is a mechanism for the libsvn_wc (and possibly
>> libsvn_client) to determine the capabilities of the working copy at
>> runtime (this will be needed for deciding whether to use compressed
>> pristines).
> FWIW, I tried the idea of using LZ4 to compress the pristines and storing small
> pristines as blobs in the `PRISTINE` table.  I was particularly interested in
> how such change would affect the performance and what kind of obstacles
> would have to be dealt with.

Nice! I did some simpler tests by compressing exported trees, but this
is definitely better.

> In the attachment you will find a more or less functional implementation of
> this idea that might be useful to some extent.  The patch is a proof of
> concept: it doesn't include the WC compatibility bits and most certainly
> doesn't have everything necessary in place.  But in the meanwhile, I think
> that is might give a good approximation of what can be expected from the
> approach.
>
> The patch applies to the `better-pristines` branch.
>
> A couple of observations:
>
>  - As expected, the combined size of the pristines is halved when the data
>    itself is compressible, thus making the working copy 25% smaller.

Yes, that was my observation as well. In fact, though, storing small
BLOBs in the database itself should have even better effects, since the
space on disk actually used by a file is rounded up to the nearest
cluster size, but SQLite's blocks are typically much smaller than that.


>  - A variety of the callers currently access the pristine contents by reading
>    the corresponding files.  That doesn't work in case of compressed pristines
>    or pristines stored as BLOBs.
>
>    I think that ideally we would want to use streams as much as possible, and
>    only spill the uncompressed pristine contents to temporary files when we
>    need to pass them to external tools, etc.; and that temporary files need
>    to be backed by a work queue to avoid leaving them in place in case of an
>    application crash.

Yes and yes. Keeping those temporary spilled files on disk could turn
out to be a problem, finding a reasonable time to delete them without
having to run cleanup will be rather important, I think.


>    The patch does that kind of plumbing to some extent, but that part of the
>    work is not complete.  The starting point is around wc_db_pristine.c:
>    svn_wc__db_pristine_get_path().
>
>  - Using BLOBs to store the pristine contents didn't have a measurable impact
>    on the speed of the WC operations such as checkout in my experiments on
>    Windows.  These experiments were not comprehensive, and also I didn't run
>    the tests on *nix.

I wouldn't expect much change in performance but would expect better use
of the disk, as explained above.

>  - There's also the deprecated svn_wc_get_pristine_copy_path() public API that
>    would require plumbing to maintain compatibility; the patch performs it by
>    spilling the pristine contents result into a temporary file whose lifetime
>    is attached to the `result_pool`.

Ack; that's one reasonable definition of "lifetime." But I suspect that
any users of that function expect the pristine file to survive at least
to the next WC cleanup.

>  (I probably won't be able to continue the work on this patch in the nearby
>  future; posting this in case it might be useful.)

Thanks, it definitely is useful!

-- Brane


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

Branko Čibej
On 29.10.2018 14:27, Bert Huijben wrote:
> On Windows' NTFS implementation very small files (probably something
> like < 256 bytes, but this is not documented/strictly stable) are
> stored in the directory table and so don't use 'a whole cluster'.

Right — and this is exactly the kind of optimisation we're aiming for by
putting small(ish) pristine blobs directly into the working copy database.

-- Brane

Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Proof of concept of the better-pristines (LZ4 + storing small pristines as BLOBs) (Was: Re: svn commit: r1843076)

Branko Čibej
In reply to this post by Evgeny Kotkov
On 22.10.2018 22:14, Evgeny Kotkov wrote:

> @@ -104,9 +103,13 @@
>  CREATE TABLE PRISTINE (
> ··/*·Alternative·MD5·checksum·used·for·communicating·with·older
> ·····repositories.·Not·strictly·guaranteed·to·be·unique·among·table·rows.·*/
>
> -··md5_checksum··TEXT·NOT·NULL
> +··md5_checksum··TEXT·NOT·NULL,
> +
> +··data··BLOB
> ··);
> +/*·TODO:·While·here,·perhaps·add·`WITHOUT·ROWID`·for·the·new·schema·table?·*/
>
> +

Note that this is no longer how schema changes will work. Instead, we
create the oldest supported schema (currently 1.8) and then roll forward
to the desired format.

We could probably add WITHOUT ROWID anyway, but that will only affect
new working copies, not upgrades of existing ones.

-- Brane