Re: Why does NTFS compression reduces size of .pack-files by ~50 %?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: Why does NTFS compression reduces size of .pack-files by ~50 %?

Nathan Hartman
On Sat, Sep 21, 2019 at 10:15 AM Thorsten Schöning <[hidden email]> wrote:
Hi all,

I have some repo generating revisions automatically based on some
events of some software, to track some changes to some status and
config files automatically. This repos currently contains ~ 145'000
revisions and because the overhead of individual files is pretty high,
I'm packing them once a week.

That repo currently has a size of ~750 MiB on disk and I have a backup
on my Windows using NTFS. Just out of interest I enabled file system
compression for that repo and found that the storage needs could be
reduced to ~350 MiB of data.

That was pretty surprising for me, because some months ago I enabled
zlib compression level 9 and fully dumped/loaded that repo to benefit
of the compression. NTFS compression is known to prefer performance in
favour of overall savinbgs, so I would have expected to not get much
difference.

Looking at some of the pack files, there's a lot of ASCII and blocks
of binary 0s in them. Maybe NTFS compression is benefitting of those?
In general, is the data in the pack files for revs compressed or not?

From my understanding the data of the individual revs is reused and if
that is compressed, the pack files are as well. Looking at fsfs.conf,
compression for packed revprops need to be enabled additionally, but
that really only applies to the revprops, correct? Those don't count
much in my scenario.

Is there any way to further optimize/compress the pack files using SVN?

My repos are hosted under Linux using ext4, if it's normal what I see
and can't be optimized further using SVN itself, I 'm considering
switching to BTRFs with it's file system compression. Going to see the
same benefits like for NTFS pretty likely.

Thanks for your advices!

Mit freundlichen Grüßen,

Thorsten Schöning

Interesting. I'll try running some tests on my systems. It will probably take me a week or so to get to it since I have a big backlog of things at the moment.

A few questions:

Which SVN server version?

Which compression type?

Have you tried a dump and load lately to see if the resulting repository is smaller?