inodes usage of svn server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

inodes usage of svn server

Keva-Slient

inodes usage of my svn server grows up quickly  as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

is there any way storing all version files into one file?  or any other way to solve this problem.


------------------
.........
Vanity...My favourite sin.
    --<<The Devil's Advocate>>
 
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Nathan Hartman
On Jan 2, 2018, at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>
>
> inodes usage of my svn server grows up quickly  as repositories commits increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
> is there any way storing all version files into one file?  or any other way to solve this problem.

See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Nico Kadel-Garcia-2
In reply to this post by Keva-Slient
On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>
> inodes usage of my svn server grows up quickly  as repositories commits
> increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

>
> is there any way storing all version files into one file?  or any other way
> to solve this problem.
>
>
> ------------------
> .........
> Vanity...My favourite sin.
>     --<<The Devil's Advocate>>
>
Reply | Threaded
Open this post in threaded view
|

回复: inodes usage of svn server

Keva-Slient

thank you very much. that's exact what i want.
------------------
.........
Vanity...My favourite sin.
    --<<The Devil's Advocate>>
 


------------------ 原始邮件 ------------------
发件人: "Nico Kadel-Garcia";<[hidden email]>;
发送时间: 2018年1月3日(星期三) 中午11:20
收件人: "Keva-Slient"<[hidden email]>;
抄送: "users"<[hidden email]>;
主题: Re: inodes usage of svn server

On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>
> inodes usage of my svn server grows up quickly  as repositories commits
> increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

>
> is there any way storing all version files into one file?  or any other way
> to solve this problem.
>
>
> ------------------
> .........
> Vanity...My favourite sin.
>     --<<The Devil's Advocate>>
>
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Keva-Slient
In reply to this post by Nathan Hartman

thank you very much. that's what i want
------------------
.........
Vanity...My favourite sin.
    --<<The Devil's Advocate>>
 


------------------ Original ------------------
From:  "Nathan Hartman";<[hidden email]>;
Date:  Wed, Jan 3, 2018 11:08 AM
To:  "users"<[hidden email]>;
Cc:  "Keva-Slient"<[hidden email]>;
Subject:  Re: inodes usage of svn server

On Jan 2, 2018, at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>
>
> inodes usage of my svn server grows up quickly  as repositories commits increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
> is there any way storing all version files into one file?  or any other way to solve this problem.

See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Bo Berglund
In reply to this post by Nico Kadel-Garcia-2
On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia
<[hidden email]> wrote:

>On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>>
>> inodes usage of my svn server grows up quickly  as repositories commits
>> increasing.
>> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
>There is "svnadmin pack", which needs to run on the Subversion
>repository server.
>

Is this a one-time cleanup operation or does it need to be executed
regularly? I.e. does running "svnadmin pack" on the repo consolidate
the many files into a smaller number of big files and set some
repository property such that it will work that way in the future?

This is what I found in svnbook 1.7 page 178:

"By concatenating all the files of a completed
shard into a single “pack” file and then removing the original
per-revision files, svnadmin pack reduces the file count within a
given shard down to just a single file."

and

"Repacking packed shards is legal, but will have no effect on the disk
usage of the repository."

What exactly is a "shard"?
Is it one of the numerical directories each containing exactly 1000
files?

If so the "shard" storage on my repos seem to only cost at most 4%
extra disk space compared to using a single file (comparing file sizes
against disk usage).

Or will compression of the "svnadmin pack" operation make the combined
size of the files even smaller? I.e. it does not only save unused file
allocation space but actually packs the content even better?



--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Daniel Shahaf-2
Bo Berglund wrote on Wed, 03 Jan 2018 07:50 +0100:
> Is this a one-time cleanup operation or does it need to be executed
> regularly?

Regularly.

> I.e. does running "svnadmin pack" on the repo consolidate
> the many files into a smaller number of big files and set some
> repository property such that it will work that way in the future?

The former.

> This is what I found in svnbook 1.7 page 178:
>
> "By concatenating all the files of a completed
> shard into a single “pack” file and then removing the original
> per-revision files, svnadmin pack reduces the file count within a
> given shard down to just a single file."
>
> and
>
> "Repacking packed shards is legal, but will have no effect on the disk
> usage of the repository."
>
> What exactly is a "shard"?
> Is it one of the numerical directories each containing exactly 1000
> files?
>

Yes.

> If so the "shard" storage on my repos seem to only cost at most 4%
> extra disk space compared to using a single file (comparing file sizes
> against disk usage).

The concern was inode usage, not disk space usage.  Packing reduces the
number of inodes by 99.8%.

> Or will compression of the "svnadmin pack" operation make the combined
> size of the files even smaller? I.e. it does not only save unused file
> allocation space but actually packs the content even better?

'svnadmin pack' neither recompresses nor redeltifies.  It just
concatenates the files.
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Nico Kadel-Garcia-2
In reply to this post by Bo Berglund
On Wed, Jan 3, 2018 at 1:50 AM, Bo Berglund <[hidden email]> wrote:

> On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia
> <[hidden email]> wrote:
>
>>On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <[hidden email]> wrote:
>>>
>>> inodes usage of my svn server grows up quickly  as repositories commits
>>> increasing.
>>> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>>
>>There is "svnadmin pack", which needs to run on the Subversion
>>repository server.
>>
>
> Is this a one-time cleanup operation or does it need to be executed
> regularly? I.e. does running "svnadmin pack" on the repo consolidate
> the many files into a smaller number of big files and set some
> repository property such that it will work that way in the future?
>
> This is what I found in svnbook 1.7 page 178:
>
> "By concatenating all the files of a completed
> shard into a single “pack” file and then removing the original
> per-revision files, svnadmin pack reduces the file count within a
> given shard down to just a single file."
>
> and

Disk space != inodes. You asked about inodes. File systems have a
maximum number of inodes, the maximum number of files directories,
links, etc. that they know how to count for that filesystem at the
time the filesystem is built. Storing information about files takes
space on a filesystem, sot it's notrmally pre-allocated when making
the filesystem. From your own description, you've run out of inodes
and need to free some up, or rebuild your environment to better handle
structures that handle many small files.

> "Repacking packed shards is legal, but will have no effect on the disk
> usage of the repository."
>
> What exactly is a "shard"?
> Is it one of the numerical directories each containing exactly 1000
> files?

It's one of the files in those directories.

> If so the "shard" storage on my repos seem to only cost at most 4%
> extra disk space compared to using a single file (comparing file sizes
> against disk usage).

> Or will compression of the "svnadmin pack" operation make the combined
> size of the files even smaller? I.e. it does not only save unused file
> allocation space but actually packs the content even better?

Please stop confusing "inode" with "disk space" It's more like you've
run out of boxes to put things in, i.e. inodes, than running out of
space to store store the boxes (i.e., disk space).

> --
> Bo Berglund
> Developer in Sweden
>
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Bo Berglund
On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia
<[hidden email]> wrote:

>Please stop confusing "inode" with "disk space" It's more like you've
>run out of boxes to put things in, i.e. inodes, than running out of
>space to store store the boxes (i.e., disk space).

Please do not confuse me with the original poster "Keva-Slient"....
I merely stepped in to clarify if I might also have the problem, which
he apparently is having.
But I do not think so (I am on Windows Server 2016 with an NTFS file
system).


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Nico Kadel-Garcia-2
On Wed, Jan 3, 2018 at 11:54 AM, Bo Berglund <[hidden email]> wrote:

> On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia
> <[hidden email]> wrote:
>
>>Please stop confusing "inode" with "disk space" It's more like you've
>>run out of boxes to put things in, i.e. inodes, than running out of
>>space to store store the boxes (i.e., disk space).
>
> Please do not confuse me with the original poster "Keva-Slient"....
> I merely stepped in to clarify if I might also have the problem, which
> he apparently is having.
> But I do not think so (I am on Windows Server 2016 with an NTFS file
> system).

Sorry about the quotation confusion.

NTFS also has limits on the maximum number of files for a filesystem.
There is no complete escape from the underlying limitations of
trade-offs of easy processing versus performance versus capacity that
lead to these limitations. There are also notable performance limits
on having too many files in a directory. We could have a long
discussion about the fundamentals: suffice to say that organizing a
bunch of associated bits into a single file, rather than scattering
many such bits around a filesystem and overpopulating a directory or a
filesystem is not your friend.

> --
> Bo Berglund
> Developer in Sweden
>
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Dave Huang-2
On 1/3/2018 13:19, Nico Kadel-Garcia wrote:
> NTFS also has limits on the maximum number of files for a filesystem.

FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)

> There are also notable performance limits on having too many files in
> a directory.

I don't know enough about NTFS internals to say whether that's the case
with NTFS or not, but in the context of this discussion, the default SVN
shard size is 1000 revisions, which I don't think could be considered
"too many files in a directory".

While I don't have any actual numbers, my gut feeling is that packing
the repo isn't really needed on NTFS in terms of day to day performance
of clients using the repo, or in terms of filesystem limitations. That
said, I do it anyway because backing up the repo does go faster when
dealing with a couple dozen large files vs. tens of thousands of small
files.

--
Name: Dave Huang         |  Mammal, mammal / their names are called /
INet: [hidden email] |  they raise a paw / the bat, the cat /
Telegram: @dahanc        |  dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 42 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+
PL++
Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Branko Čibej
On 03.01.2018 21:13, Dave Huang wrote:

> On 1/3/2018 13:19, Nico Kadel-Garcia wrote:
>> NTFS also has limits on the maximum number of files for a filesystem.
>
> FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
> Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)
>
>> There are also notable performance limits on having too many files in
>> a directory.
>
> I don't know enough about NTFS internals to say whether that's the
> case with NTFS or not, but in the context of this discussion, the
> default SVN shard size is 1000 revisions, which I don't think could be
> considered "too many files in a directory".
>
> While I don't have any actual numbers, my gut feeling is that packing
> the repo isn't really needed on NTFS in terms of day to day
> performance of clients using the repo, or in terms of filesystem
> limitations. That said, I do it anyway because backing up the repo
> does go faster when dealing with a couple dozen large files vs. tens
> of thousands of small files.

And there's your answer to the question of day to day performance of the
repository: Subversion also has to open many vs. just one file when its
reading historical revisions, so packing will definitely help to reduce
the number of directory lookups and file opens (the latter are
notoriously slow on Windows). Depending on usage patterns, the
performance boost may be significant.

-- Brane




Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Bo Berglund
On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej <[hidden email]>
wrote:

>And there's your answer to the question of day to day performance of the
>repository: Subversion also has to open many vs. just one file when its
>reading historical revisions, so packing will definitely help to reduce
>the number of directory lookups and file opens (the latter are
>notoriously slow on Windows). Depending on usage patterns, the
>performance boost may be significant.

THat seems to be a valid cause for using the function then!
I will look at my new converted repository in view of this info.
Thanks!


--
Bo Berglund
Developer in Sweden

Reply | Threaded
Open this post in threaded view
|

Re: inodes usage of svn server

Branko Čibej
On 03.01.2018 22:04, Bo Berglund wrote:

> On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej <[hidden email]>
> wrote:
>
>> And there's your answer to the question of day to day performance of the
>> repository: Subversion also has to open many vs. just one file when its
>> reading historical revisions, so packing will definitely help to reduce
>> the number of directory lookups and file opens (the latter are
>> notoriously slow on Windows). Depending on usage patterns, the
>> performance boost may be significant.
> THat seems to be a valid cause for using the function then!
> I will look at my new converted repository in view of this info.
> Thanks!

It's safe to create a post-commit hook that runs 'svnadmin pack'
asynchrouously, or schedule a task that does that (hourly or daily,
depending on your needs). The operation is cheap if it has nothing to do.

-- Brane

Reply | Threaded
Open this post in threaded view
|

svn over apache2 logging problem

Keva-Slient
In reply to this post by Nico Kadel-Garcia-2
here is my config in dav_svn.conf:
<Location /svn>                                                                                                                                           
    DAV svn                                                                                                                                               
    SVNParentPath /wns/data/svn                                                                                                                 
    SVNListParentPath On                                                                                                                             
    AuthType Basic                                                                                                                                     
    AuthName "Subversion Repository"                                                                                                         
    AuthUserFile /wns/etc/apache2/dav_svn.passwd                                                                                       
    Require valid-user                                                                                                                         
</Location>                                                                                                                                                                                                                 #CustomLog /var/log/apache2/svn_logfile "%{%Y-%m-%d %T}t %u@%h %>s repo:%{SVN-REPOS-NAME}e %{SVN-ACTION}e (%B Bytes in %T Sec)" env=SVN-ACTION                                                      
CustomLog /var/log/apache2/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION    


all of the log settings are not working.
it only creates log file. there is not any content written when i commit , checkout or update.
why is this?


     
------------------
.........
Vanity...My favourite sin.
    --<<The Devil's Advocate>>