Add vacuum support to hissqlite-util and ovsqlite-util

doc/pod/hissqlite-util.pod

@@ -4,20 +4,26 @@ hissqlite-util - History manipulation utility for hissqlite
 
 =head1 SYNOPSIS
 
-B<hissqlite-util> [B<-AcdhsV>] [B<-O> I<order>] [B<-p> I<path>]
+B<hissqlite-util> [B<-AcdhsvV>] [B<-O> I<order>] [B<-p> I<path>]
 
 =head1 DESCRIPTION
 
 B<hissqlite-util> is an administrative interface to the hissqlite history
 method for INN.  It only works on hissqlite history databases, not on any
 other type of INN history.  It allows the administrator to audit the
-database for problems, dump its contents, and report entry counts and the
-schema version.  It can also report database storage utilization.
+database for problems, dump its contents, report entry counts and the
+schema version, report database storage utilization, and vacuum the
+database to reclaim free pages.
 
-All of its actions are read-only, so it is safe to run while INN is running.
-B<hissqlite-util> opens the database directly; because B<innd> holds the
-write lock for only a single statement at a time, it waits at most briefly
-for any in-progress write.
+All actions except B<-v> are read-only, so it is safe to run them while
+INN is running.  B<hissqlite-util> opens the database directly; because
+B<innd> holds the write lock for only a single statement at a time, it
+waits at most briefly for any in-progress write.
+
+The B<-v> option rewrites the database file (SQLite C<VACUUM>) and needs
+write access.  When B<innd> is running, B<hissqlite-util> pauses it for the
+duration of the vacuum and resumes it afterwards.  When the news system is
+stopped, the vacuum runs offline.
 
 =head1 REQUIREMENTS
 
@@ -87,10 +93,34 @@ The database-derived values are collected from one consistent read snapshot.
 Logical size is the page count multiplied by the page size.  Non-freelist
 page capacity is an estimate based on SQLite's page accounting, not a measure
 of how full each page is.  Freelist pages can be reused by later writes or
-recovered with a full C<VACUUM>.  File sizes are point-in-time values and may
-change while B<innd> or another process is writing.  Files that do not
-currently exist, such as a WAL or rollback journal, have a reported size of
-zero.
+recovered with a full C<VACUUM> (see B<-v>).  File sizes are point-in-time
+values and may change while B<innd> or another process is writing.  Files
+that do not currently exist, such as a WAL or rollback journal, have a
+reported size of zero.
+
+=item B<-v>
+
+Vacuum the history database: rewrite it with SQLite C<VACUUM> so freelist
+pages are removed and the file shrinks toward its logical size.  Prints
+page and freelist statistics before and after the rewrite.
+
+A vacuum needs free disk space roughly equal to the current database file
+size (SQLite builds a compact copy, then replaces the original).  On a
+large history this can take a long time and will block other writers; when
+B<innd> is running, it is paused for the duration.  Readers may block while
+the exclusive rewrite runs.
+
+Vacuuming is not needed in normal operation.  SQLite keeps deleted pages on
+a freelist and reuses them for later inserts, so a steady-state spool that
+expires roughly as much as it receives tends to stabilize near a high-water
+mark size.  That high-water mark is usually desirable: the file does not
+churn on every expire, and subsequent growth can reuse free pages without
+extending the file.  Run B<-v> only when you deliberately want a smaller
+file, for example after a large permanent reduction in retention (so the
+spool will not grow back to the old size), after a bulk delete, or when
+changing the page size (which requires a vacuum; see hissqlite(5)).
+
+See also L</VACUUMING> in hissqlite(5).
 
 =item B<-V>
 
@@ -116,6 +146,10 @@ Print storage and entry statistics:
 
     hissqlite-util -s
 
+After shrinking retention and letting expire finish, reclaim freelist space:
+
+    hissqlite-util -v
+
 =head1 HISTORY
 
 Written by Kevin Bowling for InterNetNews.

doc/pod/hissqlite.pod

@@ -146,6 +146,60 @@ that reads a I<separate> source history, which must not be written while it
 runs; see hissqlite-convert(8).)
 
 C<hissqlite-util -s> reports database storage and entry statistics.
+C<hissqlite-util -v> vacuums the database when freelist space should be
+reclaimed; see L</VACUUMING>.
+
+=head1 VACUUMING
+
+hissqlite does B<not> vacuum the database automatically, and that is
+intentional.
+
+When rows are deleted (for example when B<expire> drops remembered
+Message-IDs past I</remember/>), SQLite does not usually shrink the file.
+It places free pages on a freelist and reuses them for later inserts.  For a
+news spool that receives and expires at a steady rate, the database therefore
+grows to a high-water mark and then stays there: freelist pages absorb the
+churn of expire and intake without constantly extending or truncating the
+file.  That behaviour is efficient.  Keeping the high-water mark avoids
+repeated filesystem allocations, keeps more of the B-tree's working set in a
+stable file layout, and means expire does not pay the cost of a full rewrite.
+
+A full C<VACUUM> rewrites the entire database into a compact copy and
+replaces the original.  It reclaims freelist space (and can apply a new page
+size if C<PRAGMA page_size> was set beforehand), but it needs free disk space
+on the order of the current file size, takes time proportional to the
+database size, and needs exclusive write access while it runs.  Automatic
+vacuuming after every expire would therefore be expensive and usually
+pointless: the next day of traffic would grow the file back toward the same
+high-water mark.
+
+Vacuum manually with B<hissqlite-util> B<-v> only when the smaller size is
+actually wanted for the long term, for example:
+
+=over 2
+
+=item *
+
+After a large, permanent reduction in retention (so the spool will not grow
+back to the old size).
+
+=item *
+
+After a bulk delete or migration that left a large freelist (check with
+B<hissqlite-util> B<-s>; freelist share is reported there).
+
+=item *
+
+When changing I<hissqlitepagesize> on an existing database.  The page size is
+applied only when the database is created; changing it later requires setting
+C<PRAGMA page_size> and vacuuming (or rebuilding with B<makehistory> /
+B<hissqlite-convert>).  Prefer matching the page size to the filesystem block
+or ZFS I<recordsize> from the start; see I<hissqlitepagesize> above.
+
+=back
+
+In normal operation, leave freelist pages alone and treat the high-water mark
+as the steady-state size of an expiring spool.
 
 =head1 EXPIRATION
 

doc/pod/news.pod

@@ -65,7 +65,9 @@ The new hissqlite history storage method is accompanied with a history
 manipulation utility, B<hissqlite-util>, also contributed by Kevin Bowling,
 that performs some basic consistency checks and dump operations on an history
 database using the hissqlite method.  It can also report SQLite storage
-utilization, history entry counts and file sizes.
+utilization, history entry counts and file sizes, and vacuum the database
+with B<-v> to reclaim freelist space when needed (not required in normal
+steady-state operation; see hissqlite(5)).
 
 =item *
 
@@ -123,6 +125,14 @@ counts.  Thanks to Kevin Bowling for this enhancement.
 
 =item *
 
+A new B<-v> option in B<ovsqlite-util> runs SQLite C<VACUUM> to reclaim
+freelist pages after large permanent shrinks of retention or bulk deletes.
+Vacuuming is not automatic and is usually unnecessary for a steady-state
+expiring spool (the freelist high-water mark is left in place on purpose);
+see ovsqlite(5).  Thanks to Kevin Bowling for this enhancement.
+
+=item *
+
 B<innreport> now recognizes the logs of cleanfeed-ng, the maintained and
 lightweight continuation of the historical Cleanfeed filter.  A dedicated
 section is also generated in daily Usenet reports to summarize audit and

doc/pod/ovsqlite-util.pod

@@ -6,7 +6,7 @@ ovsqlite-util - Overview manipulation utility for ovsqlite
 
 =head1 SYNOPSIS
 
-B<ovsqlite-util> [B<-AFghioOs>] [B<-a> I<article>] [B<-n> I<newsgroup>]
+B<ovsqlite-util> [B<-AFghioOsv>] [B<-a> I<article>] [B<-n> I<newsgroup>]
 [B<-p> I<path>]
 
 =head1 DESCRIPTION
@@ -16,7 +16,8 @@ method for INN.  It only works on ovsqlite overview databases, not on any
 other type of INN overview.  (See dedicated ovdb_stat(8) and tdx-util(8)
 programs for ovdb and tradindexed overview methods.)  It allows the
 administrator to dump various information about the internal state of the
-overview, audit it for errors, and fix these errors.
+overview, audit it for errors, fix these errors, and vacuum the database
+to reclaim free pages.
 
 To audit the entire overview database for problems, use B<-A>.  The checks
 can take an extended period of time, depending on the number of newsgroups
@@ -26,7 +27,9 @@ reported to standard error.  Use B<-F> to correct the errors found.
 It is safe to run this utility while the server is running.  In case fixes
 should be made to the database (when using the B<-F> flag), B<ovsqlite-util>
 will properly pause the server before performing its operations, and then
-resume it.
+resume it.  The B<-v> (vacuum) flag also pauses a running B<innd> when
+possible; for a large vacuum it is better to stop the news system so
+B<ovsqlite-server> and readers are not holding the database open.
 
 The number of overview records stored in the database can be obtained with the
 C<inndf -n> command.
@@ -164,10 +167,38 @@ Database-derived values are collected from one consistent read snapshot.
 The logical database size is the page count multiplied by the page size.
 Non-freelist page capacity is an estimate derived from SQLite's page
 accounting, not a measure of how full each page is.  Freelist pages can be
-reused by later writes or recovered with a full C<VACUUM>.  File sizes are
-point-in-time values and may change while the server is writing.  Files that
-do not currently exist, such as a WAL or rollback journal, have a reported
-size of zero.
+reused by later writes or recovered with a full C<VACUUM> (see B<-v>).  File
+sizes are point-in-time values and may change while the server is writing.
+Files that do not currently exist, such as a WAL or rollback journal, have a
+reported size of zero.
+
+=item B<-v>
+
+Vacuum the overview database: rewrite it with SQLite C<VACUUM> so freelist
+pages are removed and the file shrinks toward its logical size.  Prints
+page and freelist statistics before and after the rewrite.  It cannot be
+combined with B<-n> or B<-a>.
+
+A vacuum needs free disk space roughly equal to the current database file
+size (SQLite builds a compact copy, then replaces the original).  On a
+large overview this can take a long time and needs exclusive access to the
+database.  When B<innd> is running, it is paused for the duration.  For a
+reliable vacuum on a busy server, stop the news system with B<rc.news stop>
+first so B<ovsqlite-server> and B<nnrpd> are not holding connections open;
+otherwise C<VACUUM> waits on the busy timeout until other connections
+release their locks.
+
+Vacuuming is not needed in normal operation.  SQLite keeps deleted pages on
+a freelist and reuses them for later inserts, so a steady-state spool that
+expires roughly as much as it receives tends to stabilize near a high-water
+mark size.  That high-water mark is usually desirable: the file does not
+churn on every expireover run, and subsequent growth can reuse free pages
+without extending the file.  Run B<-v> only when you deliberately want a
+smaller file, for example after a large permanent reduction in retention,
+after purging many groups, or when changing the page size of an existing
+database (which requires a vacuum; see ovsqlite(5)).
+
+See also L</VACUUMING> in ovsqlite(5).
 
 =back
 
@@ -202,12 +233,17 @@ Print storage and record statistics:
 
     ovsqlite-util -s
 
+After shrinking retention and letting expireover finish, reclaim freelist
+space (preferably with the news system stopped):
+
+    ovsqlite-util -v
+
 =head1 HISTORY
 
 Written by Julien ÉLIE for InterNetNews.
 
 =head1 SEE ALSO
 
-expireover(8), inndf(8), makehistory(8), nnrpd(8), ovsqlite(5).
+expireover(8), inndf(8), makehistory(8), nnrpd(8), ovsqlite(5), rc.news(8).
 
 =cut

doc/pod/ovsqlite.pod

@@ -181,6 +181,64 @@ for read operations.  This eliminates the IPC round-trip and server
 serialization bottleneck, allowing read performance to scale with the
 number of concurrent readers.
 
+=head1 VACUUMING
+
+ovsqlite does B<not> vacuum the database automatically, and that is
+intentional.
+
+When overview rows are deleted (for example by B<expireover>), SQLite does
+not usually shrink F<ovsqlite.db>.  It places free pages on a freelist and
+reuses them for later inserts.  For a news spool that receives and expires
+at a steady rate, the overview database therefore grows to a high-water mark
+and then stays there: freelist pages absorb the churn of expireover and
+intake without constantly extending or truncating the file.  That behaviour
+is efficient.  Keeping the high-water mark avoids repeated filesystem
+allocations and means expireover does not pay the cost of a full rewrite on
+every run.
+
+A full C<VACUUM> rewrites the entire database into a compact copy and
+replaces the original.  It reclaims freelist space (and can apply a new page
+size if C<PRAGMA page_size> was set beforehand), but it needs free disk space
+on the order of the current file size, takes time proportional to the
+database size, and needs exclusive access while it runs.  Automatic
+vacuuming after every expireover would therefore be expensive and usually
+pointless: the next day of traffic would grow the file back toward the same
+high-water mark.
+
+Vacuum manually with B<ovsqlite-util> B<-v> only when the smaller size is
+actually wanted for the long term, for example:
+
+=over 2
+
+=item *
+
+After a large, permanent reduction in retention (so the overview will not
+grow back to the old size).
+
+=item *
+
+After purging many newsgroups or other bulk deletes that left a large
+freelist (check with B<ovsqlite-util> B<-s>; freelist share is reported
+there).
+
+=item *
+
+When changing I<pagesize> on an existing database.  The I<pagesize>
+parameter in F<ovsqlite.conf> is consulted only when creating a new database.
+To change the page size of an existing overview, stop the news system, set
+C<PRAGMA page_size> to the desired value, and vacuum (B<ovsqlite-util> B<-v>
+reclaims freelist space; applying a new page size can be done with the
+B<sqlite3> CLI as C<PRAGMA page_size=N; VACUUM;> on the stopped database),
+or rebuild overview with B<makehistory>.  Prefer choosing an appropriate
+page size when the database is first created.
+
+=back
+
+For a large vacuum, stop the news system with B<rc.news stop> so
+B<ovsqlite-server> and B<nnrpd> are not holding the database open.  In normal
+operation, leave freelist pages alone and treat the high-water mark as the
+steady-state size of an expiring spool.
+
 =head1 HISTORY
 
 Initial implementation of ovsqlite written by Bo Lindbergh