diff --git a/doc/pod/hissqlite-util.pod b/doc/pod/hissqlite-util.pod index 7fb3713fb..0b2f15b70 100644 --- a/doc/pod/hissqlite-util.pod +++ b/doc/pod/hissqlite-util.pod @@ -4,20 +4,26 @@ hissqlite-util - History manipulation utility for hissqlite =head1 SYNOPSIS -B [B<-AcdhsV>] [B<-O> I] [B<-p> I] +B [B<-AcdhsvV>] [B<-O> I] [B<-p> I] =head1 DESCRIPTION B is an administrative interface to the hissqlite history method for INN. It only works on hissqlite history databases, not on any other type of INN history. It allows the administrator to audit the -database for problems, dump its contents, and report entry counts and the -schema version. It can also report database storage utilization. +database for problems, dump its contents, report entry counts and the +schema version, report database storage utilization, and vacuum the +database to reclaim free pages. -All of its actions are read-only, so it is safe to run while INN is running. -B opens the database directly; because B holds the -write lock for only a single statement at a time, it waits at most briefly -for any in-progress write. +All actions except B<-v> are read-only, so it is safe to run them while +INN is running. B opens the database directly; because +B holds the write lock for only a single statement at a time, it +waits at most briefly for any in-progress write. + +The B<-v> option rewrites the database file (SQLite C) and needs +write access. When B is running, B pauses it for the +duration of the vacuum and resumes it afterwards. When the news system is +stopped, the vacuum runs offline. =head1 REQUIREMENTS @@ -87,10 +93,34 @@ The database-derived values are collected from one consistent read snapshot. Logical size is the page count multiplied by the page size. Non-freelist page capacity is an estimate based on SQLite's page accounting, not a measure of how full each page is. Freelist pages can be reused by later writes or -recovered with a full C. File sizes are point-in-time values and may -change while B or another process is writing. Files that do not -currently exist, such as a WAL or rollback journal, have a reported size of -zero. +recovered with a full C (see B<-v>). File sizes are point-in-time +values and may change while B or another process is writing. Files +that do not currently exist, such as a WAL or rollback journal, have a +reported size of zero. + +=item B<-v> + +Vacuum the history database: rewrite it with SQLite C so freelist +pages are removed and the file shrinks toward its logical size. Prints +page and freelist statistics before and after the rewrite. + +A vacuum needs free disk space roughly equal to the current database file +size (SQLite builds a compact copy, then replaces the original). On a +large history this can take a long time and will block other writers; when +B is running, it is paused for the duration. Readers may block while +the exclusive rewrite runs. + +Vacuuming is not needed in normal operation. SQLite keeps deleted pages on +a freelist and reuses them for later inserts, so a steady-state spool that +expires roughly as much as it receives tends to stabilize near a high-water +mark size. That high-water mark is usually desirable: the file does not +churn on every expire, and subsequent growth can reuse free pages without +extending the file. Run B<-v> only when you deliberately want a smaller +file, for example after a large permanent reduction in retention (so the +spool will not grow back to the old size), after a bulk delete, or when +changing the page size (which requires a vacuum; see hissqlite(5)). + +See also L in hissqlite(5). =item B<-V> @@ -116,6 +146,10 @@ Print storage and entry statistics: hissqlite-util -s +After shrinking retention and letting expire finish, reclaim freelist space: + + hissqlite-util -v + =head1 HISTORY Written by Kevin Bowling for InterNetNews. diff --git a/doc/pod/hissqlite.pod b/doc/pod/hissqlite.pod index bb62d87ee..7ab8bd124 100644 --- a/doc/pod/hissqlite.pod +++ b/doc/pod/hissqlite.pod @@ -146,6 +146,60 @@ that reads a I source history, which must not be written while it runs; see hissqlite-convert(8).) C reports database storage and entry statistics. +C vacuums the database when freelist space should be +reclaimed; see L. + +=head1 VACUUMING + +hissqlite does B vacuum the database automatically, and that is +intentional. + +When rows are deleted (for example when B drops remembered +Message-IDs past I), SQLite does not usually shrink the file. +It places free pages on a freelist and reuses them for later inserts. For a +news spool that receives and expires at a steady rate, the database therefore +grows to a high-water mark and then stays there: freelist pages absorb the +churn of expire and intake without constantly extending or truncating the +file. That behaviour is efficient. Keeping the high-water mark avoids +repeated filesystem allocations, keeps more of the B-tree's working set in a +stable file layout, and means expire does not pay the cost of a full rewrite. + +A full C rewrites the entire database into a compact copy and +replaces the original. It reclaims freelist space (and can apply a new page +size if C was set beforehand), but it needs free disk space +on the order of the current file size, takes time proportional to the +database size, and needs exclusive write access while it runs. Automatic +vacuuming after every expire would therefore be expensive and usually +pointless: the next day of traffic would grow the file back toward the same +high-water mark. + +Vacuum manually with B B<-v> only when the smaller size is +actually wanted for the long term, for example: + +=over 2 + +=item * + +After a large, permanent reduction in retention (so the spool will not grow +back to the old size). + +=item * + +After a bulk delete or migration that left a large freelist (check with +B B<-s>; freelist share is reported there). + +=item * + +When changing I on an existing database. The page size is +applied only when the database is created; changing it later requires setting +C and vacuuming (or rebuilding with B / +B). Prefer matching the page size to the filesystem block +or ZFS I from the start; see I above. + +=back + +In normal operation, leave freelist pages alone and treat the high-water mark +as the steady-state size of an expiring spool. =head1 EXPIRATION diff --git a/doc/pod/news.pod b/doc/pod/news.pod index 0d450adda..1e6da75ec 100644 --- a/doc/pod/news.pod +++ b/doc/pod/news.pod @@ -65,7 +65,9 @@ The new hissqlite history storage method is accompanied with a history manipulation utility, B, also contributed by Kevin Bowling, that performs some basic consistency checks and dump operations on an history database using the hissqlite method. It can also report SQLite storage -utilization, history entry counts and file sizes. +utilization, history entry counts and file sizes, and vacuum the database +with B<-v> to reclaim freelist space when needed (not required in normal +steady-state operation; see hissqlite(5)). =item * @@ -123,6 +125,14 @@ counts. Thanks to Kevin Bowling for this enhancement. =item * +A new B<-v> option in B runs SQLite C to reclaim +freelist pages after large permanent shrinks of retention or bulk deletes. +Vacuuming is not automatic and is usually unnecessary for a steady-state +expiring spool (the freelist high-water mark is left in place on purpose); +see ovsqlite(5). Thanks to Kevin Bowling for this enhancement. + +=item * + B now recognizes the logs of cleanfeed-ng, the maintained and lightweight continuation of the historical Cleanfeed filter. A dedicated section is also generated in daily Usenet reports to summarize audit and diff --git a/doc/pod/ovsqlite-util.pod b/doc/pod/ovsqlite-util.pod index 8f0f7dfe5..6372634e4 100644 --- a/doc/pod/ovsqlite-util.pod +++ b/doc/pod/ovsqlite-util.pod @@ -6,7 +6,7 @@ ovsqlite-util - Overview manipulation utility for ovsqlite =head1 SYNOPSIS -B [B<-AFghioOs>] [B<-a> I
] [B<-n> I] +B [B<-AFghioOsv>] [B<-a> I
] [B<-n> I] [B<-p> I] =head1 DESCRIPTION @@ -16,7 +16,8 @@ method for INN. It only works on ovsqlite overview databases, not on any other type of INN overview. (See dedicated ovdb_stat(8) and tdx-util(8) programs for ovdb and tradindexed overview methods.) It allows the administrator to dump various information about the internal state of the -overview, audit it for errors, and fix these errors. +overview, audit it for errors, fix these errors, and vacuum the database +to reclaim free pages. To audit the entire overview database for problems, use B<-A>. The checks can take an extended period of time, depending on the number of newsgroups @@ -26,7 +27,9 @@ reported to standard error. Use B<-F> to correct the errors found. It is safe to run this utility while the server is running. In case fixes should be made to the database (when using the B<-F> flag), B will properly pause the server before performing its operations, and then -resume it. +resume it. The B<-v> (vacuum) flag also pauses a running B when +possible; for a large vacuum it is better to stop the news system so +B and readers are not holding the database open. The number of overview records stored in the database can be obtained with the C command. @@ -164,10 +167,38 @@ Database-derived values are collected from one consistent read snapshot. The logical database size is the page count multiplied by the page size. Non-freelist page capacity is an estimate derived from SQLite's page accounting, not a measure of how full each page is. Freelist pages can be -reused by later writes or recovered with a full C. File sizes are -point-in-time values and may change while the server is writing. Files that -do not currently exist, such as a WAL or rollback journal, have a reported -size of zero. +reused by later writes or recovered with a full C (see B<-v>). File +sizes are point-in-time values and may change while the server is writing. +Files that do not currently exist, such as a WAL or rollback journal, have a +reported size of zero. + +=item B<-v> + +Vacuum the overview database: rewrite it with SQLite C so freelist +pages are removed and the file shrinks toward its logical size. Prints +page and freelist statistics before and after the rewrite. It cannot be +combined with B<-n> or B<-a>. + +A vacuum needs free disk space roughly equal to the current database file +size (SQLite builds a compact copy, then replaces the original). On a +large overview this can take a long time and needs exclusive access to the +database. When B is running, it is paused for the duration. For a +reliable vacuum on a busy server, stop the news system with B +first so B and B are not holding connections open; +otherwise C waits on the busy timeout until other connections +release their locks. + +Vacuuming is not needed in normal operation. SQLite keeps deleted pages on +a freelist and reuses them for later inserts, so a steady-state spool that +expires roughly as much as it receives tends to stabilize near a high-water +mark size. That high-water mark is usually desirable: the file does not +churn on every expireover run, and subsequent growth can reuse free pages +without extending the file. Run B<-v> only when you deliberately want a +smaller file, for example after a large permanent reduction in retention, +after purging many groups, or when changing the page size of an existing +database (which requires a vacuum; see ovsqlite(5)). + +See also L in ovsqlite(5). =back @@ -202,12 +233,17 @@ Print storage and record statistics: ovsqlite-util -s +After shrinking retention and letting expireover finish, reclaim freelist +space (preferably with the news system stopped): + + ovsqlite-util -v + =head1 HISTORY Written by Julien ÉLIE for InterNetNews. =head1 SEE ALSO -expireover(8), inndf(8), makehistory(8), nnrpd(8), ovsqlite(5). +expireover(8), inndf(8), makehistory(8), nnrpd(8), ovsqlite(5), rc.news(8). =cut diff --git a/doc/pod/ovsqlite.pod b/doc/pod/ovsqlite.pod index ab0aedda9..8267feb08 100644 --- a/doc/pod/ovsqlite.pod +++ b/doc/pod/ovsqlite.pod @@ -181,6 +181,64 @@ for read operations. This eliminates the IPC round-trip and server serialization bottleneck, allowing read performance to scale with the number of concurrent readers. +=head1 VACUUMING + +ovsqlite does B vacuum the database automatically, and that is +intentional. + +When overview rows are deleted (for example by B), SQLite does +not usually shrink F. It places free pages on a freelist and +reuses them for later inserts. For a news spool that receives and expires +at a steady rate, the overview database therefore grows to a high-water mark +and then stays there: freelist pages absorb the churn of expireover and +intake without constantly extending or truncating the file. That behaviour +is efficient. Keeping the high-water mark avoids repeated filesystem +allocations and means expireover does not pay the cost of a full rewrite on +every run. + +A full C rewrites the entire database into a compact copy and +replaces the original. It reclaims freelist space (and can apply a new page +size if C was set beforehand), but it needs free disk space +on the order of the current file size, takes time proportional to the +database size, and needs exclusive access while it runs. Automatic +vacuuming after every expireover would therefore be expensive and usually +pointless: the next day of traffic would grow the file back toward the same +high-water mark. + +Vacuum manually with B B<-v> only when the smaller size is +actually wanted for the long term, for example: + +=over 2 + +=item * + +After a large, permanent reduction in retention (so the overview will not +grow back to the old size). + +=item * + +After purging many newsgroups or other bulk deletes that left a large +freelist (check with B B<-s>; freelist share is reported +there). + +=item * + +When changing I on an existing database. The I +parameter in F is consulted only when creating a new database. +To change the page size of an existing overview, stop the news system, set +C to the desired value, and vacuum (B B<-v> +reclaims freelist space; applying a new page size can be done with the +B CLI as C on the stopped database), +or rebuild overview with B. Prefer choosing an appropriate +page size when the database is first created. + +=back + +For a large vacuum, stop the news system with B so +B and B are not holding the database open. In normal +operation, leave freelist pages alone and treat the high-water mark as the +steady-state size of an expiring spool. + =head1 HISTORY Initial implementation of ovsqlite written by Bo Lindbergh diff --git a/history/hissqlite/hissqlite-util.in b/history/hissqlite/hissqlite-util.in index 9964726b7..5f3e371eb 100644 --- a/history/hissqlite/hissqlite-util.in +++ b/history/hissqlite/hissqlite-util.in @@ -14,7 +14,7 @@ use POSIX qw(strftime); $0 =~ s!.*/!!; # Version of this tool. -my $VERSION = "1.2"; +my $VERSION = "1.3"; # Bail out if the needed DBI Perl module is not installed. eval { @@ -29,7 +29,7 @@ eval { my $dbfile = "history.sqlite"; my $usage = "Usage: - $0 [-AcdhsV] [-O order] [-p path] + $0 [-AcdhsvV] [-O order] [-p path] Options: -A Audit the history database for problems, and report them to @@ -43,6 +43,7 @@ Options: -p path Read $dbfile database file in path directory instead of default pathhistory directory as set in inn.conf. -s Print database storage and history entry statistics. + -v Vacuum the database (rewrite to reclaim freelist space). -V Print the version of this tool and the database schema version. "; @@ -53,7 +54,7 @@ sub HELP_MESSAGE { } my %opt; -getopts("AcdhO:p:sV", \%opt) || die $usage; +getopts("AcdhO:p:svV", \%opt) || die $usage; HELP_MESSAGE() if defined($opt{'h'}); @@ -62,6 +63,7 @@ $modes++ if defined($opt{'A'}); $modes++ if defined($opt{'c'}); $modes++ if defined($opt{'d'}); $modes++ if defined($opt{'s'}); +$modes++ if defined($opt{'v'}); $modes++ if defined($opt{'V'}); die "Only one action allowed at the same time\n\n$usage" @@ -82,25 +84,29 @@ my $dbdir; } my $dbpath = "$dbdir/$dbfile"; my $datasource = "dbi:SQLite:dbname=$dbpath"; +my $pausemsg = "hissqlite-util vacuum"; +my $ispaused = 0; -# All implemented modes are read-only, so open the database read-only. +# Read-only modes open the database read-only. Vacuum needs write access. # SQLITE_OPEN_READONLY is 0x00000001. my $SQLITE_OPEN_READONLY = 0x00000001; +my $need_write = defined($opt{'v'}); # Open the connection. The username and password fields are left empty. # Enabling RaiseError permits not checking every return error codes. -my $dbh = DBI->connect( - $datasource, '', '', - { - PrintError => 0, - RaiseError => 1, - AutoCommit => 1, - sqlite_open_flags => $SQLITE_OPEN_READONLY, - }, +my %connect_attr = ( + PrintError => 0, + RaiseError => 1, + AutoCommit => 1, ); +$connect_attr{sqlite_open_flags} = $SQLITE_OPEN_READONLY + if not $need_write; + +my $dbh = DBI->connect($datasource, '', '', \%connect_attr); # Extra safety net against any accidental write in the read-only modes. -$dbh->do("pragma query_only = 1;"); +$dbh->do("pragma query_only = 1;") + if not $need_write; # Raise the busy timeout value to a huge number of milliseconds to prevent # timeouts because of database locks while innd holds the writer. @@ -114,6 +120,9 @@ if (defined($opt{'V'})) { dump_history(); } elsif (defined($opt{'s'})) { dump_statistics(); +} elsif (defined($opt{'v'})) { + pause_if_running(); + vacuum_database(); } else { # Default action: audit. audit_history(); @@ -124,6 +133,17 @@ $dbh->disconnect; exit(0); +END { + # In case we bail out while being paused, make sure that the show goes on! + if ($ispaused) { + if (system "$INN::Config::newsbin/ctlinnd -s go '$pausemsg'") { + die "$0: failed to resume INN with " + . "\"ctlinnd -s go '$pausemsg'\" command " + . "=> please check why and *manually* resume it\n"; + } + } +} + # Return the schema version stored in the misc table, or undef if absent. sub get_version { my $statement = $dbh->prepare("select value from misc where key = ?;"); @@ -218,6 +238,81 @@ sub file_size { return defined($size) ? $size : 0; } +# Collect page and freelist statistics for vacuum before/after reporting. +sub page_stats { + my ($page_size) = $dbh->selectrow_array("pragma page_size;"); + my ($page_count) = $dbh->selectrow_array("pragma page_count;"); + my ($freelist_count) = $dbh->selectrow_array("pragma freelist_count;"); + my $logical_size = $page_count * $page_size; + my $freelist_size = $freelist_count * $page_size; + return ($page_size, $page_count, $freelist_count, $logical_size, + $freelist_size); +} + +# Pause a running innd so it is not writing during VACUUM. If innd is not +# running (or ctlinnd is unavailable, as when this script is run from the +# source tree with -p), proceed offline. +sub pause_if_running { + my $ctlinnd; + { + no warnings 'once'; ## no critic (ProhibitNoWarnings) + $ctlinnd = defined($INN::Config::newsbin) + ? "$INN::Config::newsbin/ctlinnd" + : undef; + } + return if !defined($ctlinnd) || !-x $ctlinnd; + + # ctlinnd mode fails when innd is not running. + return if system("$ctlinnd -s mode >/dev/null 2>&1") != 0; + + if (system("$ctlinnd -s pause '$pausemsg'")) { + die "$0: failed to pause INN, aborting\n"; + } + $ispaused = 1; +} + +# Rewrite the database to reclaim freelist pages (-v option). +sub vacuum_database { + my ( + $page_size, $page_count, $freelist_count, $logical_size, + $freelist_size, + ) = page_stats(); + my $file_size = file_size($dbpath); + + print "Database file: $dbpath\n"; + print "Page size: $page_size bytes\n"; + print "Before vacuum:\n"; + print " Page count: $page_count\n"; + print " Freelist pages: $freelist_count\n"; + print " Logical database size: $logical_size bytes\n"; + print " Freelist size: $freelist_size bytes\n"; + print " Database file size: $file_size bytes\n"; + + if ($freelist_count == 0) { + print "No freelist pages to reclaim; vacuuming anyway to " + . "compact the file.\n"; + } + + # VACUUM rewrites the whole database. Free disk space roughly equal to + # the current file size is required for the temporary copy. + print "Running VACUUM (this may take a while)...\n"; + $dbh->do("vacuum;"); + + ( + $page_size, $page_count, $freelist_count, $logical_size, + $freelist_size, + ) = page_stats(); + $file_size = file_size($dbpath); + + print "After vacuum:\n"; + print " Page count: $page_count\n"; + print " Freelist pages: $freelist_count\n"; + print " Logical database size: $logical_size bytes\n"; + print " Freelist size: $freelist_size bytes\n"; + print " Database file size: $file_size bytes\n"; + print "Vacuum complete.\n"; +} + # Print database storage and entry statistics (-s). sub dump_statistics { # DBD::SQLite's begin_work uses BEGIN IMMEDIATE, which requests a write diff --git a/scripts/innreport_inn.pm b/scripts/innreport_inn.pm index 28cf71ae7..4b82add39 100644 --- a/scripts/innreport_inn.pm +++ b/scripts/innreport_inn.pm @@ -1707,8 +1707,10 @@ sub collect($$$$$$) { =~ /^python: dynamic authorization access type is not known: /o; # during daily expiration return 1 if $left =~ /^\S+ rejected Expiring process \d+$/o; - # during ovsqlite-util + # during ovsqlite-util / hissqlite-util return 1 if $left =~ /^\S+ rejected ovsqlite-util fixes$/o; + return 1 if $left =~ /^\S+ rejected ovsqlite-util vacuum$/o; + return 1 if $left =~ /^\S+ rejected hissqlite-util vacuum$/o; # during scanlogs return 1 if $left =~ /^\S+ rejected Flushing log and syslog files$/o; return 1 if $left =~ /^\S+ rejected Snapshot log and syslog files$/o; diff --git a/storage/ovsqlite/ovsqlite-util.in b/storage/ovsqlite/ovsqlite-util.in index 1fb96960b..fecd32d2c 100644 --- a/storage/ovsqlite/ovsqlite-util.in +++ b/storage/ovsqlite/ovsqlite-util.in @@ -30,7 +30,7 @@ eval { my $dbfile = "ovsqlite.db"; my $usage = "Usage: - $0 [-AFghioOs] [-a article] [-n newsgroup] [-p path] + $0 [-AFghioOsv] [-a article] [-n newsgroup] [-p path] Options: -a article Specify an article number or a range of article numbers on @@ -53,6 +53,7 @@ Options: -p path Read $dbfile database file in path directory instead of default pathoverview directory as set in inn.conf. -s Print database storage and overview record statistics. + -v Vacuum the database (rewrite to reclaim freelist space). "; sub HELP_MESSAGE { @@ -64,7 +65,7 @@ my @unique_hash_keys_unused = qw(sqlite_allow_multiple_statements); # perltidy -wuk my %opt; -getopts("a:AFghin:oOp:s", \%opt) || die $usage; +getopts("a:AFghin:oOp:sv", \%opt) || die $usage; HELP_MESSAGE() if defined($opt{'h'}); @@ -76,6 +77,7 @@ $modes++ if defined($opt{'i'}); $modes++ if defined($opt{'o'}); $modes++ if defined($opt{'O'}); $modes++ if defined($opt{'s'}); +$modes++ if defined($opt{'v'}); die "Can't use both -A and -F\n\n$usage" if defined($opt{'A'}) @@ -85,12 +87,12 @@ die "Only one action allowed at the same time\n\n$usage" die "A newsgroup must be specified with -n\n\n$usage" if (!defined($opt{'n'}) && (defined($opt{'g'}) || defined($opt{'o'}) || defined($opt{'O'}))); -die "The -n option cannot be used with -s\n\n$usage" +die "The -n option cannot be used with -s or -v\n\n$usage" if defined($opt{'n'}) - and defined($opt{'s'}); -die "The -a option cannot be used with -s\n\n$usage" + and (defined($opt{'s'}) || defined($opt{'v'})); +die "The -a option cannot be used with -s or -v\n\n$usage" if defined($opt{'a'}) - and defined($opt{'s'}); + and (defined($opt{'s'}) || defined($opt{'v'})); my ($low, $high, $compress, $basedict); my $sql_extraclause_artinfo = ""; @@ -102,8 +104,9 @@ my $dbdir; } my $dbpath = "$dbdir/$dbfile"; my $datasource = "dbi:SQLite:dbname=$dbpath"; -my $pausemsg = "ovsqlite-util fixes"; # Message when pausing INN. - # Known line in innreport. +# Pause message for -F and -v; keep the -F wording for innreport recognition. +my $pausemsg + = defined($opt{'v'}) ? "ovsqlite-util vacuum" : "ovsqlite-util fixes"; my $ispaused = 0; # To determine the length of compressed overview data. @@ -168,12 +171,15 @@ if (defined($opt{'a'})) { load_settings(); # Pause the server if changes need being done, so that the overview is not -# updated by another process at the same time. +# updated by another process at the same time. Vacuum pauses a running innd +# when possible, but also works offline (news system stopped). if (defined($opt{'F'})) { if (system "$INN::Config::newsbin/ctlinnd -s pause '$pausemsg'") { die "$0: failed to pause INN, aborting\n"; } $ispaused = 1; +} elsif (defined($opt{'v'})) { + pause_if_running(); } if (defined($opt{'g'})) { @@ -186,6 +192,8 @@ if (defined($opt{'g'})) { dump_artinfo_overchan(); } elsif (defined($opt{'s'})) { dump_statistics(); +} elsif (defined($opt{'v'})) { + vacuum_database(); } else { # Default action: audit, and also fix if -F is given. check_groupinfo_consistency(); @@ -439,6 +447,87 @@ sub file_size { return defined($size) ? $size : 0; } +# Collect page and freelist statistics for vacuum before/after reporting. +sub page_stats { + my ($page_size) = $dbh->selectrow_array("pragma page_size;"); + my ($page_count) = $dbh->selectrow_array("pragma page_count;"); + my ($freelist_count) = $dbh->selectrow_array("pragma freelist_count;"); + my $logical_size = $page_count * $page_size; + my $freelist_size = $freelist_count * $page_size; + return ($page_size, $page_count, $freelist_count, $logical_size, + $freelist_size); +} + +# Pause a running innd so it is not writing during VACUUM. If innd is not +# running (or ctlinnd is unavailable, as when this script is run from the +# source tree with -p), proceed offline. VACUUM still needs exclusive access +# to the database file, so stop ovsqlite-server (and prefer a quiet period +# with few nnrpd readers) for large vacuums; see ovsqlite-util(8). +sub pause_if_running { + my $ctlinnd; + { + no warnings 'once'; ## no critic (ProhibitNoWarnings) + $ctlinnd = defined($INN::Config::newsbin) + ? "$INN::Config::newsbin/ctlinnd" + : undef; + } + return if !defined($ctlinnd) || !-x $ctlinnd; + + # ctlinnd mode fails when innd is not running. + return if system("$ctlinnd -s mode >/dev/null 2>&1") != 0; + + if (system("$ctlinnd -s pause '$pausemsg'")) { + die "$0: failed to pause INN, aborting\n"; + } + $ispaused = 1; +} + +# Rewrite the database to reclaim freelist pages (-v option). +# VACUUM cannot run inside a transaction, so temporarily enable AutoCommit. +sub vacuum_database { + my ( + $page_size, $page_count, $freelist_count, $logical_size, + $freelist_size, + ) = page_stats(); + my $file_size = file_size($dbpath); + + print "Database file: $dbpath\n"; + print "Page size: $page_size bytes\n"; + print "Before vacuum:\n"; + print " Page count: $page_count\n"; + print " Freelist pages: $freelist_count\n"; + print " Logical database size: $logical_size bytes\n"; + print " Freelist size: $freelist_size bytes\n"; + print " Database file size: $file_size bytes\n"; + + if ($freelist_count == 0) { + print "No freelist pages to reclaim; vacuuming anyway to " + . "compact the file.\n"; + } + + # VACUUM rewrites the whole database. Free disk space roughly equal to + # the current file size is required for the temporary copy. + print "Running VACUUM (this may take a while)...\n"; + { + local $dbh->{AutoCommit} = 1; ## no critic (ProhibitLocalVars) + $dbh->do("vacuum;"); + } + + ( + $page_size, $page_count, $freelist_count, $logical_size, + $freelist_size, + ) = page_stats(); + $file_size = file_size($dbpath); + + print "After vacuum:\n"; + print " Page count: $page_count\n"; + print " Freelist pages: $freelist_count\n"; + print " Logical database size: $logical_size bytes\n"; + print " Freelist size: $freelist_size bytes\n"; + print " Database file size: $file_size bytes\n"; + print "Vacuum complete.\n"; +} + # Print database storage and overview record statistics (-s option). sub dump_statistics { $dbh->do("savepoint ovsqlite_statistics;"); diff --git a/tests/lib/hissqlite-util.t b/tests/lib/hissqlite-util.t index e99feae80..4a1d900aa 100755 --- a/tests/lib/hissqlite-util.t +++ b/tests/lib/hissqlite-util.t @@ -1,6 +1,7 @@ #! /bin/sh # -# Smoke test for hissqlite-util -s and -c against a minimal history database. +# Smoke test for hissqlite-util -s, -c, and -v against a minimal history +# database. # # Written by Kevin Bowling in 2026. @@ -39,13 +40,14 @@ if [ ! -r "$util_source" ] \ exit 0 fi -echo 5 +echo 7 tmpdir="$(pwd)/his-tmp-util" rm -rf "$tmpdir" mkdir -p "$tmpdir" -# Create a minimal hissqlite schema with one real and one remembered entry. +# Create a minimal hissqlite schema with one real and one remembered entry, +# then delete many filler rows so freelist pages exist for the vacuum test. if ! perl -MDBI -MDBD::SQLite -e ' use strict; use warnings; @@ -86,6 +88,20 @@ $dbh->do(q{ values(?, ?, ?, ?, ?) }, undef, pack("H*", "ffeeddccbbaa99887766554433221100"), 1600000001, 1600000001, 0, undef); +# Insert and delete filler rows so VACUUM has freelist pages to reclaim. +my $ins = $dbh->prepare(q{ + insert into hist(hash, arrived, posted, expires, token) + values(?, ?, ?, ?, ?) +}); +for my $i (1 .. 500) { + my $hash = pack("N", $i) . ("\0" x 12); + $ins->execute($hash, 1600001000 + $i, 1600001000 + $i, 0, undef); +} +my $keep1 = pack("H*", "00112233445566778899aabbccddeeff"); +my $keep2 = pack("H*", "ffeeddccbbaa99887766554433221100"); +$dbh->do(q{ + delete from hist where hash != ? and hash != ? +}, undef, $keep1, $keep2); $dbh->disconnect; ' "$tmpdir/history.sqlite"; then printcount "not ok" "# create test database" @@ -93,6 +109,8 @@ $dbh->disconnect; printcount "not ok" "# schema version statistic" printcount "not ok" "# entry counts" printcount "not ok" "# count command" + printcount "not ok" "# vacuum command" + printcount "not ok" "# counts after vacuum" rm -rf "$tmpdir" exit 1 fi @@ -102,6 +120,10 @@ stats_output=$(perl "$util_source" -s -p "$tmpdir" 2>&1) stats_rc=$? count_output=$(perl "$util_source" -c -p "$tmpdir" 2>&1) count_rc=$? +vac_output=$(perl "$util_source" -v -p "$tmpdir" 2>&1) +vac_rc=$? +count_after=$(perl "$util_source" -c -p "$tmpdir" 2>&1) +count_after_rc=$? if [ $stats_rc -eq 0 ]; then printcount "ok" "# statistics command" @@ -140,4 +162,25 @@ else printcount "not ok" "# count command" fi +if [ $vac_rc -eq 0 ] \ + && echo "$vac_output" | grep -q '^Vacuum complete\.$' \ + && echo "$vac_output" | grep -q '^ Freelist pages: 0$'; then + printcount "ok" "# vacuum command" +else + echo "# hissqlite-util -v failed (rc=$vac_rc):" + echo "$vac_output" | sed 's/^/# /' + printcount "not ok" "# vacuum command" +fi + +if [ $count_after_rc -eq 0 ] \ + && echo "$count_after" | grep -q '^total 2$' \ + && echo "$count_after" | grep -q '^real 1$' \ + && echo "$count_after" | grep -q '^remembered 1$'; then + printcount "ok" "# counts after vacuum" +else + echo "# counts after vacuum unexpected (rc=$count_after_rc):" + echo "$count_after" | sed 's/^/# /' + printcount "not ok" "# counts after vacuum" +fi + rm -rf "$tmpdir" diff --git a/tests/overview/ovsqlite-integ.t b/tests/overview/ovsqlite-integ.t index 9d0b9ea57..361fa2b7b 100755 --- a/tests/overview/ovsqlite-integ.t +++ b/tests/overview/ovsqlite-integ.t @@ -69,7 +69,7 @@ for prog in "$server" "$writer" "$reader"; do fi done -echo 21 +echo 23 # Set up temp directory with config files. Use absolute paths because # the C test programs chdir to the test data directory before reading @@ -165,9 +165,10 @@ else printcount "ok" "# server stopped (signal)" fi -# Tests 6-13: report database statistics with ovsqlite-util. +# Tests 6-15: report database statistics and vacuum with ovsqlite-util. # Run the source directly with -p so the test does not depend on an installed -# INN::Config and innconfval. +# INN::Config and innconfval. The server is already stopped, so vacuum can +# take an exclusive lock offline. if [ ! -r "$util_source" ] \ || ! perl -MDBI -MDBD::SQLite -MCompress::Zlib -e 1 \ >/dev/null 2>&1; then @@ -179,6 +180,8 @@ if [ ! -r "$util_source" ] \ printcount "ok" "# skip ovsqlite-util dependencies unavailable" printcount "ok" "# skip ovsqlite-util dependencies unavailable" printcount "ok" "# skip ovsqlite-util dependencies unavailable" + printcount "ok" "# skip ovsqlite-util dependencies unavailable" + printcount "ok" "# skip ovsqlite-util dependencies unavailable" else stats_output=$(perl "$util_source" -s -p "$tmpdir" 2>&1) stats_rc=$? @@ -235,9 +238,30 @@ else else printcount "not ok" "# WAL file size statistic" fi + + vac_output=$(perl "$util_source" -v -p "$tmpdir" 2>&1) + vac_rc=$? + if [ $vac_rc -eq 0 ] \ + && echo "$vac_output" | grep -q '^Vacuum complete\.$'; then + printcount "ok" "# vacuum command" + else + echo "# ovsqlite-util -v failed (rc=$vac_rc):" + echo "$vac_output" | sed 's/^/# /' + printcount "not ok" "# vacuum command" + fi + + stats_after=$(perl "$util_source" -s -p "$tmpdir" 2>&1) + if echo "$stats_after" | grep -q '^Total articles: 99$' \ + && echo "$stats_after" | grep -q '^Total groups: 11$'; then + printcount "ok" "# counts after vacuum" + else + echo "# unexpected counts after vacuum:" + echo "$stats_after" | sed 's/^/# /' + printcount "not ok" "# counts after vacuum" + fi fi -# Tests 14-18: run the reader (5 TAP tests from inside). +# Tests 16-20: run the reader (5 TAP tests from inside). reader_output=$($reader 2>&1) reader_rc=$? reader_count=$(echo "$reader_output" | grep -c "^ok ") @@ -257,7 +281,7 @@ else printcount "not ok" "# reopen" fi -# Tests 19-21: server restart resilience. +# Tests 21-23: server restart resilience. # Simulate INN restart: start a new server against the existing WAL database, # shut it down, verify the reader can still read all data. This is the # scenario where nnrpd outlives a server restart cycle.