Repository navigation
Docker integration tests, and the database bugs they found - #9
Merged
Merged
Conversation
- mysql_storage.sql: TYPE=MyISAM was removed in MySQL 5.5 (ENGINE=), and '0000-00-00' datetime defaults are rejected by 5.7+'s default strict mode, so the shipped schema did not load. It now uses InnoDB and utf8mb4 (MyISAM's 1000-byte key limit is too small for utf8mb4 varchar(255) indexes). - 'COUNT(*) AS check': CHECK is a reserved word since MySQL 8.0.16, so group_exists() -- called by almost every command -- was a syntax error.
MySQLdb starts with autocommit off. On the MyISAM tables Papercut was written for that didn't matter, but on InnoDB (the default since MySQL 5.5, and what most Phorum/phpBB databases use today) posted articles were never committed, and their locks blocked other connections. Connections also didn't set a character set, so non-ASCII text came back garbled (mojibake).
The rows from fetchall() were formatted whole, producing message-ids like <(2,)@group> instead of <2@group>.
smtplib's sendmail() only accepts ASCII strings, so a post containing e.g. 'é' raised UnicodeEncodeError after the article was saved: the client was told '441 Posting failed' for a post that went through, and no moderator or mailing-list email was sent. The messages are now built with email.message.EmailMessage (RFC 2047 headers, UTF-8 body).
- get_STAT() returned cursor.execute()'s result, which is always None with psycopg2, so every article was reported missing (STAT, and the server's article lookups) - NEXT/LAST used MySQL's 'LIMIT 1, 0', which is a syntax error in PostgreSQL (and would select no rows in MySQL) - new-thread posts used CURRENT_TIMESTAMP(), invalid in PostgreSQL - the auth module ignored dbhost and dbpass, so it only worked over a local socket without a password
- The queries mixed comma joins with LEFT JOIN ('FROM posts A, posts_text
B LEFT JOIN users C ON A.poster_id ...'). Since MySQL 5.0.12 JOIN binds
tighter than the comma, so A isn't visible in the ON clause and ARTICLE,
HEAD, XOVER, XHDR and XPAT all failed. The comma joins are now
parenthesized.
- cursor.insert_id() only existed in very old MySQLdb versions, so every
post failed; it is cursor.lastrowid now.
XOVER used the topic id as the message number of the article a post replies to, so threads were mixed up (it only looked right for topics whose id happened to equal their first post's id). It now references the topic's first post, as HEAD already did.
NEWNEWS takes a wildmat (e.g. comp.*), but the server refused anything except a bare '*' that wasn't an existing group name with 411. The mysql and Phorum backends then compared the pattern with '=' instead of LIKE, so it would not have matched anyway.
docs/integration-testing-plan.md: docker-compose.test.yml runs MySQL 8.4 (and optionally 5.7) and PostgreSQL 16, with fixture schemas and data for the mysql, Phorum, phpBB, PHP-Nuke and PostNuke backends (loading the repo's own schema and patch scripts), Phorum config files and an SMTP sink. 114 tests drive the real server with nntplib, covering reading, posting (including moderator/mailing-list email), logins and injection payloads. Also a CI job and 'make test-integration'. The bugs these found are fixed in the preceding commits.
On Python 3.13+ nntplib comes from standard-nntplib, which only the Docker test image installs, so the unit test jobs failed while collecting tests/integration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Docker integration testing plan, the last step of both test plans. Stacked on #8.
docker-compose.test.ymlruns MySQL 8.4 (plus optional 5.7) and PostgreSQL 16, loads fixtures for every SQL backend using the repo's own schema and patch scripts, and runs 114 integration tests. They drive the real server withnntplib: reading, posting (including Phorum's moderator and mailing-list emails, via an SMTP sink), logins, and injection payloads. A new CI job runs them on every PR.Bugs found and fixed (one commit each)
mysqlmysql_storage.sqldidn't load on MySQL 5.5+;COUNT(*) AS checkis a syntax error on 8.0.16+, which broke almost every command<(2,)@group>; NEWNEWS refused wildmats likecomp.*UnicodeEncodeErrorwhile emailing moderators, after saving, so the client was told the post failedphorum_pgsqlLIMIT 1, 0andCURRENT_TIMESTAMP()are invalid in PostgreSQL; the auth module ignoreddbhost/dbpassLEFT JOINbroke ARTICLE/HEAD/XOVER/XHDR/XPAT on MySQL 5+;cursor.insert_id()made every post fail; XOVER mixed up threadsTesting
docker compose -f docker-compose.test.yml run --rm papercut: 481 passed (all unit and integration tests), 5 skipped, 1 xfailed.--profile mysql57): 88 passed.pytestlocally (no Docker): 367 passed.