fix(pdf): stop re-requesting a license for a PDF Audible will not give - #1976
Merged
Conversation
…already has Reported in issue #1973: a scheduled liberate run re-requested a content license for the same 59 titles every 15 minutes, 1397 refused requests in six hours, because nothing about a failed PDF was remembered and the PDF step asked Audible afresh every time. The two download paths have been converging for a while - a PDF is now named and placed by the audiobook path's own logic, and verified like one - and every part of this bug lives where that stopped short. Both steps ask Audible for the same license. The request asks for pdf_url alongside the content reference, and LicenseInfo dropped it, so DownloadPdf turned round and requested an identical license to read the field the first response had already returned. Carry PdfUrl on LicenseInfo and give both steps one ILicensedDownload contract: a license may be supplied to a step, and the one a step ended up using is published for the next step for the same title. The CLI and the GUI queue hand it on, so a title costs one license request per run however many steps want something from it. A carried license is retried once with a fresh one if it does not work, since Audible's links are signed and a long decrypt can run between the two steps. Where the audiobook step obtained no license there is nothing to hand on and the supplement step does not run, which deletes a bug rather than guarding it: Completed fires from a finally, so a refused audio download was followed at once by a PDF request that reproduced the refusal. Error now means the same for a PDF as for a book. The audiobook step has always skipped LiberatedStatus.Error through AudioExists, and NeedsPdfDownload agrees, but DownloadPdf selected on PdfExists and so retried an errored PDF forever. A license that is granted and carries no pdf_url - the 'No PDF URL available' in the report - is Audible saying the title has no PDF, and is written off that same way instead of failing identically on every run. It stays resettable by everything that resets a book: --force, a named title, Set PDF Not Downloaded. Refusals now reach ProcessSingleAsync, which has always recorded them for whichever step throws one; DownloadPdf swallowed everything and recorded nothing. It keeps swallowing what the classifier does not recognise, which is what stopped a missing PDF from taking the app down with it. A bulk CLI run leaves alone the titles the last scan did not find, by the same Downloadable rule every multi-title path in the app already uses, and the PDF back-fill pass waits on a refused title just as the first pass does. --force and a named title still attempt everything. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A multi-title download left a waited-on title out only when it needed its audiobook, on the grounds that the audiobook download was what Audible had refused. A PDF is fetched through the same license request, so a title needing nothing but its PDF was requested again on every run of the very thing the wait exists to stop. The skip reason's wording now comes from the shared message, so the app and the CLI say the same thing about the same skip. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Only a newly imported book ever recorded a supplement, so a title that gained a PDF after its first import never got one, and a title that lost its PDF went on claiming one - which in issue #1973 is why three titles Audible has no PDF for were still being asked for. Sync from updateBook as well, and give Book set-semantics for the one supplement Audible reports per title. The duplicate guard compared the incoming url to itself, so it happened to mean 'this book already has a supplement' and a url that had changed was silently ignored. A supplement is dropped only when Audible says outright that no supplement url is available. A missing url says nothing by itself: episodes come from the catalog, which is never asked for pdf_url, so there it means 'not asked'. A PDF already downloaded is left alone either way, since the file is on disk and the library should go on saying so. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
…ules DownloadPdfLicenseTests drives the step with and without a license in hand. No Audible account exists in those tests, so a test that completes at all is the proof that the step did not go asking for a license of its own. BulkRunSelectionTests drives the real run loop against a real library database: which titles each pass reaches, that an absent title is left alone by both, that --force still attempts it, and that a refused title is waited on by the follow-up pass and reported once rather than twice. SupplementSyncTests covers the scan rules, in a new DtoImporterService.Tests project. The two replaced assertions - that a PDF-only title is never waited on, and that a --pdf run is never held back - encoded the belief that a PDF is a different request from the audiobook, which is what this branch disproves. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The refused-downloads page said the opposite - that a --pdf run is never held back because a PDF is a different request - and neither page mentioned that a bulk run now leaves alone the titles the last scan did not find, or what happens when Audible lists a PDF it will not deliver. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Counting every title the last scan did not find reports a number no run was going to act on: most of a large library's absent titles need nothing at all. The reporter's library would have been told thousands of titles were skipped where 54 would have been attempted. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A new DtoImporterService.Tests pinned MSTest 4.2.2, copied from a project file master has since moved to 4.3.3, so the merge held nine test projects at 4.3.3 and one at 4.2.2. NuGet lifted the shared MSTest.TestFramework to 4.3.3 while the 4.2.2 metapackage kept supplying the adapter, and the adapter refuses to run against a framework of a different version: 'Zero tests ran' on every platform. ApplicationServices references DtoImporterService, so its test project already sees the importers and needs no package reference of its own. One fewer project file to keep in step is the point. Co-authored-by: rmcrackan <rmcrackan@gmail.com>
…load-retry-storm-a274 Co-authored-by: rmcrackan <rmcrackan@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1973.
What the log shows
Log202608_009.logis 26scan+liberatecycles at 15-minute intervals over about six hours, against a 10,834-title library across 17 accounts.DownloadPdfattempts, all failures, over only 59 distinct titles - the same 59 every run, 24-26 times each.Skipped 6 titles that recently failed to download. Audible denied a download license: 5 (next attempt in about 10 hours). Only the PDF pass loops.No PDF URL availableand 2 x license denied.Every one of those 1,397 attempts is a full ADRM content-license request, because
GetPdfDownloadLinkAsyncreadspdf_urloff the audiobook license. That is ~230 refused license requests an hour.Why the PDF path and not the audiobook path
The two paths have been converging for a while - a PDF is now named and placed by the audiobook path's own logic (
54485c08), verified like one (537f53d8), and part of the sameliberaterun (#1947) - and every symptom here lives where that stopped short. Most tellingly, both steps ask Audible for the same license and neither knew the other did: the request already asks forpdf_url, andLicenseInfodropped it, soDownloadPdfturned round and requested an identical license to read the field the first response had returned.So each change removes a divergence rather than adding a PDF-shaped copy of audiobook machinery.
Changes
One license request per title per run.
LicenseInfocarriesPdfUrl, and both steps share oneILicensedDownloadcontract: a license may be supplied to a step, and the license a step ended up using is published for the next step for the same title. The CLI and the GUI queue hand it on. Where the audiobook step obtained no license there is nothing to hand on and the PDF step does not run - which deletes a bug rather than guarding it, sinceCompletedfires from afinally, so a refused audio download was followed at once by a PDF request that reproduced the refusal. A carried license is retried once with a fresh one if it does not work, because Audible's links are signed and a long decrypt can run between the two steps.get-licenseandliberate --licenseserializeLicenseInfo, so a saved license now carries its PDF link too.Errormeans the same for a PDF as for a book. The audiobook step has always skippedLiberatedStatus.ErrorthroughAudioExists, andNeedsPdfDownloadagrees, butDownloadPdfselected onPdfExistsand so retried an errored PDF forever. A license that is granted and carries nopdf_url- theNo PDF URL availablein the report - is Audible saying the title has no PDF, and is now written off that same way instead of failing identically every run. It stays resettable by everything that resets a book:--force, a named title, Set PDF Not Downloaded. This is the answer to the reporter's question: no, those three titles should not be advertising a PDF.One place that remembers a refusal.
ProcessSingleAsynchas always recorded refusals for whichever step throws one;DownloadPdfswallowed everything and recorded nothing. It now lets the refusalsDownloadFailureClassifierrecognises through, and keeps swallowing everything else - which is what stopped a missing PDF from taking the app down with it.liberate --pdfis no longer exempt from the wait, and the PDF back-fill pass consults it like the first pass.One definition of which titles a bulk run may attempt. A bulk CLI run leaves alone the titles the last scan did not find, by the same
Downloadablerule every multi-title path in the app already uses, and says so in one summary line.--forceand a named title still attempt everything.A book's supplement reflects the last scan. Only
addNewBookrecorded one, so a title that gained a PDF never got it and a title that lost one kept claiming it. Now synced fromupdateBooktoo, dropping a supplement only when Audible says outright that no url is available, and never for a PDF already downloaded.Verification
The reporter's scenario, reproduced without credentials
Scripts/seed-demo-accounts.csplusScripts/seed-demo-library.csproduce a library containing every class of title from the report, includingDEMO027- absent from the last scan, audio downloaded, PDF outstanding - which is the class the reporter's 54 titles belong to. The same seeded state was then run againstmaster(0934701d) and this branch.Skipped 3 titles absent from your last library scan...The four PDF attempts that disappear beyond the absent one are the requests
mastermade immediately after each audiobook download failed.cli_liberate_before_and_after.txt
liberate --forcestill attempts all three absent titles.A written-off PDF in the grid
grid_pdf_tooltip_written_off_vs_outstanding.mp4
Two adjacent rows, both showing a green lamp with a PDF badge. The first has been written off because Audible granted a license with no PDF in it; the second is simply outstanding.
Tooltip: Liberated, PDF could not be downloaded and will not be tried again
Tooltip: Liberated, PDF NOT downloaded, Audible Plus title, Click to download
The status bar counts it as unavailable rather than dropping it from every count, which is what
Errorused to do:Status bar reading PDFs: NOT d/l
Tests
1502 tests pass, on current master (which brings MSTest 4.3.3 and AudibleApi 11.0.4.1). New coverage:
DownloadPdfLicenseTests- the step with and without a license in hand. No Audible account exists in those tests, so a test that completes at all proves the step did not go asking for a license of its own.BulkRunSelectionTests- the real run loop against a real library database: which titles each pass reaches, that an absent title is left alone by both, that--forcestill attempts it, and that a refused title is waited on by the follow-up pass and reported once rather than twice.SupplementSyncTests- the scan rules, inApplicationServices.Tests, which already sees the importers throughApplicationServices.Two existing assertions were replaced because they encoded the belief this branch disproves: that a PDF-only title is never waited on, and that a
--pdfrun is never held back.The first CI run failed on all three platforms with
Zero tests ran, from aDtoImporterService.Testsproject this branch briefly added: it pinned MSTest 4.2.2, copied from a project file master had since moved to 4.3.3, so the merge held nine test projects at 4.3.3 and one at 4.2.2. NuGet lifted the sharedMSTest.TestFrameworkto 4.3.3 while the 4.2.2 metapackage kept supplying the adapter, and the adapter refuses to run against a framework of a different version. The project is gone and its tests moved; there is now no project-file change on this branch at all.Left for later
A PDF-only queue item that fails still raises the queue's Abort / Retry / Ignore question, where Ignore sets the audiobook's status to Error. That predates this branch, and the change makes it rarer rather than worse (a written-off PDF now reports success, since nothing went wrong and nothing is left to attempt), but it is worth a look on its own.
Audible's own docs note that the library listing's
pdf_url"doesn't work anymore - authentication data are missing in the provided link", and that the working route is an authenticatedGET https://www.audible.{tld}/companion-file/{asin}, whichApi.AdHocAuthenticatedGetAsynccould reach. That might actually fetch the three missing PDFs rather than merely stop retrying them, but it cannot be verified without a real account.To show artifacts inline, enable in settings.