Quiet continuous batching at default verbosity - #48314
Conversation
A plain generate_batch printed five WARNING lines (attention auto-switch, warmup start/finish, progress-bar note) and a progress bar; generate prints nothing. Status messages move to info, and the progress bar is opt-in. Actionable warnings (unset eos_token_id, warmup failure, dropped requests) stay at warning.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
CI recapDashboard: View test results in Grafana |
remi-or
left a comment
remi-or
left a comment
There was a problem hiding this comment.
Some nits but otherwise we are good! Thanks for the improvement
| if version is not None: | ||
| target_implem = f"flash_attention_{version}" # no "paged|" prefix here to enter the branch below | ||
| logger.warning( | ||
| logger.info( |
There was a problem hiding this comment.
I think this is better as a warning, because a user initializing with sdpa should k now we change the implementation from under them.
| @@ -1183,10 +1183,10 @@ def continuous_batching_context_manager( | |||
| ) | |||
| if warmup and not manager.warmed_up: | |||
| # Warmup is long (~30 sec): best to signal the user it's happening than let them think the manager is stuck | |||
There was a problem hiding this comment.
| # Warmup is long (~30 sec): best to signal the user it's happening than let them think the manager is stuck | |
| # TODO: have a progress bar for the warmup as well, like other inference engine |
because I agree this might be a bit too verbose but we still need some visibility for the users who want some
remi-or
left a comment
remi-or
left a comment
There was a problem hiding this comment.
Some nits but otherwise we are good! Thanks for the improvement
3 participants
What does this PR do?
Fixes #48313. Part of #48294.
Changes:
warningtoinfo.Solving n requestsprogress bar becomes opt-in (progress_bar=Falseby default): likegenerate,generate_batchprints nothing at default verbosity.warning: unseteos_token_id, warmup failure, requests dropped during shutdown.Validation: the issue's repro prints only the
eos_token_idwarning after this change (the repro's GenerationConfig genuinely lacks one; the remaining[transformers] You do not have flash_attn installed...line comes from the kernels integration, out of this PR's scope)Who can review?
@remi-or (continuous batching)