⚡ Bolt: Use async Groq client to prevent event loop blocking - #145
⚡ Bolt: Use async Groq client to prevent event loop blocking#145Adityasingh-8858 wants to merge 1 commit into
Conversation
Co-authored-by: Deepaksingh7238 <110552872+Deepaksingh7238@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
There was a problem hiding this comment.
Pull request overview
This PR updates the FastAPI backend to use Groq’s asynchronous client to avoid blocking the asyncio event loop during LLM calls, and records the optimization as a Bolt learning note.
Changes:
- Switched from
GroqtoAsyncGroqinbackend/main.py. - Updated Groq chat completion calls to be awaited inside async route handlers.
- Added a Bolt note documenting the reasoning and expected impact.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| backend/main.py | Uses AsyncGroq and await for Groq completions in async endpoints to reduce event loop blocking. |
| .jules/bolt.md | Adds a short learning/action entry about preferring async API clients in FastAPI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| global groq_client | ||
| if groq_client is None: | ||
| groq_client = Groq(api_key=GROQ_API_KEY) | ||
| chat_completion = groq_client.chat.completions.create( | ||
| # ⚡ Bolt Optimization: Use AsyncGroq to prevent blocking the FastAPI event loop during network I/O | ||
| # Expected Impact: Significantly increases concurrent request capacity when generating AI voices. | ||
| groq_client = AsyncGroq(api_key=GROQ_API_KEY) |
| global groq_client | ||
| if groq_client is None: | ||
| groq_client = Groq(api_key=GROQ_API_KEY) | ||
| # ⚡ Bolt Optimization: Use AsyncGroq to prevent blocking the FastAPI event loop during network I/O | ||
| # Expected Impact: Avoids 500ms+ latency spikes on other endpoints while waiting for LLM completions. | ||
| groq_client = AsyncGroq(api_key=GROQ_API_KEY) |
💡 What: Replaced the synchronous
Groqclient withAsyncGroqand updated thechat.completions.createcalls to useawaitinbackend/main.py. Added explanatory comments detailing the expected impact.🎯 Why: Synchronous network I/O in FastAPI blocks the asyncio event loop. Using the standard
Groqclient meant that during the 500ms-2s it takes to generate an AI summary or voice response, the server could not process any other incoming requests, severely limiting concurrency.📊 Impact: Significantly increases concurrent request capacity and eliminates latency spikes on other endpoints caused by event loop blocking.
🔬 Measurement: Can be verified by running a load test against the API. While one request is blocked waiting for Groq, other endpoints (like
/roomsor/participants) will now respond immediately instead of timing out.PR created automatically by Jules for task 5310748832100092226 started by @Deepaksingh7238