Sync Cohere client blocks async event loop on every LLM call

[DeFiVC]

What

The CohereClient class at src/services/cohere_client.py:23 instantiates cohere.ClientV2 (the synchronous Cohere SDK client). All methods — generate(), generate_json(), and embed() — are declared async and awaited by callers, but the underlying .chat() and .embed() calls execute synchronously on the main asyncio event loop thread.

Why

Every LLM/embedding call freezes the entire FastAPI server for the duration of the Cohere API request (typically 1-10 seconds). Under concurrent requests, the server becomes effectively single-threaded — all other requests queue behind the blocking call. This makes the service unable to handle concurrent users and defeats the purpose of using FastAPI/async.

Scope

In scope:

• Replace cohere.ClientV2 with cohere.AsyncClientV2 in the Cohere client
• Update all method calls to use await on the async client methods
• Verify the async client has the same API surface

Out of scope:

• Changing the prompt structure or LLM parameters
• Adding caching or rate limiting
• Changing the FastAPI application structure

Acceptance Criteria

• CohereClient uses cohere.AsyncClientV2 instead of cohere.ClientV2
• All generate(), generate_json(), and embed() methods are truly async
• The FastAPI server can handle concurrent LLM requests without blocking
• ruff check src/ tests/ passes
• pytest passes

Technical Context

File to modify:

• src/services/cohere_client.py — class CohereClient (line 23)

Current code:

self._client = cohere.ClientV2(api_key=self._api_key)

Fix:

self._client = cohere.AsyncClientV2(api_key=self._api_key)

The AsyncClientV2 has the same method signatures but returns awaitable coroutines. All callers already await the results, so no caller changes should be needed.

Callers:

• src/services/generation.py — course generation
• src/services/quiz_generator.py — quiz generation
• src/services/feedback_engine.py — feedback generation
• src/services/embedding.py — embedding generation
• scripts/build_index.py — knowledge base index builder

[Aycode01]

can i work on this

[Adamsonoladipupo]

Can I work on this please?

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Okorie2000-code]

Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.

[Joeloo1]

Hi maintainer,

I'd like to work on this issue.

My approach is to replace cohere.ClientV2 with cohere.AsyncClientV2, update the client methods to use await for all Cohere API calls, and verify that the async API remains compatible with the existing implementation. I'll also validate that concurrent requests no longer block the FastAPI event loop, ensure all affected callers continue to work correctly, and run ruff check and pytest before submitting the PR.

I'd appreciate it if this issue could be assigned to me. Thanks!