[Portal UI] Customizable QoS (i.e. retry logic) provided by customers

[bulutcambazi]

Objective

Implement a centralized & customizable (UI based) auto-retry toggle for failed RPC calls in Portal.

Origin Document

Current RPC calls have ~90% success rate, requiring retry logic implementation for most calls.

A centralized retry mechanism would improve developer experience by handling failures automatically through either a UI toggle in Portal or HTTP headers.

Goals

• Improve RPC call reliability for customers
• Allow customers to configure their service for their own needs
• Reduce implementation burden on developers
• Maintain network health while handling retries

Deliverables

• Design & implement a retry mechanism with configurable parameters in the PATH repo
• Implement UI toggle in Portal UI for enabling + configuring auto-retry
• Add support for retry configuration via HTTP headers
• Allow setting how many retries a customer is okay with
• Allow applying a retry setting to either one or all user endpoints
• Add metrics/logging for retry attempts and success rates

Non-goals / Non-deliverables

• Building custom retry logic for specific use cases
• Implementing complex backoff strategies
• Adding retry support for non-RPC endpoints

General deliverables

• Comments: Document retry logic and configuration options
• Testing: Add unit tests for retry mechanism and integration tests for UI/API
• Documentation: Update Portal documentation with retry configuration details
• Makefile: Add targets for testing retry functionality

---

Creator: @bulutcambazi
Co-owners: @Olshansk @fredteumer

[Olshansk]

@bulutcambazi Ty for the issue!

I've updated the details. You can see the previous versions of the issue here:

But I'm also uploading a screenshot for posterity: