Skip to content

perf(serve): append streamed output without copying accumulated text - #185

Open
Afloat16 wants to merge 1 commit into
Edge0-AI:mainfrom
Afloat16:afloat/20261111-next-linear-response-assembly
Open

Afloat16 wants to merge 1 commit into
Edge0-AI:mainfrom
Afloat16:afloat/20261111-next-linear-response-assembly

Conversation

@Afloat16

Copy link
Copy Markdown

What & why

Each Responses text/tool-argument delta copied previously accumulated output, repeatedly copying growing answer, reasoning and tool-argument prefixes.

Append to the JSON-owned string in place, preserving geometric growth and existing event/complete-response structure.

Area

Engine, CLI or server (edge0-infer/).

How was it tested?

moe_test_serve_protocol tests/fixtures/portal/requests.json

The target and request/event fixtures pass. Three regressions append 256 chunks of 1024 bytes and verify the exact 262144-byte result. Each fixed path makes 4 allocations of at least 32 KiB; the original answer path makes 464 and fails the bound. Model throughput was not measured.

All 14 L0 checks passed in a local integration with the other focused fixes. git diff --check passed.

Local C++ environment: Linux x86_64, GCC 14.2, CPU preset. Targeted builds downgrade pre-existing ignored-attributes, dangling-reference, range-loop-construct and misleading-indentation warnings (plus maybe-uninitialized in the existing diagnostics source); these accommodations are not source changes. The all-target build is blocked by existing assets_test.cpp use of std::count_if without . A clean full L0/L1 result, GPU tests and model-asset execution are not claimed.

Checklist

  • Full ctest --preset cpu -L "L0|L1" passes locally
  • Added/updated focused regressions and ran the affected target
  • No public interface/configuration change requiring additional documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant