Repository navigation
[fix] remote score 라벨 점수가 한국어 이슈에서 라벨을 고르게 한다 - #562
Merged
Merged
Conversation
…s select a label Lore: - Intent: Make `remote score --judge none` pick bug, enhancement, or documentation from Korean issue text at the default threshold. - Why: The old score was 0.65*overlap + 0.35*heuristic, so a keyword hit alone capped at 0.35 and never reached 0.70, and whitespace tokens such as "오류가" never equalled the keyword "오류". 224 of 229 single-label issues selected no label (#561). - Changes: - Rule labels score as (their evidence tokens) / (all evidence tokens); Korean keywords match a token prefix, English keywords match the whole token, and fewer than 2 evidence tokens in total abstains with score 0. - Labels without a keyword rule, explicit score input, related-issue scoring, CLI/MCP fields, and the 0.70 default stay as they were. - Add the quality table test (issues below #232), keyword/evidence boundary tests, scripts/autoresearch/label_score_eval.py, and the DDD inventory entries. - Verify: gates G1-G4 met; label_score_eval.py on #232-560 selects 56 with 54 correct (96.43%), on #0-231 selects 43 with 36 correct (83.72%, below the 85% bar); self-verify 27/27 (go test -race, go vet, gofmt; scratch venv supplied pydantic/typer). - Risk: Low; the keyword list was tuned on #0-231, and the #232-560 check is not fully independent. English keywords with attached particles are not matched.
4 tasks
Lore: - Intent: Record the publication and the intent cross-check in the verified-execution report. - Why: execution complete needs a committed report that names the draft PR; the report is the only path allowed to change after pr_head. - Changes: - Add the intent cross-check and the draft PR link with the label override. - Verify: Not-tested: documentation-only change to the report; the code verification is unchanged since the previous commit. - Risk: Low.
This was referenced Oct 15, 2026
…e branch Lore: - Intent: Keep the verified-execution report accurate after main was merged and the cycle was reseeded to generation 3. - Why: PR #562's first CI stopped at lint on main's debt (fixed by #564); the branch now contains that fix and the checks were re-run on the merged head. - Changes: - Add the generation 3 re-verification section (focused tests, label eval, self-verify in a CI-like temporary HOME, CI results for the merge commit). - Verify: Not-tested: report-only change; code is unchanged since the previous review. - Risk: Low.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
요약
issueops remote score --judge none은 한국어 이슈에서 bug·enhancement·documentation 라벨을 거의 고르지 못했습니다. 키워드가 맞아도 점수가 최대 0.35라 기본 threshold 0.70에 닿지 않았고, 조사가 붙은 "오류가" 같은 낱말은 키워드 "오류"와 같지 않았기 때문입니다. 이 PR은 세 라벨의 점수를 "이슈에서 찾은 유형 근거 중 그 라벨이 차지하는 비율"로 바꿔, 근거가 충분하고 한쪽으로 쏠릴 때만 라벨을 고르게 합니다. 변경 전에는 단일 라벨 이슈 229개 중 정답 라벨을 고른 경우가 4개였고, 지금은 개발·검증 두 구간을 합친 230개에서 99개를 골라 90개가 정답입니다.Closes #561
변경 내용
add는address에 맞지 않습니다.type_evidence=<라벨 근거>/<전체 근거>와matched_tokens=<맞은 토큰>두 줄입니다.scripts/autoresearch/label_score_eval.py는 저장소의 단일 라벨 이슈를 읽어remote score의 선택 수와 정답 비율을 잽니다.gh로 이 저장소를 읽기만 하며 CI에는 연결하지 않았습니다.바뀌지 않는 것은 question처럼 키워드 규칙이 없는 라벨의 점수와 evidence 문자열, 명시한
score입력, 관련 이슈 점수, CLI·MCP 응답 필드, threshold 기본값입니다.확인한 것
label_score_eval.pylabel_score_eval.py --checkissueopsremote,remotecmd,issueopscli패키지 테스트self-verify(go test -race ./...,go vet ./...,gofmt포함)self-verify는 처음에 이 머신의 python에pydantic이 없어skills/slack-delegatePython 테스트에서 실패했습니다. 이번 변경과 무관한 환경 문제여서 저장소 밖 임시 환경에pydantic과typer를 설치해 같은 명령을 다시 실행했습니다. 구현은 독립 리뷰어가 검토해 통과시켰습니다.확인하지 못했거나 한계가 있는 것은 다음과 같습니다.
error가처럼 조사가 붙으면 근거가 되지 않습니다. 한글 접두 일치도 음절 단위라거부되는거부된다를 잡지 못합니다.리뷰 포인트
broken,feature,support,readme,오동작은 개발 구간에 나오지 않는 일반 어휘입니다. 과하거나 모자란 단어가 있는지 봐 주세요.type_evidence=1/1로 보입니다. 기권이 눈에 띄도록 문구를 바꿀지는 이 PR 범위 밖이라 두었습니다.