Skip to content

Add review workflow to approve predicted phrases before injection - #5267

Draft
Kaushik-Kumar-CEG wants to merge 7 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/review-ml-phrases
Draft

Add review workflow to approve predicted phrases before injection#5267
Kaushik-Kumar-CEG wants to merge 7 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/review-ml-phrases

Conversation

@Kaushik-Kumar-CEG

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG commented Aug 16, 2026

Copy link
Copy Markdown

part of #5244, builds on #5243. Adds a review pass in front of the injection tool
from #5262 so a maintainer sees a predicted phrase before it is written

a wrong required phrase drops a real license detection, so nothing reaches a rule
on the model's word alone

this PR:

  • review_ml_phrases.py - three commands. predict scores every predicted
    phrase and writes them to a jsonl file, review walks the uncertain ones past
    a maintainer with a diff and y/n/e/q, apply injects what came back approved.
    --dry-run, --limit, --license-expression, --auto-threshold,
    --review-threshold
  • test_review_ml_phrases.py - 67 tests, the tagger and tokenizer are faked and
    the crf is built in the test, so no weights and no network

injection goes through add_ml_phrases.inject, so is_good,
find_phrase_spans_in_text and add_required_phrase_to_rule still do the work.
extract_spans and first_subword_positions come from train_model. the diff
preview is add_required_phrase_to_rule under dry_run rather than my own span
picking, so what is shown is what gets written. nothing in src/ changes

needs #5222 and #5262 to import, same as #5262 itself

notes

predict_words decodes with viterbi and returns no score, so there was nothing
to build the tiers on. predict scores the decoded path twice, once free and
once with every label but the decoded one pinned out of reach at the span's
words. the difference is the probability mass on paths that tag that span the way
it was decoded. it says what the model thinks, not that the phrase is right,
is_good and the review step are still the gate

--auto-threshold 0.95 and --review-threshold 0.60 are placeholders, the
checkpoint is not published yet so there was nothing to calibrate against.
predict prints a confidence histogram so the numbers can be picked from real
data later

phrases a human approves still get source: ml_model, that is what inject
sets. the review file keeps the decision and the original predicted text if you
want to tell them apart

Rule.dump() opens without newline='' so on windows it rewrites every line
ending and the diff covers the whole file. not from this script, it hits anything
calling rule.dump(), but worth knowing before a batch run on windows

checks

67 tests pass with no model. pytest tests/licensedcode/ -k required_phrase
still passes, 44 tests

fed the gold labels back in as a perfect prediction over 400 marked rules with
the markers stripped: 382 phrases survive is_good, 379 inject, a second
apply injects nothing and leaves every file byte identical. one real write
checked with git diff and restored

the confidence maths is checked against brute force enumeration on a small crf,
agrees to 1e-12

disclosure : used Claude to help review and clean up bugs in script

@Kaushik-Kumar-CEG
Kaushik-Kumar-CEG marked this pull request as draft August 16, 2026 17:46
@Kaushik-Kumar-CEG Kaushik-Kumar-CEG changed the title add review workflow for ml predicted required phrases Add review workflow to approve predicted phrases before injection Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant