AUTO_YES
“The contractor shall submit a monthly report.”
shall.Snippets are intentionally focused on the decisions illustrated above.
Read the source, preserve table rows, track section context, and create a stable unit ID for each meaningful piece of text.
# rfplab/segment.py for raw in text.splitlines(): cleaned = _clean(raw) if SECTION_HDR.match(cleaned) or cleaned.startswith("#"): section_path = cleaned.lstrip("# ").strip() continue if line.lstrip().startswith("|"): # One table row is one auditable unit. units.append(Unit(..., section_path, "table_row", body, ...)) continue for sent in _split_sentences(body): if len(sent.split()) >= min_words: units.append(Unit(..., section_path, "sentence", sent, ...))
{
"unit_id": "u00042",
"doc_id": "RFP Sample 4",
"section_path": "C.3.2 Reporting Requirements",
"unit_type": "sentence",
"text": "The contractor shall submit a monthly report.",
"text_hash": "a1b2c3d4e5f6",
"ordinal": 42
}The router uses transparent rules for only high-confidence cases. Ambiguity remains eligible for model review.
# rfplab/rules.py m = MANDATORY.search(text) # shall, must, required to if m: if ISSUER_SUBJECT.match(text): return DEFER, f"issuer_subject_shall:{m.group().lower()}" return AUTO_YES, f"mandatory_modal:{m.group().lower()}" if utype == "table_row": return DEFER, "table_row" if SOFT.search(text): return DEFER, f"soft_modal:{s.group().lower()}" if words and words[0].lower().strip(".,:;") in IMPERATIVE_VERBS: return DEFER, "imperative" return AUTO_NO, "no_obligation_signal"
“The contractor shall submit a monthly report.”
shall.“The Government shall provide access to the facility.”
shall appears, but the issuer—not the vendor—is the subject.“This RFP describes the agency’s current operating environment.”
The model receives only uncertain units and must return one concise record per input unit.
# rfplab/classify.py for b in batch: lines.append(f"[{b['idx']}] ({b['unit_type']} | {b['reason']}) {b['text']}") # The model must emit exactly: INDEX|LABEL|WHO|TRIGGER # LABEL = mandatory | desired | not_requirement | borderline parsed = parse_response(text)
Only deferred units are included. The router’s reason is supplied as a hint, while the model makes the final label decision.
Classify each numbered unit. Lines are contiguous. [1] (sentence | soft_modal:should) The contractor should provide a monthly status report. [2] (table_row | table_row) System availability | 99.9% monthly uptime [3] (sentence | issuer_subject_shall:shall) The Government shall provide facility access. Return exactly 3 lines, indices 1-3.
You classify individual units of text from a Request for Proposal (RFP). Your ONLY job is to decide whether each unit states a REQUIREMENT the vendor must respond to. You do not find or summarize requirements - each unit is given to you already segmented. DEFINITION A requirement is anything the vendor could be scored on or held to. Test: "If we ignored this, could our proposal be marked non-compliant?" If yes, it is a requirement. DECISION PROCEDURE - apply in order, stop at the first that decides: 1. WHO is obligated? If the unit obligates the ISSUER (for example, "the Government will provide...") or no one (background, history, scope narrative), label not_requirement. 2. IS IT BINDING? shall/must/is required to -> mandatory. should/recommended/preferred -> desired. may/can/optional/at its discretion -> not_requirement. 3. IS IT ACTIONABLE? A specific capability, deliverable, standard, or constraint -> requirement. A bare definition or reference with no obligation -> not_requirement. 4. If none of the above clearly decides, label borderline. Do NOT default to mandatory when unsure. RULES - Judge each unit as written. Do not infer obligations that are not stated. - "will" usually describes the ISSUER's actions - check the subject before treating it as a requirement. - A table row is a requirement ONLY if it states a performance obligation, a deliverable, or a threshold the vendor must meet. A row that merely names or numbers a contract line item, price element, or cost category is contract STRUCTURE, not a requirement -> not_requirement. Examples that are NOT requirements: a CLIN number paired with a task title, a pricing/cost cell, or a labor-hour rate line. - Lines are contiguous; use neighbours for context but judge only the numbered unit itself. OUTPUT FORMAT - one line per unit, pipe-delimited, nothing else. No prose, no JSON, no markdown fences: INDEX|LABEL|WHO|TRIGGER LABEL = mandatory | desired | not_requirement | borderline WHO = vendor | issuer | none TRIGGER = the exact word or phrase that decided it (or "none") Return exactly one line for every index you were given.
Partial model output is treated as an operational failure, not accepted as a partial answer. The system checks response format, valid labels, and complete index coverage before it applies any decision.
# rfplab/classify.py sent = {b["idx"] for b in batch} got = set(parsed) if got != sent: stats["validation_failures"] += 1 if depth < max_depth and len(chunk) > 1: stats["retries"] += 1 mid = len(chunk) // 2 _do_batch(chunk[:mid], ...) _do_batch(chunk[mid:], ...) return # Any unresolved unit is preserved as borderline.
For the three-unit batch shown in Stage 2, the model must return exactly one pipe-delimited line for each submitted index.
1|desired|vendor|should 2|mandatory|vendor|99.9% monthly uptime 3|not_requirement|issuer|The Government shall
1. Parse each line only if it follows: INDEX|LABEL|WHO|TRIGGER
2. Accept only supported labels:
mandatory | desired | not_requirement | borderline
3. Compare the returned index set with the sent index set exactly.
Sent: {1, 2, 3}
Returned: {1, 2, 3} - valid
Returned: {1, 3} - invalid: index 2 is missing
Returned: {1, 2, 3, 4} - invalid: unexpected index 4
4. If the set is invalid, split the batch in half and retry each half,
up to the configured retry depth.
5. If a line still cannot be parsed after retries, preserve that unit as:
label=borderline, trigger=unparsed, status=unparsed
6. If the API call itself errors, preserve every unit in that batch as
borderline with status=error and record the failure.Cross-check the final label against deterministic signals and reference integrity, then create one prioritized human-review queue across all repeated runs.
# rfplab/rules.py if label in {"mandatory", "desired"} and ISSUER_SUBJECT.match(text): flags.append("issuer_subject_but_labeled_requirement") if label == "not_requirement" and MANDATORY.search(text) and VENDOR_TOKEN.search(text): flags.append("binding_modal_but_labeled_non_requirement") xref = classify_xref(text, known_sections) if xref == "external": flags.append("unresolved_cross_reference")
These checks do not overwrite the model’s label. They add an explicit reason for a person to inspect the source unit.
1. Issuer-subject safety check A unit labeled mandatory/desired starts with “Government”, “COR”, “GSA”, “client”, or “contracting officer”, and does not clearly name the vendor. -> issuer_subject_but_labeled_requirement 2. Binding-modal safety check A unit is labeled not_requirement but contains a vendor token plus “shall”, “must”, or “required to” (excluding a bare list-introducing stub). -> binding_modal_but_labeled_non_requirement 3. Trigger contradiction check A unit is labeled mandatory but its cited trigger contains “may”, “optional”, “preferred”, or similar permission language. -> trigger_contradicts_mandatory 4. Reference integrity check A unit points to an external Attachment, Appendix, or Exhibit, or to an internal section that cannot be found in the source document. -> unresolved_cross_reference or unresolved_section_reference
Its label differs across repeated runs. Example: mandatory x2, not_requirement x1. It is placed first even if it also has a flag.
Its label is stable, but any run produced a reconciliation warning. The report retains the union of flags across repeats.
Its label is consistently borderline: the model could not make a defensible yes/no decision.
For each stable unit ID across the repeated runs: if labels differ: add P1 UNSTABLE else if any reconciliation flag exists: add P2 FLAGGED else if the label is borderline: add P3 BORDERLINE otherwise: omit from the review queue Each unit appears once at its highest applicable priority. The queue is sorted P1 -> P2 -> P3, then by unit ID.
A compact reliability snapshot using three valid repetitions per document/model pair. This measures consistency and operational behavior; it does not establish correctness without a human-labeled golden set.
| Document | Model | Requirement range | Stability | Review queue | Avg. tokens/run (input / output) | Avg. live cost/run |
|---|---|---|---|---|---|---|
| Large Project Scope | Sonnet 4.5 | 90–92 | 98.8% | 8 | 5,819 / 1,312 | $0.0371 |
| Large Project Scope | Sonnet 5 | 86–90 | 98.1% | 17 | 8,247 / 7,410 | $0.0906 |
| RFP Sample 4 | Sonnet 4.5 | 178–179 | 98.9% | 15 | 8,597 / 1,553 | $0.0491 |
| RFP Sample 4 | Sonnet 5 | 175–177 | 98.6% | 16 | 12,126 / 5,554 | $0.0798 |
| Technical Direction | Sonnet 4.5 | 329–334 | 99.5% | 23 | 20,885 / 4,061 | $0.1236 |
| Technical Direction | Sonnet 5 | 328–332 | 99.1% | 26 | 29,562 / 13,019 | $0.1893 |
Both configurations were highly consistent. Stability stayed above 98% on every document, with Sonnet 4.5 slightly higher in all three comparisons in this three-run sample.
Sonnet 5 used materially more output tokens. On the Technical Direction RFP, it averaged 13,019 output tokens per run versus 4,061 for Sonnet 4.5; its per-run cost was correspondingly higher.
Review queue size is useful evidence, not an error total. It combines unstable labels, deterministic reconciliation flags, and consistent borderline cases so people review the riskiest units first. Accuracy still requires a labeled golden set.
One representative example of each outcome from every document, using the Sonnet 4.5 comparison runs. Source text is verbatim; the decision column records why it reached that outcome.
| Document | Outcome | Section | Verbatim source text | Decision evidence |
|---|---|---|---|---|
| Large Project Scope | Requirement | Preamble | The proposals must be signed by an official authorized to bind the offeror, and it shall contain a | Mandatory - binding modal must. |
| Large Project Scope | Review: P1 | 1.5 Format for Proposals | Also, include in this section suggestions and ideas as to how you can | Unstable across 3 runs: desired ×2, mandatory ×1. |
| Large Project Scope | Rejected | Preamble | This example RFP is provided to guide applicants in preparing | Not a requirement - contextual narrative; no_obligation_signal. |
| RFP Sample 4 | Requirement | 1.0 Introduction | The selected vendor will be expected to complete the contracted scope of work within the specified timeframe, under the general direction and coordination of the City’s Engineering Department as authorized by the City Manager. | Mandatory - vendor-facing commitment; trigger will be expected to. |
| RFP Sample 4 | Review: P1 | 7.0 Proposal Questions and Answers | Violation of this restriction will be considered a violation of the rules and be grounds for disqualification of the Vendor’s proposal. | Unstable across 3 runs: not_requirement ×2, mandatory ×1. |
| RFP Sample 4 | Rejected | 1.0 Introduction | The City of Dubuque, Iowa is soliciting competitive sealed proposals from qualified professional vendors to complete a project involved with performing updates at the Multicultural Family Center for the City of Dubuque. | Not a requirement - issuer background; no_obligation_signal. |
| Technical Direction | Requirement | B.1 General Description | The work shall be performed in accordance with all sections of the awarded task order and the Contractor’s Basic Contract, under which the resulting task order will be placed. | Mandatory - binding modal shall. |
| Technical Direction | Review: P1 | F.1.2 Duty Hours | US | 5 days per week x 8 hours per day | Unstable across 3 runs: mandatory ×2, not_requirement ×1. |
| Technical Direction | Rejected | Preamble | This Task Order (TO) is identified by task order number and contract number listed in blocks 2 and 3 of the Form 300. | Not a requirement - descriptive reference; no_obligation_signal. |