Technical walkthrough

How we extract and evaluate requirements

Design principle: documents are segmented deterministically before any model is called. The LLM can change a classification, but it cannot create, remove, or renumber source units.

Pipeline stages and responsible code

Snippets are intentionally focused on the decisions illustrated above.

Stage 0

Load and segment

Read the source, preserve table rows, track section context, and create a stable unit ID for each meaningful piece of text.

# rfplab/segment.py
for raw in text.splitlines():
    cleaned = _clean(raw)
    if SECTION_HDR.match(cleaned) or cleaned.startswith("#"):
        section_path = cleaned.lstrip("# ").strip()
        continue
    if line.lstrip().startswith("|"):
        # One table row is one auditable unit.
        units.append(Unit(..., section_path, "table_row", body, ...))
        continue
    for sent in _split_sentences(body):
        if len(sent.split()) >= min_words:
            units.append(Unit(..., section_path, "sentence", sent, ...))

Example output: one segmented unit

{
  "unit_id": "u00042",
  "doc_id": "RFP Sample 4",
  "section_path": "C.3.2 Reporting Requirements",
  "unit_type": "sentence",
  "text": "The contractor shall submit a monthly report.",
  "text_hash": "a1b2c3d4e5f6",
  "ordinal": 42
}
Stage 1

Route obvious cases

The router uses transparent rules for only high-confidence cases. Ambiguity remains eligible for model review.

# rfplab/rules.py
m = MANDATORY.search(text) # shall, must, required to
if m:
    if ISSUER_SUBJECT.match(text):
        return DEFER, f"issuer_subject_shall:{m.group().lower()}"
    return AUTO_YES, f"mandatory_modal:{m.group().lower()}"
if utype == "table_row":
    return DEFER, "table_row"
if SOFT.search(text):
    return DEFER, f"soft_modal:{s.group().lower()}"
if words and words[0].lower().strip(".,:;") in IMPERATIVE_VERBS:
    return DEFER, "imperative"
    return AUTO_NO, "no_obligation_signal"

AUTO_YES

“The contractor shall submit a monthly report.”

Reason: binding vendor-facing modal: shall.

DEFER

“The Government shall provide access to the facility.”

Reason: shall appears, but the issuer—not the vendor—is the subject.

AUTO_NO

“This RFP describes the agency’s current operating environment.”

Reason: no obligation signal; it is contextual narrative.
Stage 2

Classify deferred units

The model receives only uncertain units and must return one concise record per input unit.

# rfplab/classify.py
for b in batch:
    lines.append(f"[{b['idx']}] ({b['unit_type']} | {b['reason']}) {b['text']}")

# The model must emit exactly: INDEX|LABEL|WHO|TRIGGER
# LABEL = mandatory | desired | not_requirement | borderline
parsed = parse_response(text)

Example user message sent to the LLM

Only deferred units are included. The router’s reason is supplied as a hint, while the model makes the final label decision.

Classify each numbered unit. Lines are contiguous.

[1] (sentence | soft_modal:should) The contractor should provide a monthly status report.
[2] (table_row | table_row) System availability | 99.9% monthly uptime
[3] (sentence | issuer_subject_shall:shall) The Government shall provide facility access.

Return exactly 3 lines, indices 1-3.
Current system prompt sent with every batch
You classify individual units of text from a Request for Proposal (RFP).
Your ONLY job is to decide whether each unit states a REQUIREMENT the vendor
must respond to. You do not find or summarize requirements - each unit is
given to you already segmented.

DEFINITION
A requirement is anything the vendor could be scored on or held to. Test:
"If we ignored this, could our proposal be marked non-compliant?" If yes, it
is a requirement.

DECISION PROCEDURE - apply in order, stop at the first that decides:
1. WHO is obligated? If the unit obligates the ISSUER (for example, "the
   Government will provide...") or no one (background, history, scope
   narrative), label not_requirement.
2. IS IT BINDING? shall/must/is required to -> mandatory.
   should/recommended/preferred -> desired.
   may/can/optional/at its discretion -> not_requirement.
3. IS IT ACTIONABLE? A specific capability, deliverable, standard, or
   constraint -> requirement. A bare definition or reference with no
   obligation -> not_requirement.
4. If none of the above clearly decides, label borderline. Do NOT default to
   mandatory when unsure.

RULES
- Judge each unit as written. Do not infer obligations that are not stated.
- "will" usually describes the ISSUER's actions - check the subject before
  treating it as a requirement.
- A table row is a requirement ONLY if it states a performance obligation, a
  deliverable, or a threshold the vendor must meet. A row that merely names or
  numbers a contract line item, price element, or cost category is contract
  STRUCTURE, not a requirement -> not_requirement.
  Examples that are NOT requirements: a CLIN number paired with a task title,
  a pricing/cost cell, or a labor-hour rate line.
- Lines are contiguous; use neighbours for context but judge only the
  numbered unit itself.

OUTPUT FORMAT - one line per unit, pipe-delimited, nothing else. No prose,
no JSON, no markdown fences:
INDEX|LABEL|WHO|TRIGGER

LABEL = mandatory | desired | not_requirement | borderline
WHO = vendor | issuer | none
TRIGGER = the exact word or phrase that decided it (or "none")

Return exactly one line for every index you were given.
Stage 2 safeguard

Validate every response

Partial model output is treated as an operational failure, not accepted as a partial answer. The system checks response format, valid labels, and complete index coverage before it applies any decision.

# rfplab/classify.py
sent = {b["idx"] for b in batch}
got = set(parsed)
if got != sent:
    stats["validation_failures"] += 1
    if depth < max_depth and len(chunk) > 1:
        stats["retries"] += 1
        mid = len(chunk) // 2
        _do_batch(chunk[:mid], ...)
        _do_batch(chunk[mid:], ...)
        return
# Any unresolved unit is preserved as borderline.

Example valid LLM response

For the three-unit batch shown in Stage 2, the model must return exactly one pipe-delimited line for each submitted index.

1|desired|vendor|should
2|mandatory|vendor|99.9% monthly uptime
3|not_requirement|issuer|The Government shall
What “validate” means in this pipeline
1. Parse each line only if it follows: INDEX|LABEL|WHO|TRIGGER
2. Accept only supported labels:
   mandatory | desired | not_requirement | borderline
3. Compare the returned index set with the sent index set exactly.
   Sent:     {1, 2, 3}
   Returned: {1, 2, 3}  - valid
   Returned: {1, 3}     - invalid: index 2 is missing
   Returned: {1, 2, 3, 4} - invalid: unexpected index 4
4. If the set is invalid, split the batch in half and retry each half,
   up to the configured retry depth.
5. If a line still cannot be parsed after retries, preserve that unit as:
   label=borderline, trigger=unparsed, status=unparsed
6. If the API call itself errors, preserve every unit in that batch as
   borderline with status=error and record the failure.
Stage 4

Reconcile and prioritize review

Cross-check the final label against deterministic signals and reference integrity, then create one prioritized human-review queue across all repeated runs.

# rfplab/rules.py
if label in {"mandatory", "desired"} and ISSUER_SUBJECT.match(text):
    flags.append("issuer_subject_but_labeled_requirement")
if label == "not_requirement" and MANDATORY.search(text) and VENDOR_TOKEN.search(text):
    flags.append("binding_modal_but_labeled_non_requirement")
xref = classify_xref(text, known_sections)
if xref == "external":
    flags.append("unresolved_cross_reference")

Stage 4 sequence (no LLM call)

1. Start with Stage 2 outputLabel, obligated party, trigger, and source unit.
2. Run Python checksCompare the label and trigger against deterministic signals.
3. Attach flagsKeep the label; add any reasons a person should inspect it.
4. Build the queueAcross repeats, assign P1, P2, or P3 review priority.

How reconciliation flags a decision

These checks do not overwrite the model’s label. They add an explicit reason for a person to inspect the source unit.

Deterministic checks applied to every labeled unit
1. Issuer-subject safety check
   A unit labeled mandatory/desired starts with “Government”, “COR”, “GSA”,
   “client”, or “contracting officer”, and does not clearly name the vendor.
   -> issuer_subject_but_labeled_requirement

2. Binding-modal safety check
   A unit is labeled not_requirement but contains a vendor token plus
   “shall”, “must”, or “required to” (excluding a bare list-introducing stub).
   -> binding_modal_but_labeled_non_requirement

3. Trigger contradiction check
   A unit is labeled mandatory but its cited trigger contains “may”,
   “optional”, “preferred”, or similar permission language.
   -> trigger_contradicts_mandatory

4. Reference integrity check
   A unit points to an external Attachment, Appendix, or Exhibit, or to an
   internal section that cannot be found in the source document.
   -> unresolved_cross_reference or unresolved_section_reference

How a unit receives a human-review priority

P1 UNSTABLE

Its label differs across repeated runs. Example: mandatory x2, not_requirement x1. It is placed first even if it also has a flag.

P2 FLAGGED

Its label is stable, but any run produced a reconciliation warning. The report retains the union of flags across repeats.

P3 BORDERLINE

Its label is consistently borderline: the model could not make a defensible yes/no decision.

Queue construction and ordering
For each stable unit ID across the repeated runs:
  if labels differ:              add P1 UNSTABLE
  else if any reconciliation flag exists: add P2 FLAGGED
  else if the label is borderline: add P3 BORDERLINE
  otherwise:                     omit from the review queue

Each unit appears once at its highest applicable priority.
The queue is sorted P1 -> P2 -> P3, then by unit ID.
Live Anthropic evaluation

What three live runs across three RFPs show

A compact reliability snapshot using three valid repetitions per document/model pair. This measures consistency and operational behavior; it does not establish correctness without a human-labeled golden set.

18
live extraction runs summarized
3 documents × 2 models × 3 repeats
354,442
total model tokens
255,708 input + 98,734 output
98.1%–99.5%
unit-label stability across the six test configurations
0
API errors and validation failures across the 18 summarized runs
DocumentModelRequirement rangeStabilityReview queueAvg. tokens/run
(input / output)
Avg. live cost/run
Large Project ScopeSonnet 4.590–9298.8%85,819 / 1,312$0.0371
Large Project ScopeSonnet 586–9098.1%178,247 / 7,410$0.0906
RFP Sample 4Sonnet 4.5178–17998.9%158,597 / 1,553$0.0491
RFP Sample 4Sonnet 5175–17798.6%1612,126 / 5,554$0.0798
Technical DirectionSonnet 4.5329–33499.5%2320,885 / 4,061$0.1236
Technical DirectionSonnet 5328–33299.1%2629,562 / 13,019$0.1893

Both configurations were highly consistent. Stability stayed above 98% on every document, with Sonnet 4.5 slightly higher in all three comparisons in this three-run sample.

Sonnet 5 used materially more output tokens. On the Technical Direction RFP, it averaged 13,019 output tokens per run versus 4,061 for Sonnet 4.5; its per-run cost was correspondingly higher.

Review queue size is useful evidence, not an error total. It combines unstable labels, deterministic reconciliation flags, and consistent borderline cases so people review the riskiest units first. Accuracy still requires a labeled golden set.

Representative extraction output

What the pipeline keeps, flags, and rejects

One representative example of each outcome from every document, using the Sonnet 4.5 comparison runs. Source text is verbatim; the decision column records why it reached that outcome.

DocumentOutcomeSectionVerbatim source textDecision evidence
Large Project ScopeRequirementPreambleThe proposals must be signed by an official authorized to bind the offeror, and it shall contain aMandatory - binding modal must.
Large Project ScopeReview: P11.5 Format for ProposalsAlso, include in this section suggestions and ideas as to how you canUnstable across 3 runs: desired ×2, mandatory ×1.
Large Project ScopeRejectedPreambleThis example RFP is provided to guide applicants in preparingNot a requirement - contextual narrative; no_obligation_signal.
RFP Sample 4Requirement1.0 IntroductionThe selected vendor will be expected to complete the contracted scope of work within the specified timeframe, under the general direction and coordination of the City’s Engineering Department as authorized by the City Manager.Mandatory - vendor-facing commitment; trigger will be expected to.
RFP Sample 4Review: P17.0 Proposal Questions and AnswersViolation of this restriction will be considered a violation of the rules and be grounds for disqualification of the Vendor’s proposal.Unstable across 3 runs: not_requirement ×2, mandatory ×1.
RFP Sample 4Rejected1.0 IntroductionThe City of Dubuque, Iowa is soliciting competitive sealed proposals from qualified professional vendors to complete a project involved with performing updates at the Multicultural Family Center for the City of Dubuque.Not a requirement - issuer background; no_obligation_signal.
Technical DirectionRequirementB.1 General DescriptionThe work shall be performed in accordance with all sections of the awarded task order and the Contractor’s Basic Contract, under which the resulting task order will be placed.Mandatory - binding modal shall.
Technical DirectionReview: P1F.1.2 Duty HoursUS | 5 days per week x 8 hours per dayUnstable across 3 runs: mandatory ×2, not_requirement ×1.
Technical DirectionRejectedPreambleThis Task Order (TO) is identified by task order number and contract number listed in blocks 2 and 3 of the Form 300.Not a requirement - descriptive reference; no_obligation_signal.