Skip to content
UsecaseBoard
← All use cases
devtoEvidence Validation

Validate a local job-scam checker's evidence before issuing verdicts

At a glance

OfferProof combines Gemma extraction with explicit rules and quote checks after genuine messages exposed false positives. This workflow is listed under Evidence Validation, Local AI, Evaluation.

SourceBuilding a tool for these workflows? See weekly rates.

What they did

1) Have Python 3.11+, Ollama and enough memory for Gemma 3 4B. Download Surge77/offerproof, create and activate a virtual environment, run `pip install -e .`, and download `gemma3:4b` through Ollama. 2) Save a sample offer as text and run `offerproof check offer.txt`. Inspect the quoted evidence beside each signal. 3) When extending the checker, put wording rules in `signals.yaml` and company domains in `companies.yaml`. Have Gemma answer narrow signal questions with quotations; reject absent quotes and quotes whose wording does not support the specific signal. Let Python combine the validated signals into the verdict. 4) Run `python eval/run.py` to compare rules, model and combined results on labeled messages. Inspect missed scams and genuine messages flagged incorrectly before accepting a change.

How it went

On 20 author-written samples, the revised combination caught 11 of 12 scams and falsely flagged none of eight genuine messages.

Worth knowing

The evaluation is synthetic and small. An unflagged message remains unverified; it is never certified genuine.

Save ideas in this browser. Upvotes count across the board.

Prompt excerpt

No prompt excerpt was provided by the source.

Sponsored

CodeBuddy

Tencent Cloud AI coding assistant for IDE and CLI.

Try CodeBuddy ↗
Explore all →