Validate a local job-scam checker's evidence before issuing verdicts
At a glance
OfferProof combines Gemma extraction with explicit rules and quote checks after genuine messages exposed false positives. This workflow is listed under Evidence Validation, Local AI, Evaluation.
- Agents / tools
- OfferProof, Gemma 3 4B, Ollama, Python
SourceBuilding a tool for these workflows? See weekly rates.
What they did
1) Have Python 3.11+, Ollama and enough memory for Gemma 3 4B. Download Surge77/offerproof, create and activate a virtual environment, run `pip install -e .`, and download `gemma3:4b` through Ollama. 2) Save a sample offer as text and run `offerproof check offer.txt`. Inspect the quoted evidence beside each signal. 3) When extending the checker, put wording rules in `signals.yaml` and company domains in `companies.yaml`. Have Gemma answer narrow signal questions with quotations; reject absent quotes and quotes whose wording does not support the specific signal. Let Python combine the validated signals into the verdict. 4) Run `python eval/run.py` to compare rules, model and combined results on labeled messages. Inspect missed scams and genuine messages flagged incorrectly before accepting a change.
How it went
On 20 author-written samples, the revised combination caught 11 of 12 scams and falsely flagged none of eight genuine messages.
Worth knowing
The evaluation is synthetic and small. An unflagged message remains unverified; it is never certified genuine.