Pipeline (ordered): native extract → metadata → OCR fallback → normalization → scoring.
| Stage | Action / Rationale |
|---|---|
| Raw extraction | DOCX runs (font, color), PDF content-stream scan (color ops, font-size). Use PyMuPDF, pdfminer.six, python-docx. |
| Metadata | Read PDF XMP / DOCX coreProps / image alt text — often contains keywords. |
| OCR fallback | Run Tesseract or cloud OCR on pages when native text sparse or suspicious; compare native vs OCR. |
| Normalization | Unicode NFC, remove control & zero-width chars, collapse whitespace, canonicalize emails/dates/phones. |
| Provenance | Store raw/native/OCR blobs and reason codes for audit. |
Core features and exact thresholds you can implement immediately:
[-]invisible_density = invisible_chars / total_chars. Flag if > 0.01 (1%).FFFFFF or PDF stream ops like rg 1 1 1/RG 1 1 1. If span length > 10 chars and color==background → suspicious.
tiny_font_fraction = spans_lt_4pt / total_spans. Flag if > 0.01.
native_chars == 0 && ocr_chars > 100 → image-onlyocr_native_ratio = ocr_chars / max(1, native_chars). Flag if ratio > 3.keywords_in_metadata > 5 and absent from visible text → suspicious.
A b c). Flag if occurrences > 5.
Weights (binary scanner example): OCR/native ratio 40%, invisible_density 20%, white_spans 20%, image_pages_ratio 20%. Threshold: score ≥ 0.5 → manual review.
Example calculation:
// features normalized 0..1
ocr_native_norm = 0.9 // maps to 0.36 with weight
invis_norm = 0.02 // maps to 0.20 with weight
white_flag = 1.0 // maps to 0.20
image_ratio = 0.0 // maps to 0.0
score = 0.4*0.9 + 0.2*0.02 + 0.2*1.0 + 0.2*0.0
// score ≈ 0.76 → flag for manual review
native = extract_text(file)
ocr = ocr_all_pages(file)
invis = count_regex(native, ZWSP_REGEX)
ocr_native_ratio = (len(ocr)+1)/(len(native)+1)
invis_density = invis / max(1, len(native))
white_spans_flag = detect_white_spans(file) ? 1 : 0
image_pages_ratio = pages_with_images / total_pages
score = 0.4 * norm(ocr_native_ratio,1,10)
+ 0.2 * norm(invis_density,0,0.05)
+ 0.2 * white_spans_flag
+ 0.2 * image_pages_ratio
if score >= 0.5:
enqueue_manual_review(file, reason_codes)
else:
continue_parse_and_map_fields(file)
python-docx, PyMuPDF (fitz), pdfminer.sixTesseract (local) or Google/Azure OCR (scale)exiftool, custom PDF content-stream scannerocr/native ratio; flag if >3 or native==0 && ocr>100.Zero-width: [-]
Single-letter fragmentation (example): (?:[^s])+
PDF white ops (search): "rg 1 1 1" or "RG 1 1 1"
Direct, implementable: use three axes (native extraction, metadata, OCR), compute these ratios (OCR/native, invisible_char_density, tiny_font_fraction) and route scores above threshold to human review. This catches >95% of white-text and image-based evasions in practice.