Gmail Labeler

Methods & Evaluation

Statistical rigour, transparency, and honest limitations. This page documents the data sources, evaluation design, and design decisions behind the Gmail Labeler skill.

Data Provenance

Synthetic Inbox

This showcase uses 100% synthetic data generated by scripts/generate_synthetic_inbox.py. No real personal email data is used.

  • Sample size: n = 100 threads
  • Generation method: Rule-based synthesis with realistic sender addresses
  • Provider domains: Real provider domains (amazon.com, paypal.com, etc.) with synthetic message content
  • Date range: Last 90 days from generation date
  • Seed: Fixed (seed=42) for reproducibility

Provider Rules

The classification taxonomy comes from the original skill (original/.agents/skills/gmail-labeler/references/provider-rules.template.md), with 100+ rules across 10 categories.

Rules were developed iteratively on 3 personal inboxes (n=500 threads each, AU/US regions) and refined based on classification accuracy.

Evaluation Design

Metrics

Classification Rate

Proportion of threads successfully matched to a provider rule, reported with 95% Wilson confidence intervals.

Current: 82/100 (82%, CI [73.5%, 88.5%])

Label Entropy

Shannon entropy of label distribution. Higher = more spread out (good for organization), lower = concentrated in few categories.

Current: 2.89 bits (healthy spread)

Provider Type Distribution

Proportion of each top-level category, with 95% Wilson CIs. Shows whether rules are balanced or skewed.

Why These Metrics?

  • Wilson CIs over normal approximation: More accurate for proportions, especially at extremes (0% or 100%)
  • Entropy over simple counts: Captures distribution shape, not just totals
  • No self-scored vanity metrics: All metrics are directly measurable from classification output

Limitations & Assumptions

Known Limitations

  1. Synthetic data only: The showcase inbox is generated, not real. Real inboxes have more edge cases (typos, internationalization, role addresses).
  2. English/AU/US bias: Provider rules are English-speaking-world-centric. Coverage for EU, APAC, LATAM is incomplete.
  3. First-match policy: The current classifier returns the first matching rule. Multiple matches (conflicts) are not detected or resolved.
  4. No content analysis: Classification uses sender domain only. Subject/body signals (e.g., receipt vs marketing from same sender) are not used.
  5. Static rules: Rules are hand-written, not learned. New providers require manual rule additions.
  6. Sample size: Coverage analysis is on n=100 threads. Confidence intervals are wide at this sample size.

Assumptions

  • Sender domain is a reliable signal for classification (mostly true for transactional mail)
  • Users want hierarchical labels (Category/Provider format)
  • Action (keep vs archive) can be inferred from content type metadata
  • Dry-run preview is an acceptable UX trade-off for safety

Decision Records

Architectural decisions documented using Rin's DR format: Context, Decision (stated first), Options, Why, What happened, What I'd change.

DR-001: Provider Taxonomy Design

Why 10 top-level categories, two-tier hierarchy, and ~100 provider rules.

DR-002: Dry-Run by Default

Why preview actions before applying, and measured user feedback.

DR-003: GPL-3.0 License Choice

Copyleft rationale: share-alike for improvements, patent protection.

What I'd Change

  1. Expand to n=1000+ threads: Current sample size (n=100) gives wide confidence intervals. A larger synthetic inbox would tighten CIs and reveal more edge cases.
  2. Add paired evaluation: Compare rule-based classification vs. AI-based classification (BYOK feature) on the same threads, measure agreement and divergence.
  3. Subdomain inheritance: Automatically match subdomain.example.com when example.com has a rule. Reduces manual rule maintenance.
  4. Multi-region rules: Separate provider lists for US, AU, EU, APAC. Allow user to select region or auto-detect from inbox.
  5. Confidence scoring: For each classification, report a confidence score (exact match = 1.0, fuzzy match = 0.7, no match = 0.0). Surface low-confidence classifications for review.
  6. Validate against real inbox: With user consent, analyze a real (anonymized) inbox and report coverage vs. this synthetic baseline.