Streaming dataset validation, unambiguous syntax repair, conversational sequencing, SimHash deduplication, and PII scrubbing for Hugging Face, ChatML, and enterprise fine-tuning pipelines. Cleans JSONL, JSON, and CSV files with guaranteed reproducibility.
$ dataset-doctor audit --input training_corpus.jsonl [STREAM] Parsing 100,000 records via streaming buffer (25 MB cap)... [SYNTAX] Repaired 142 trailing commas & unbalanced brackets [ROLES] Normalized 89 casing typos: 'human'→'user', 'bot'→'assistant' [PII] Redacted 24 API keys, 18 credit cards (Luhn-verified) [DEDUPE] 64-bit SimHash Hamming index (distance <= 3): • Exact duplicates removed: 312 • Near-duplicates clustered: 148 [SPLIT] 80/10/10 Train/Val/Test generated: ZERO cross-split leak [SEMANTIC] Semantic instruction audit: max 100 sampled records (0 contradiction) --------------------------------------------------------------- STATUS: COMPLETE · SHA-256 Manifest emitted · Quality PDF ready
Recovers invalid JSONL syntax: strips trailing commas, normalizes single quotes, repairs unbalanced brackets via stack analysis, and cleans control characters without hallucinating content.
{'instruction': 'How do I invert a binary tree in Python?', 'output': 'To invert a binary tree, swap left and right children recursively.',}
{"instruction": "What is the speed of light?", "output": "Approximately 299,792,458 meters per second.",\x00}
{"instruction": "Explain recursion", "output": "A function calling itself until base case."
{"instruction": "How do I invert a binary tree in Python?", "output": "To invert a binary tree, swap left and right children recursively."}
{"instruction": "What is the speed of light?", "output": "Approximately 299,792,458 meters per second."}
{"instruction": "Explain recursion", "output": "A function calling itself until base case."}
ARCHITECTURAL RIGOR SIX DETERMINISTIC ENGINES
Unambiguous JSONL recovery handles trailing commas, unbalanced brackets via stack analysis, single quotes, and control character stripping. Broken records that cannot be proven valid are cleanly isolated into rejected logs.
Neutralizes formula execution attacks in spreadsheet data. Prepends safe quote escapes to dangerous prefix characters (=, +, -, @, \t, \r) to prevent DDE code execution during downstream review.
Enforces strict role alternation (system → user → assistant), guarantees system instructions are anchored at index 0, verifies assistant completion on trailing turns, and fixes role casing errors.
64-bit SimHash Hamming distance with 8-bit chunk pigeonhole indexing. Eliminates boilerplate duplication, repeated web crawls, and synthetic sample spam in O(1) candidate lookup time.
High-entropy regex and validation algorithms detect AWS, OpenAI, and GitHub tokens, RFC-compliant email, phone numbers, Luhn-verified credit cards, SSNs, and IPv4 addresses before training starts.
Generates deterministic 80/10/10 Train/Val/Test splits with built-in contamination detection. Proves that no validation or test evaluation prompt has leaked into the fine-tuning training set.
Deterministic checks run in pure memory with strict tenant isolation. Optional semantic audits (for contradictory instructions and prompt injection detection) combine deterministic data-quality checks with isolated language-model analysis. To guarantee execution deadlines within our 600-second worker timeout, semantic audits execute in bounded batches of up to 100 sampled semantic records per job (~20.7 records/minute measured throughput).
{
"engine_version": "ELYRAON Dataset Doctor v1.0",
"execution_engine": "isolated_semantic_engine_v1",
"limits": {
"max_upload_size_bytes": 26214400,
"max_semantic_records": 100,
"retention_days": 30
},
"input_dataset": {
"records": 100000,
"sha256": "6d7715631013d80e20282da775746c4e..."
},
"outputs": {
"cleaned_dataset": {
"records": 98452,
"sha256": "8a94bc72ef83b562140a7dc380126a1b..."
},
"rejected_records": {
"records": 1548,
"sha256": "e3b0c44298fc1c149afbf4c8996fb924..."
},
"change_log": {
"modifications": 3412,
"sha256": "9b12cf510a7493bce15024738dafe19a..."
}
},
"deterministic_verification": "Reproducible via identical seed"
}
TRANSPARENT MONETIZATION CREDIT-BASED SYSTEM
1 Credit cleans 10,000 records. Preview scans up to 1,000 records are 100% free with no credit card required.
Starter Pack ($10 for 10 credits)
For individual researchers and teams launching initial model fine-tunes.
For production fine-tuning pipelines and continuous RLHF evaluation cycles.
For data engineering teams and enterprise foundation model training.
Upload your training corpus for deterministic syntax repair, deduplication, and PII scrubbing.
Deterministic parsing for JSONL, JSON array, and CSV formats (up to 25 MB max).
Download real artifacts produced by ELYRAON Dataset Doctor test suites on this host.
12 instruction records ingested, 9 cleaned, 3 defective rejected, syntax recovered.
Download Quality PDF ↓7 records scanned, 2 sanitized, AWS keys, emails, Luhn credit cards, and SSNs redacted.
Download Privacy PDF ↓15 conversations partitioned, role sequences enforced, zero cross-split leakage verified.
Download Splits PDF ↓Run a free 1,000-record scan with immediate visual reports. Zero credit card required to inspect formatting errors, duplicates, and leakage. Dataset Doctor is currently accepting teams for our controlled commercial pilot (Stripe test mode verified; live billing held for owner launch review).
Deployment Note: Enterprise customer onboarding and authenticated dataset ingestion run at doctor.elyraon.com.co over HTTPS with strict 25 MB max body size limits.