What's the difference between knowledge-intensive and reasoning-intensive seed tasks?

Knowledge-intensive covers 39 tasks and about 3 million samples, focused on factual and domain knowledge recall. Reasoning-intensive covers 34 tasks and about 1.5 million samples, focused on logical deduction and multi-step thinking. They're handled separately so the mixed training data balances capabilities instead of skewing toward one dimension.

How much did Task-Seeded SDG improve Nemotron-3 Nano?

After mixing it into post-training at the ~100B token scale, GPQA jumped from 30.8 to 41.9 (+11.1), MMLU-Pro gained +1.8, coding ability gained +1.9, and common sense understanding gained +1.6. In ablation experiments, adding context pushed GPQA-Diamond CoT from 34.85 to 45.96 — a gap of over 11 points.

Why store answers as semantic text instead of option letters?

If answers are stored as just 'A, B, C, D' option letters, the model tends to memorize option position instead of actually understanding the content, and accuracy drops once question order changes. Storing semantic text (the full answer statement) forces the model to learn the underlying concept, which generalizes better — a design principle NVIDIA specifically emphasizes in the Nemotron pipeline.

What's a common mistake when building synthetic data?

The most common mistake is betting everything on a single task type to boost scores, or leaving task ratios unbalanced so the model overfits to one evaluation style. NVIDIA's experiments show that broad coverage across 70 tasks and 700 subtasks, with careful balancing of knowledge vs. reasoning ratios, is what lets multiple capability dimensions grow together — otherwise you risk gains on GPQA while coding or common sense regress.

How is Task-Seeded SDG different from typical LLM distillation data?

Typical distillation usually generates large volumes of similar QA pairs from a single strong model, which easily overfits to that generator's style. Task-Seeded SDG instead uses existing lm-eval-harness evaluation tasks as seeds, forcing coverage across knowledge, reasoning, coding, and common sense, then adds reasoning chains and domain annotations. It's essentially a methodology of 'deriving data structure from evaluation breadth.'

Can I apply this pipeline to my own model training or fine-tuning?

Yes, but first check whether your task seeds are diverse enough — don't bet on a single capability. A good approach is to pick 20-50 tasks from lm-eval-harness relevant to your target use case, split them into knowledge and reasoning buckets for generation, store answers as semantic text, and control the mix ratio. Even with under a million samples, structured design still beats blindly pumping up volume.

Task-Seeded Synthetic QA Data Generation for Nemotron Pretraining

Q: What is Task-Seeded SDG (Synthetic Data Generation)?

A five-stage data generation pipeline NVIDIA built for the Nemotron family. It picks about 70 public tasks (roughly 700 subtasks) from lm-eval-harness as seeds, uses a large language model to generate QA pairs that differ in content but match the seed tasks' skill level, appends reasoning chains and domain knowledge, then filters and packages everything for post-training.

This article is a deep-dive from JudyAI Lab — an AI engineering playbook series with 100+ published guides, 5,000+ weekly readers across 60+ countries, focused on the practical side of running AI agents, trading systems, and content pipelines in production.

📰 Key Takeaways

NVIDIA built a five-stage “Task-Seeded SDG” (Synthetic Data Generation) pipeline for the Nemotron model family: it pulls roughly 70 public tasks (about 700 subtasks) from lm-eval-harness, split into knowledge-intensive (39 tasks, ~3M samples) and reasoning-intensive (34 tasks, ~1.5M samples) seed categories, uses a large language model to generate QA pairs that differ in content but match the seed tasks’ skill level, then appends reasoning chains and domain knowledge before uniformly filtering and packaging the data. In ablation experiments, the context-enriched version won decisively: GPQA-Diamond CoT rose from 34.85 to 45.96 (+11.11), AGIEval-en CoT gained +6.16, and MMLU-Pro 5-shot gained +2.44. Mixing this synthetic data into Nemotron-3 Nano’s post-training (at the ~100B token scale) ultimately pushed GPQA from 30.8 to 41.9 (+11.1), with MMLU-Pro +1.8, coding ability +1.9, and common sense understanding +1.6 — multiple dimensions improving together, confirming that broad task coverage effectively prevents overfitting to a single evaluation style. Key design principles include: answers should be stored as semantic text rather than option letters, and when mixing datasets, task ratios must be carefully balanced to ensure stable, across-the-board gains in knowledge, reasoning, and coding capability.

💬 JudyAI Lab Take

NVIDIA’s five-stage “Task-Seeded SDG” pipeline for the Nemotron family is the first concrete demonstration of how a structured method can produce training data at scale — letting a small model improve across multiple evaluations at once, instead of just chasing a high score on one task.

What’s most worth paying attention to here is the deliberate split between “knowledge-intensive” and “reasoning-intensive” seed tasks, and the careful balancing of task ratios when mixing them into post-training. The ablation results are clear: the context-enriched version pushed GPQA-Diamond CoT from 34.85 to 45.96, a gap of over 11 points. That tells us synthetic data quality isn’t just about generation volume — it’s about structural design. Broad coverage across roughly 70 public tasks and 700 subtasks is exactly what prevents the model from overfitting to a specific evaluation style. The fact that coding ability, common sense understanding, and reasoning ability all improved together shows that task coverage breadth is itself an overfitting defense. One other detail worth remembering: answers should be stored as semantic text rather than option letters, so the model actually learns semantic understanding instead of memorizing option position.

If you’re adding synthetic training data to your own model or application, it’s worth asking first: are my task seeds diverse enough, or am I only betting on a single capability dimension?

📅 Source Info

Published: 2026-06-04T11:24
Original source: https://huggingface.co/blog/nvidia/task-seeded-sdg

Task-Seeded Synthetic QA Data Generation for Nemotron Pretraining

📰 Key Takeaways

💬 JudyAI Lab Take

📅 Source Info

🔗 Further Reading

References

📰 Key Takeaways#

💬 JudyAI Lab Take#

📅 Source Info#

🔗 Further Reading#

References#

Get our weekly AI digest:

📰 Key Takeaways

💬 JudyAI Lab Take

📅 Source Info

🔗 Further Reading

References