#3232nd of 41 in Learning and cookbooksFor research, robotics and education
arc-task-gen
Generator of fresh ARC-AGI-1-style tasks for private model evaluation
- Stars
- 11k
- License
- MIT
- Last commit
- Aug 2026
- Language
- Python
3 on 4 Aug 202610,983 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Generates new ARC-AGI-1-style tasks that resemble the public dataset, written to a tasks.json in standard ARC train/test format. It gives a private evaluation set for comparing public-benchmark scores with scores on unseen tasks. It accompanies the BDH-CQ paper, which reports 29.5% pass@2 for a 150M-parameter model on the public ARC-AGI-1 evaluation set.
Who it is for: Researchers evaluating models on ARC-AGI-1-style reasoning tasks
Strengths
- Output follows standard ARC JSON format, compatible with existing ARC evaluation harnesses
- Produces tasks unlikely to be in model training data, useful for checking benchmark familiarity
- MIT licensed, written in Python
Weaknesses
- README gives no usage steps; setup is deferred to instructions.md
- README does not say how the generator works or what it needs to run
- Does not include the BDH-CQ model; the repo only generates tasks
- No releases published
What it needs
- Models: ARC task JSON
Also in Learning and cookbooks
See all 41| Rank | Project | Score |
|---|---|---|
| 1 | ai-engineering-from-scratchHands-on AI engineering curriculum with 542 lessons across 20 phases | 89 out of 100 |
| 2 | generative-ai-for-beginners21-lesson course on building generative AI apps in Python and TypeScript | 81 out of 100 |
| 3 | ai-agents-for-beginners18-lesson course on building AI agents with Microsoft Agent Framework | 78 out of 100 |
| #4 | llms-from-scratchCode for the book on building a GPT-style LLM in PyTorch | 77 out of 100 |
| #5 | ml-for-beginners12-week, 26-lesson classic machine learning course using Scikit-learn | 76 out of 100 |