Distill information

1. Overview

DeepSeek-R1 is an open-source AI model developed by a Chinese company, focusing on reasoning capabilities and designed to compete directly with OpenAI o1. In various tests, DeepSeek-R1 demonstrated performance on par with or exceeding OpenAI o1, especially in mathematical tasks and multi-step reasoning.

However, OpenAI o1 holds an advantage in certain areas such as programming and general knowledge. Notably, DeepSeek-R1 is significantly more cost-effective and allows community customization, whereas OpenAI o1 remains a proprietary model.

2. Detailed Test Results

2.1. Mathematics

2.2. Programming

2.3. General Knowledge

3. Distilled Models Performance

Purpose of Distillation:

• Distillation in AI refers to the process of creating smaller, more efficient models from larger ones, retaining most of their reasoning capabilities while reducing computational requirements. DeepSeek has applied this technique to create a series of distilled models from R1, utilizing the Qwen and Llama architectures.

Distilled Models Based on Qwen:

• DeepSeek-R1-Distill-Qwen-1.5B:
  • The smallest distilled model, achieving 83.9% on MATH-500.
  • Low performance on LiveCodeBench (16.9%), indicating limited programming capabilities.
  • Suitable for basic mathematical tasks but not strong in coding.
• DeepSeek-R1-Distill-Qwen-7B:
  • High score on MATH-500 (92.8%), demonstrating strong mathematical reasoning.
  • Moderate performance on GPQA Diamond (49.1%), balancing math and factual knowledge.
  • Not optimized for complex coding tasks (LiveCodeBench 37.6%, CodeForces 1189 rating).
• DeepSeek-R1-Distill-Qwen-14B:
  • Strong performance on MATH-500 (93.9%), handling complex mathematical problems.
  • Score of 59.1% on GPQA Diamond, showing reasoning based on factual knowledge.
  • Performance on LiveCodeBench (53.1%) and CodeForces (1481 rating) indicates room for improvement in coding tasks.
• DeepSeek-R1-Distill-Qwen-32B:
  • The largest Qwen-based model, achieving the highest score on AIME 2024 (72.6%), excelling in advanced multi-step reasoning.
  • Outstanding results on MATH-500 (94.3%) and GPQA Diamond (62.1%), showing strength in both mathematical reasoning and factual knowledge.
  • Results on LiveCodeBench (57.2%) and CodeForces (1691 rating) indicate flexibility but not optimal coding performance.

Distilled Models Based on Llama:

• DeepSeek-R1-Distill-Llama-8B:
  • Strong performance on MATH-500 (89.1%) and moderate on GPQA Diamond (49.0%).
  • Lower scores on coding benchmarks like LiveCodeBench (39.6%) and CodeForces (1205 rating), indicating limitations in programming tasks.
• DeepSeek-R1-Distill-Llama-70B:
  • The largest distilled model, excelling in MATH-500 (94.5%), the best among all distilled models.
  • Achieved 86.7% on AIME 2024, making it an excellent choice for advanced mathematical reasoning.
  • Strong coding performance (LiveCodeBench 57.5%, CodeForces 1633 rating), comparable to OpenAI o1-mini or GPT-4o.

4. Operating Cost

• DeepSeek-R1 is significantly cheaper across all cost categories.
• Example: Output costs for DeepSeek-R1 are $2.19 per million tokens, compared to $60.00 per million tokens for OpenAI o1.
• This makes DeepSeek-R1 a better option for cost-sensitive applications, including large-scale AI projects or startups with limited budgets.

5. DeepSeek Deployment Requirements