第 12 名解决方案

我想感谢 Kaggle 平台和 Eedi 举办这次竞赛。在整个活动期间，我学到了很多关于召回和排序技术的知识，特别是如何利用大模型进行嵌入对比学习和重排序。最终，我获得了个人金牌，这让我离成为 Grand Master 更近了一步。

概述

主要策略涉及一个检索器（Retriever）结合一个重排序器（Reranker）。

合成数据生成

使用思维链（Chain of Thought, CoT）方法，我基于误解生成了四批合成数据，总计超过 10,000 个外部数据点。提示词（Prompt）如下：

You are an intelligent math problem-solving expert.
I will provide you with a misconception-related error reason. Based on this error, generate a math single-choice question along with four options, where only one is correct, and the other three are distractors.
The error reason provided corresponds to one of the incorrect options among the three distractors.
Below are several examples:
Example 1:
Input: Confuses the order of operations, believes addition comes before multiplication
Output: [question]
ConstructName:Use the order of operations to carry out calculations involving powers
SubjectName:BIDMAS
QuestionDesc:\\[
3 \\times 2+4-5
\\]
Where do the brackets need to go to make the answer equal \\( 13 \\) ?

[Candidate answers]
A.\\( 3 \\times(2+4)-5 \\)
B.\\( 3 \\times 2+(4-5) \\)
C.\\( 3 \\times(2+4-5) \\)
D.Does not need brackets

[Correct Answer]
\\( 3 \\times(2+4)-5 \\)

[Misconcepte Incorrect answer]
Does not need brackets

Example 2:
Input: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term
Output: [question]
ConstructName:Simplify an algebraic fraction by factorising the numerator
SubjectName:Simplifying Algebraic Fractions
QuestionDesc:Simplify the following, if possible: \\( \\frac{m^{2}+2 m-3}{m-3} \\)

[Candidate answers]
A.\\( m+1 \\)
B.\\( m+2 \\)
C.\\( m-1 \\)
D.Does not simplify

[Correct Answer]
Does not simplify

[Misconcepte Incorrect answer]
\\( m+1 \\)

Example 3:
Input: Thinks that when you cancel identical terms from the numerator and denominator, they just disappear
Output: [question]
ConstructName:Simplify an algebraic fraction by factorising the numerator
SubjectName:Simplifying Algebraic Fractions
QuestionDesc:Simplify the following, if possible: \\( \\frac{m^{2}+2 m-3}{m-3} \\)

[Candidate answers]
A.\\( m+1 \\)
B.\\( m+2 \\)
C.\\( m-1 \\)
D.Does not simplify

[Correct Answer]
Does not simplify

[Misconcepte Incorrect answer]
\\( m+2 \\)

Example 4:
Input: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term
Output: [question]
ConstructName:Simplify an algebraic fraction by factorising the numerator
SubjectName:Simplifying Algebraic Fractions
QuestionDesc:Simplify the following, if possible: \\( \\frac{m^{2}+2 m-3}{m-3} \\)

[Candidate answers]
A.\\( m+1 \\)
B.\\( m+2 \\)
C.\\( m-1 \\)
D.Does not simplify

[Correct Answer]
Does not simplify

[Misconcepte Incorrect answer]
\\( m-1 \\)

Please generate similar questions and options based on the cases above.
You first need to think about the reasoning logic according to the chain of thought, then provide the specific case.
The output format is:
<thought>Insert your thoughts on the error reason here</thought>
<output>Insert your question and options here</output>

外部数据帮助将召回率从 0.430 提高到了 0.470（单折）。

检索器 (Retriever)

整体方法涉及最初使用 SFR 模型检索 100 个候选项，随后使用 Qwen 模型训练对比学习。三个模型的集成是通过拼接嵌入实现的：emb=concat(emb1,emb2,emb3)。

• 使用 Qwen2.5-32b-instruct-awq 推断潜在错误，结合原始查询训练 Qwen2.5-14B-instruct。平均 5 折 LoRA 权重以获得 emb1。
• 使用原始查询训练 Qwen2.5-14B-instruct，并平均 5 折 LoRA 权重以获得 emb2。
• 使用来自 notebook 的开源模型作为 emb3。
• 公共分数：0.526，私有分数：0.485

重排序器 (Reranker)

对于检索器检索到的前 50 个候选项（来自 5 折 OOF 的结果），每次随机选择 25 个进行 25 类预测。最终分数是通过集成三个模型的概率权重实现的：

• 公共分数：0.597，私有分数：0.564
• Qwen2.5-32b-instruct-awq 原始查询 + vllm 推理：平均 5 折 LoRA 权重，公共分数：0.570
• Qwen2.5-14b-instruct 原始查询：平均 5 折 LoRA 权重，公共分数：0.567
• Qwen2.5-14b-instruct 查询 + 原因：平均 5 折 LoRA 权重，公共分数：0.558

无效尝试 (What Didn't Work)

• Qwen2.5-coder
• 用于 OOF 后重排序的 LGB/CatBoost
• 重排序器查询 + 误解 last_pool => 0/1