519. Feedback Prize - English Language Learning | feedback-prize-english-language-learning
我们的团队集成了下表中的26个模型（包括知识蒸馏和伪标签模型）。我们使用岭回归来整合每个模型的预测结果，并对最终集成的预测输出值进行了后处理调整。对于单模型，以下技术有助于提高分数：
| 序号 | 模型 | CV / LB | 描述 |
|---|---|---|---|
| 1 | Deberta v3 base | 0.448 | 最大长度512，伪标签推断自TensorFlow Deberta v3 base (LLRD技术)，结合比赛数据在Deberta v3 base上训练 (freezing+meanpooling) |
| 2 | Deberta v3 large | 0.448 | 同上，Deberta v3 large 训练于 meanpooling + freezing |
| 3 | Deberta v3 small | 0.447 | 知识蒸馏模型 |
| 4 | Deberta v2 xlarge | 0.448 / 0.44 | LSTM + mean pooling + freezing (相同的TensorFlow伪标签) |
| 5 | Deberta v3 large | 0.4495 / 0.43 | 最大长度 = 470，训练时递减最大长度 - 768/512/470/470 |
| 6 | Deberta v3 base | 0.4515 / 0.43 | 最大长度 =470, Mean pooling, AWP, val_steps = 250, 重初始化层 |
| 7 | Deberta xlarge | 0.4534 / 0.43 | CLS token, val steps =20 |
| 8 | deberta v3 large | 0.4495 / 0.43 | CLS Token, AWP, val steps = 250, 最大长度 768 |
| 9 | Electra large | 0.4545 | Meanpool, LSTM |
| 10 | deepset/deberta-v3-base-squad2 | 0.4522 | |
| 11 | Luke Large | 0.4551 | 最大长度 512, Mean pooling, bidirectional-LSTM |
| 12 | Deberta v3 large | 0.4467 | |
| 13 | Deberta v3 base | 0.4471 | AutoModelForTokenClassification, 最后4层拼接, mean pooling, max_len=768 |
| 14 | Deberta v3 large | 0.4469 | PL Model, 2个种子平均, CLS Token |
| 16 | Cocolm large | 0.4568 | 无 [PARAGRAPH], CLS Token, AWP |
| 17 | GPT2 Medium | 0.4648 | max_len=1024, Mean pooling, SWA |
| 18 |