ultra-feedback-dutch-cleaned-hq-spin-geitje-7b-ultra-sft_iter1
This model is a fine-tuned version of davidberenstein1957/ultra-feedback-dutch-cleaned-hq-spin-geitje-7b-ultra-sft_iter0 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.0380
- Rewards/real: -5.1867
- Rewards/generated: -23.6116
- Rewards/accuracies: 0.9778
- Rewards/margins: 18.4250
- Logps/generated: -690.4515
- Logps/real: -469.2089
- Logits/generated: -1.6815
- Logits/real: -2.1280
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 2
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/real | Rewards/generated | Rewards/accuracies | Rewards/margins | Logps/generated | Logps/real | Logits/generated | Logits/real |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.591 | 0.04 | 25 | 0.4210 | -0.2501 | -1.0788 | 0.8500 | 0.8287 | -465.1227 | -419.8426 | -2.6984 | -2.7096 |
| 0.2223 | 0.08 | 50 | 0.2173 | -0.5659 | -3.0876 | 0.9176 | 2.5217 | -485.2113 | -423.0011 | -2.6306 | -2.6446 |
| 0.168 | 0.12 | 75 | 0.1532 | -0.7060 | -4.4771 | 0.9435 | 3.7711 | -499.1060 | -424.4022 | -2.5832 | -2.6005 |
| 0.1126 | 0.16 | 100 | 0.1218 | -1.2746 | -6.3162 | 0.9509 | 5.0415 | -517.4969 | -430.0886 | -2.5961 | -2.6118 |
| 0.0854 | 0.21 | 125 | 0.0921 | -1.7944 | -9.0378 | 0.9611 | 7.2433 | -544.7130 | -435.2866 | -2.5534 | -2.5859 |
| 0.0609 | 0.25 | 150 | 0.0738 | -1.6860 | -9.1926 | 0.9639 | 7.5065 | -546.2610 | -434.2025 | -2.5875 | -2.6239 |
| 0.0654 | 0.29 | 175 | 0.0733 | -2.0360 | -9.8189 | 0.9648 | 7.7828 | -552.5237 | -437.7025 | -2.5252 | -2.5698 |
| 0.0814 | 0.33 | 200 | 0.0714 | -2.3341 | -10.2294 | 0.9630 | 7.8952 | -556.6287 | -440.6832 | -2.4634 | -2.5260 |
| 0.0356 | 0.37 | 225 | 0.0698 | -2.6697 | -11.4164 | 0.9667 | 8.7467 | -568.4990 | -444.0394 | -2.4311 | -2.5142 |
| 0.0641 | 0.41 | 250 | 0.0586 | -2.3926 | -12.3053 | 0.9694 | 9.9126 | -577.3877 | -441.2684 | -2.3106 | -2.4202 |
| 0.0442 | 0.45 | 275 | 0.0672 | -2.5170 | -11.9462 | 0.9676 | 9.4293 | -573.7975 | -442.5117 | -2.3880 | -2.4773 |
| 0.0707 | 0.49 | 300 | 0.0540 | -3.8488 | -15.1469 | 0.9667 | 11.2982 | -605.8044 | -455.8299 | -2.2564 | -2.3913 |
| 0.0683 | 0.53 | 325 | 0.0574 | -5.2977 | -18.2377 | 0.9667 | 12.9400 | -636.7123 | -470.3190 | -2.1402 | -2.3222 |
| 0.0339 | 0.58 | 350 | 0.0495 | -3.7486 | -17.2926 | 0.9731 | 13.5439 | -627.2608 | -454.8286 | -2.1701 | -2.3731 |
| 0.0648 | 0.62 | 375 | 0.0537 | -2.4302 | -13.2604 | 0.9722 | 10.8301 | -586.9390 | -441.6444 | -2.3167 | -2.4783 |
| 0.0358 | 0.66 | 400 | 0.0460 | -3.8509 | -17.3389 | 0.9741 | 13.4880 | -627.7241 | -455.8509 | -2.1735 | -2.3874 |
| 0.0532 | 0.7 | 425 | 0.0483 | -4.3261 | -18.2030 | 0.9741 | 13.8769 | -636.3655 | -460.6029 | -2.1550 | -2.3751 |
| 0.0408 | 0.74 | 450 | 0.0567 | -4.8885 | -19.7272 | 0.9741 | 14.8387 | -651.6073 | -466.2276 | -2.2982 | -2.4811 |
| 0.0434 | 0.78 | 475 | 0.0467 | -2.8677 | -16.1120 | 0.9731 | 13.2443 | -615.4548 | -446.0187 | -2.1937 | -2.4242 |
| 0.0194 | 0.82 | 500 | 0.0455 | -3.2473 | -18.4707 | 0.9769 | 15.2234 | -639.0422 | -449.8151 | -2.0107 | -2.3291 |
| 0.0227 | 0.86 | 525 | 0.0543 | -4.5805 | -20.1131 | 0.9750 | 15.5326 | -655.4664 | -463.1471 | -2.2146 | -2.4100 |
| 0.0299 | 0.91 | 550 | 0.0481 | -4.3021 | -20.3869 | 0.9731 | 16.0848 | -658.2037 | -460.3627 | -2.0552 | -2.3301 |
| 0.0218 | 0.95 | 575 | 0.0464 | -4.4619 | -20.3587 | 0.9713 | 15.8967 | -657.9220 | -461.9616 | -1.9225 | -2.2635 |
| 0.0218 | 0.99 | 600 | 0.0451 | -5.3210 | -20.9811 | 0.9722 | 15.6602 | -664.1465 | -470.5517 | -1.9518 | -2.2964 |
| 0.0093 | 1.03 | 625 | 0.0429 | -4.3395 | -19.2716 | 0.9750 | 14.9321 | -647.0515 | -460.7374 | -1.7575 | -2.1708 |
| 0.0173 | 1.07 | 650 | 0.0492 | -4.1317 | -19.0745 | 0.9704 | 14.9428 | -645.0802 | -458.6593 | -1.8155 | -2.1757 |
| 0.0059 | 1.11 | 675 | 0.0449 | -5.7336 | -23.1577 | 0.9713 | 17.4241 | -685.9126 | -474.6784 | -1.6844 | -2.1123 |
| 0.0149 | 1.15 | 700 | 0.0608 | -7.1484 | -26.1989 | 0.9713 | 19.0504 | -716.3237 | -488.8266 | -2.0142 | -2.2748 |
| 0.0105 | 1.19 | 725 | 0.0479 | -4.4948 | -20.2513 | 0.9722 | 15.7564 | -656.8477 | -462.2903 | -2.1674 | -2.3962 |
| 0.032 | 1.23 | 750 | 0.0512 | -5.0950 | -21.3230 | 0.9685 | 16.2280 | -667.5649 | -468.2917 | -2.2426 | -2.4414 |
| 0.0042 | 1.28 | 775 | 0.0462 | -4.0296 | -19.2620 | 0.9704 | 15.2324 | -646.9548 | -457.6381 | -2.2156 | -2.4379 |
| 0.0041 | 1.32 | 800 | 0.0475 | -4.0348 | -19.8410 | 0.9731 | 15.8062 | -652.7453 | -457.6903 | -2.1330 | -2.3843 |
| 0.0075 | 1.36 | 825 | 0.0428 | -4.4696 | -20.8584 | 0.9722 | 16.3888 | -662.9192 | -462.0378 | -2.1122 | -2.3718 |
| 0.004 | 1.4 | 850 | 0.0468 | -6.2822 | -25.6273 | 0.9750 | 19.3451 | -710.6078 | -480.1642 | -1.7240 | -2.1709 |
| 0.0222 | 1.44 | 875 | 0.0584 | -6.0399 | -23.0778 | 0.9759 | 17.0379 | -685.1132 | -477.7408 | -1.6544 | -2.1242 |
| 0.0063 | 1.48 | 900 | 0.0490 | -3.8721 | -19.8020 | 0.9722 | 15.9298 | -652.3550 | -456.0635 | -1.7696 | -2.2026 |
| 0.006 | 1.52 | 925 | 0.0478 | -5.2822 | -23.7504 | 0.9750 | 18.4682 | -691.8392 | -470.1639 | -1.6461 | -2.1239 |
| 0.0169 | 1.56 | 950 | 0.0455 | -4.9375 | -22.9431 | 0.9731 | 18.0057 | -683.7665 | -466.7169 | -1.6890 | -2.1447 |
| 0.0063 | 1.6 | 975 | 0.0449 | -5.9782 | -25.0564 | 0.9741 | 19.0782 | -704.8994 | -477.1242 | -1.5890 | -2.0779 |
| 0.0144 | 1.65 | 1000 | 0.0428 | -5.2622 | -22.9304 | 0.9731 | 17.6682 | -683.6391 | -469.9639 | -1.6262 | -2.0859 |
| 0.0046 | 1.69 | 1025 | 0.0411 | -5.5146 | -24.0845 | 0.9759 | 18.5698 | -695.1800 | -472.4886 | -1.6070 | -2.0934 |
| 0.002 | 1.73 | 1050 | 0.0408 | -5.4174 | -23.7610 | 0.9750 | 18.3436 | -691.9457 | -471.5163 | -1.6779 | -2.1277 |
| 0.0047 | 1.77 | 1075 | 0.0411 | -5.6837 | -24.5512 | 0.9750 | 18.8674 | -699.8467 | -474.1796 | -1.7048 | -2.1412 |
| 0.0077 | 1.81 | 1100 | 0.0404 | -5.8712 | -25.3478 | 0.9759 | 19.4766 | -707.8129 | -476.0543 | -1.6257 | -2.0917 |
| 0.0145 | 1.85 | 1125 | 0.0385 | -5.0758 | -23.2450 | 0.9741 | 18.1692 | -686.7853 | -468.0999 | -1.6509 | -2.1029 |
| 0.0038 | 1.89 | 1150 | 0.0376 | -5.2077 | -23.5236 | 0.9759 | 18.3159 | -689.5715 | -469.4194 | -1.6736 | -2.1249 |
| 0.01 | 1.93 | 1175 | 0.0379 | -5.1247 | -23.3484 | 0.9750 | 18.2238 | -687.8193 | -468.5888 | -1.6969 | -2.1383 |
| 0.0055 | 1.98 | 1200 | 0.0380 | -5.1867 | -23.6116 | 0.9778 | 18.4250 | -690.4515 | -469.2089 | -1.6815 | -2.1280 |
Framework versions
- Transformers 4.37.0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.2
- Downloads last month
- 88