ãã®èšäºã¯ãããŒãã¿ã€ã ãšã³ãžãã¢ã®å·äžèªãããšLLMå¿çšã«åãçµãã§ãããšã³ãžãã¢ã®éŽæšæžå€ªãšãªãµãŒãã£ãŒã®å²©æŸ€è«äžéã«ããå¯çš¿ã§ãã
æŠèŠ
- æ¢åã®å€§èŠæš¡èšèªã¢ãã« (LLM) ã«å¯ŸããŠå»çãã¡ã€ã³ã®ç¬èªã³ãŒãã¹ãçšãã finetuning ãè¡ããå»çç¹åã®å€§èŠæš¡èšèªã¢ãã« Llama3-Preferred-MedSwallow-70B ãéçºããŸããã
- Llama3-Preferred-MedSwallow-70B ã¯æ¥æ¬å»åž«åœå®¶è©Šéšã«ãããŠã GPT-4 ãäžåãæçžŸãåããHugging Face Hub ã§å ¬éãããŠããã¢ãã«ã®äžã§æé«æ§èœãèªããŸãã
- PFN ã§ã¯å»çã»ãã«ã¹ã±ã¢ãå«ãæ§ã ãªåéã§ã® LLM 掻çšãç®æããåŒãç¶ãç ç©¶éçºãé²ããŠãããŸãã
ã¯ããã«
å€§èŠæš¡èšèªã¢ãã« (Large Language Model, LLM) ã¯å»çããã«ã¹ã±ã¢ã®åéã§èšºæè£å©ããã¬ããžæœåºãæèŠã®èŠçŽãªã©å€å²ã«ãããã¿ã¹ã¯ã«ãããŠä»åŸå€§ããªåœ¹å²ãæããããšãæåŸ ãããŠããŸããããããå»çåéåã LLM ã®æ§ç¯ã¯ããç°¡åã§ã¯ãããŸãããããšãã°æ¥æ¬å»åž«åœå®¶è©Šéšã®5幎åã®åé¡ããŸãšãã IgakuQA [1] ãšåŒã°ãã LLM ã®ãã³ãããŒã¯ã§ã¯ GPT-4 ã§ããåæ Œç¹ã®ãªã®ãªã§ãããå®éã®è©Šéšåæ Œè ã«æ¯ã¹ããš LLM ã®ç²ŸåºŠã¯ãŸã ãŸã å£ã£ãŠããããšãåãã£ãŠããŸãã
PFN ã§ã¯ãäºååŠç¿ããèªç€Ÿã§è¡ãªã£ãŠãã PLaMo [2] ã«å ããŠãæ§ã ãªã¢ãã«ã® finetuning ã«ãåãçµãã§ããŸããç¹ã«å»çã»ãã«ã¹ã±ã¢ã®é åã§ã¯ã»ã³ã·ãã£ããªããŒã¿ãå€ãããã«å šãŠã®ããŒã¿ãã¢ãã«ã®äºååŠç¿ (pretraining) ã§äœ¿çšããŠããŸãã®ã¯ãªã¹ã¯ãé«ããæ¯èŒç容æã«ããŒã¿ãã«ã¹ã¿ãã€ãºããŠååŠç¿ãå¯èœãª finetuning ã§å¯Ÿå¿ã§ããããã«ããããšã«ã¯å€§ããªæå³ããããŸãã
ä»åãç§ãã¡ã¯ç¶ç¶äºååŠç¿ãšåŒã°ãã finetuning ææ³ãçšããŠãæ¢åã® LLM ãå»çåãã«æ¹è¯ãã Llama3-Preferred-MedSwallow-70B ãéçºããŸãããLlama3-Preferred-MedSwallow-70B ã¯ç§ãã¡ã®ç¥ãéã Hugging Face Hub äžã§å ¬éãããŠããã¢ãã«ãšããŠåããŠæ¥æ¬å»åž«åœå®¶è©Šéšã§ GPT-4 ãè¶ ããã¢ãã«ã§ãããéã¿ãå ¬éãããŠãã LLM ã®äžã§å¯äžåæ Œç¹ãéæããã¢ãã«ã§ããæ¬èšäºã§ã¯ãã¢ãã«ã®éçºææ³ãšæ§èœã®è©³çްã«ã€ããŠè§£èª¬ããŸãããªããLlama3-Preferred-MedSwallow-70B 㯠Hugging Face Hub ã§å ¬éãããŠãããããèå³ããæã¡ã®æ¹ã¯ãã²ãèªèº«ã§è©ŠããŠã¿ãŠãã ããã
https://huggingface.co/pfnet/Llama3-Preferred-MedSwallow-70B Â
(ã¢ãã«ã¯ç ç©¶ç®çã§éçºããããã®ã§ããã蚺çç®çã§ã®äœ¿çšã¯ãæ§ããã ããã)
LLM ã® finetuning ã«ã€ããŠ
ç¶ç¶äºååŠç¿
LLM ã® finetuning ã«ã¯ãæç€ºåŠç¿ (Instruction Tuning) [3] ãç¶ç¶äºååŠç¿ (Continued Pretraining) [4] ãªã©è€æ°ã®ææ³ããããŸããæç€ºåŠç¿ã§ã¯ç¹å®ã®æç€ºãšãã®è¿çã®ãã¢ãçšããŠåŠç¿ãè¡ããã¢ãã«ããŠãŒã¶ã«ãšã£ãŠãã奜ãŸããè¿çãçæããããšãç®æããŸããæç€ºåŠç¿ã¯ LLM ãå ã æã£ãŠããç¥èã匷調ããããåºåã®ããŒã³ãæ§é ãå¶åŸ¡ããã®ã«åœ¹ç«ã€äžæ¹ã§ãæ°ããç¥èã®ç²åŸã«ã¯äžåãã ãšææãããŠããŸã [5]ã
äžæ¹ãç¶ç¶äºååŠç¿ã§ã¯æåéãäºååŠç¿ãšåæ§ã®åœ¢åŒã§ã¢ãã«ã®åŠç¿ãè¡ããããäºååŠç¿ã«å«ãŸããŠããªãæ°ããç¥èã«ã察å¿ã§ãããšãããŠããŸããæ¢åã® LLM ã¯æ¥æ¬èªã®å»çç¥èãäžè¶³ããŠãããšããå顿èãããä»åã®å®éšã§ã¯2017幎以åã®å»åž«åœå®¶è©Šéšã®è§£èª¬ãå«ãã PFN ç¬èªã®å»çããŒã¿ã»ãããçšããŠç¶ç¶äºååŠç¿ãè¡ããŸããã
å»çåéã«ç¹åãã LLM ãäœæããã«ããã£ãŠãããŒã¹ã®ã¢ãã«ã«ã¯ Llama-3-Swallow-70B [6] ãéžã³ãŸãããæ±å·¥å€§ãšç£ç·ç ã®ååã°ã«ãŒãã«ãã£ãŠéçºããã Llama-3-Swallow-70B ã¯ã Meta 瀟㮠Meta-Llama-3-70B ã«å¯ŸããŠç¶ç¶äºååŠç¿ãè¡ã£ãŠæ¥æ¬èªèœåã匷åããã¢ãã«ã§ãHugging Face Hub ã§å ¬éãããŠããã¢ãã«ã®äžã§ãé«ãæ¥æ¬èªèœåãæã£ãã¢ãã«ãšãªã£ãŠããŸãããã® Llama-3-Swallow-70B ã«å¯ŸããŠãããªãç¶ç¶äºååŠç¿ãšåŸè¿°ã® QLoRA ãçµã¿åãããŠåŠç¿ãè¡ãããšã§ãLlama3-Preferred-MedSwallow-70B ãéçºããŸããã
QLoRA
ä»åã¢ãã«ã®ç¶ç¶äºååŠç¿ãè¡ãã«ããã£ãŠãQLoRA [7] ãçšããŸãããQLoRA ã¯éåå (quantization) [8] ãš LoRA [9] ãçµã¿åãããææ³ã§ãããã·ã³ãã«ã§ãããªããå€§èŠæš¡èšèªã¢ãã«ã® finetuning ãããå°ãªãèšç®ãªãœãŒã¹ã§è¡ããç¹ã§éåžžã«åŒ·åã§ããäžè¬çã«ã LLM ããã®ä»ã®æ·±å±€åŠç¿ã¢ãã«ã§ã¯ãåãã©ã¡ãŒã¿ã¯16ãããã 32ãããã§è¡šçŸãããŸããäžæ¹ãéååãããã¢ãã«ã§ã¯åãã©ã¡ãŒã¿ã4ããããªã©ã®äœããããæ°ã§è¡šçŸããããããããå°ããã¡ã¢ãªã§æ±ãããšãå¯èœã«ãªããŸããéååãè¡ã£ãŠãå€§èŠæš¡èšèªã¢ãã«ã®æšè«èœåã¯å€§ããäœäžããªãããšãç¥ãããŠãã [8]ãéååã¯å€§èŠæš¡èšèªã¢ãã«ãæè»œã«äœ¿ãäžã§æ¬ ãããªãææ³ãšãªã£ãŠããŸãã
ããããäžåºŠéååããŠããŸã£ãã¢ãã«ã¯åŠç¿æã«åŸé ããããããšããã§ãŒãã«ãªã£ãŠããŸãããã«ããŸãåŠç¿ã§ããªããšããåé¡ããããŸããããã解決ããããã« Dettmers ãã¯éååãããã¢ãã«ã§ãå¹ççã«åŠç¿ãè¡ããããã« LoRA ãçµã¿åãããåŠç¿æ¹æ³ãææ¡ããŸãã [7]ãLoRA (Low-Rank Adaptation) ãšã¯å€§èŠæš¡èšèªã¢ãã«ã® finetuning ã«ãããäœæ¬¡å æ§ [10] ã«çæ³ãåŸãææ³ã§ãå ã®ã¢ãã«ã®ãã©ã¡ãŒã¿ã倿Žããã«ãã¢ãã«ã«è¿œå ãããã¢ããã¿ãŒãšåŒã°ããäœã©ã³ã¯ã®è¡åã®ã¿ãåŠç¿ããããšã§ finetuning ãè¡ããŸã [9]ã Dettmers ãã¯ãéååããã¢ãã«ãšéååããŠããªã LoRA ã¢ããã¿ãŒãçµã¿åãããŠåŠç¿ããããšã§ãéåžžã«è»œéãªåœ¢ã§å€§èŠæš¡èšèªã¢ãã«ã® finetuning ãå®çŸã§ããããšã瀺ããŸãããããã QLoRA ã§ãã
QLoRA ã¯ãéååã LoRA ãçšããªã full parameter finetuning ãšæ¯ã¹ããšç²ŸåºŠãå£ããšããå ±åããããŸãã [11]ãå°ãªãæ°ã® GPU ã§ finetuning ãè¡ããæè»œãããåºãå©çšãããŠããŸããå®éãä»åã® QLoRA ãçšããç¶ç¶äºååŠç¿ã瀟å ã®èšç®æ©ã¯ã©ã¹ã¿ãŒã® A100 GPU 2å°ã®ã¿ã§å®çŸã§ããŠããŸãã
å»åž«åœå®¶è©Šéšã§ã®çµæ
éçºããã¢ãã«ã®æ§èœãæ¥æ¬å»åž«åœå®¶è©Šéšã§è©äŸ¡ããããã« IgakuQA [1] ã䜿çšããŸãããIgakuQA 㯠Jungo Kasaiãããã«ãã£ãŠéçºããã LLM çšã®ãã³ãããŒã¯ã§ããã2018~2022å¹Žã®æ¥æ¬å»åž«åœå®¶è©Šéšã®åé¡ã LLM ã«è§£ãããããšãã§ããŸããIgakuQA ã«ã¯æ¥æ¬èªã®åæã®åé¡ãšè±èªã«ç¿»èš³ãããåé¡ã®äž¡æ¹ãå«ãŸããŠããŸãããä»åã¯æ¥æ¬èªã§ã®å»çç¥èãèœåã«ãã©ãŒã«ã¹ãããããæ¥æ¬èªåé¡ã«å¯ŸããæçžŸãè©äŸ¡ããŸããããŸããå ã ã® IgakuQA ã®å®è£ ã§ã¯ Hugging Face Hub äžã®ã¢ãã«ãè©äŸ¡ã§ããªãã£ãããã«ç€Ÿå ã§è¿œå å®è£ ãè¡ããŸããããããã³ããçã®èšå®ã¯å ã®å®è£ ã«åŸã£ãŠããŸãã
çµæã¯ä»¥äžã®è¡š1ã«ãŸãšããŠããŸãã
| ã¢ãã«å | å¹³å | 2018 | 2019 | 2020 | 2021 | 2022 |
| Llama3-Preferred-MedSwallow-70B | 395.2 | 407 | 390 | 391 | 393 | 395 |
| GPT-4 | 388.8 | 382 | 385 | 387 | 398 | 392 |
| Llama-3-Swallow-70B-v0.1 | 348.6 | 353 | 347 | 353 | 345 | 345 |
| Meta-Llama-3-70B | 334.6 | 353 | 340 | 348 | 314 | 318 |
| Qwen2-72B | 331.2 | 320 | 325 | 325 | 326 | 360 |
| c4ai-command-r-plus | 316.2 | 321 | 303 | 320 | 302 | 335 |
| gemma-2-27b | 316 | 337 | 298 | 327 | 296 | 322 |
| Swallow-70b-NVE-hf | 291.6 | 283 | 280 | 300 | 295 | 300 |
| Swallow-MX-8x7b-NVE-v0.1 | 282.2 | 269 | 285 | 290 | 277 | 290 |
| ChatGPT | 273.2 | 266 | 250 | 266 | 297 | 287 |
衚1. æ¥æ¬å»åž«åœå®¶è©Šéš (IgakuQA) ã«ãããåã¢ãã«ã®æçžŸã2018~2022幎ã®åå¹ŽåºŠã®æçžŸãšãããã®å¹³åãèšèŒããŠããŸããGPT-4/ChatGPT ã®æçžŸã¯ [1] ããåŒçšããLlama3-Preferred-MedSwallow70B ãšãã®ä»ã®ã¢ãã«ã®æçžŸã¯ç€Ÿå ã§ã®æž¬å®çµæãçšããŠããŸãã
ä»åéçºãã Llama3-Preferred-MedSwallow-70B ã¯2018幎ãã2022幎ã®å»åž«åœå®¶è©Šéšã§å¹³å395.2ç¹ãååŸãããªãŒãã³ãªã¢ãã«ãšããŠåããŠGPT-4ãè¶ ããæçžŸãåããŸãããããã«ãå ¬éãããŠããã¢ãã«ã®äžã§ç§ãã¡ã®ç¥ãéãåããŠå»åž«åœå®¶è©Šéšã§åæ Œç¹*ãéæããã¢ãã«ãšãªã£ãŠããã2024幎7æçŸåšãHugging Face Hub äžã§å ¬éãããŠããã¢ãã«ã®äžã§å»åž«åœå®¶è©Šéšã«ãããæé«æ§èœãèªã£ãŠããŸãã
çŠå¿è¢ã®è§£æ
çŠå¿è¢ãšã¯ãå»åž«åœå®¶è©Šéšã«ãããŠäžå®æ°ä»¥äžéžæãããšããããŸã§ã®åŸç¹ã«é¢ãããäžåæ Œãšãªãéžæè¢ãæããŸããå ·äœçã«ã¯ãæ£è ã®æ»äº¡ãäžå¯éçãªèåšã®æ©èœå»çµ¶ã«çŽçµããäºé ãããæ¥µããŠéå«ççãªäºé ããªã©ãå«ãéžæè¢ãçŠå¿è¢ã«è©²åœãããšãããŠããŸãããã ãã©ã®éžæè¢ãçŠå¿è¢ã«è©²åœãããã¯äžè¬ã«ã¯å ¬éãããŠããããåäºåæ ¡ã解説æžã§ã¯å®éã®å»ççŸå Žã§çŠå¿ãšããããã®ãåºã«çŠå¿è¢ãšããŠè¡šèšããŠããåŸåã«ãããŸããããã§ã¯ IgakuQA ã®ããŒã¿ãšå»åž«åœå®¶è©Šéšã®åèæžæ å ±ãããšã«çŠå¿è¢ã®è§£æãè¡ããŸããã
以äžã¯ãä»åéçºãã Llama3-Preferred-MedSwallow-70B ãçŠå¿è¢ãéžãã§ããŸã£ãåé¡ã®äžäŸã§ãã
åé¡ (115A75): 68æ³ã®å¥³æ§ãæèäœäžã®ããææ¥è»ã§æ¬å
¥ãããã5æ¥åããæåçç¶ãåºçŸãã飿¬²äžæ¯ãšåŠæ æã®ãã宿å
æè¬ãæçšã§ããŠããªãã£ãã3æ¥åãã38âå°ã®çºç±ããããèªå®
ã§ããããŸã£ãŠãããšããã家人ãçºèŠãææ¥è»ãèŠè«ãããæ¢åŸæŽãšããŠã2幎åã«éæ©èœæ§äžåäœè
«çãçµè¶åœ¢éªšæŽæè¡ã«ãŠæåºãããããæ®åè
«çãææãããŠããã以åŸãããã³ã«ããŸã³ãšã¬ããããã·ã³ãç¶ç¶æçšäžã§ãã£ããæ¬å
¥æã®æèã¬ãã«ã¯JCS II-20ãäœæž©38.4âãè¡å§80/46mmHgãå¿ææ°122/åãæŽãåŒåžæ°24/åãSpO2 94ïŒ
ïŒroom airïŒãè¡æ¶²çååŠæèŠïŒè¡ç³65mg/dLãNa 121mEq/LãK 5.5mEq/LããŸãè¡ãã¹ã察å¿ã¯ã©ããã3ã€éžã¹ã
a: NSAIDã®æäž
b: ççé£å¡©æ¶²ã®èŒžæ¶²
c: ã°ã«ã³ãŒã¹ã®ç¹æ»Žé泚
d: ãããã³ã«ããŸã³ã®é泚
e: ã¬ããããã·ã³ã®è管ãçšããæäž
ïŒç¬¬115åå»åž«åœå®¶è©Šéšåé¡ [12] ããåŒçšïŒ
æ£è§£ã¯ b,c,d ã§ãããä»åéçºããã¢ãã«ã¯ b,d,e ãéžæããŸããã察象ãšãªã£ãŠããæ£è ã¯å¯è ã¯ãªãŒãŒãšèãããããã®ãããªå Žåãå¯è ç®è³ªã¹ããã€ããè£ããã« e ã®ç²ç¶è ºãã«ã¢ã³ãæäžããã®ã¯çŠå¿ãšãããŠãããe ã¯çŠå¿è¢ãšèããããŸãã
ä»åéçºãã Llama3-Preferred-MedSwallow-70B ã¯5幎éã®åé¡ã§èš4ã€ã®çŠå¿è¢ãéžæããŸãã**ããããŠãããã®åé¡ãäžèšã®åé¡ã®ããã«äžããããæ£è ã®ç¶æ³ããé©åãªèšºæãè¡ãå¿ èŠãããåé¡ã§ããããã®ãããªåé¡ã§ã¯äžããããæ å ±ããé©åãªãã®ãçµã¿åãããŠè«ççã«å€æãããå¿ èŠããããŸãããããããè«ççæšè«ãèŠããåé¡ã¯äžè¬çã« LLM ã«ãšã£ãŠãŸã 課é¡ãå€ãåéã§ããLLM ã®è«ççæšè«èœåããããææ³ãšã㊠Chain-of-Thought [13] ãå»çåéåãã§ã¯ MedPrompt [14] ãªã©ããããŸãããå®èšåºã§æŽ»çšã§ããã¬ãã«ã«éããã«ã¯å»çåéã«ãããè«ççæšè«èœåã®ãããªãåäžãå¿ èŠã ãšèããŠããŸãã
åæã«ã¢ãã«ã®ãã«ãã¢ãŒãã«åãéèŠãªèª²é¡ã§ããä»åãLlama3-Preferred-MedSwallow-70B ãçŠå¿è¢ãéžãã 4åã®ãã¡2åã¯ãæ¬æ¥ã®è©Šéšã§ã¯CTãªã©ã®å»çšç»åãä»ããŠããåé¡ã§ãããããããèšèªã¢ãã«ã®ãã³ãããŒã¯ãšããæ§è³ªäžãç»åã¯äžããæåæ å ±ã®ã¿ããã¢ãã«ã¯åçããŠããŸããããããã£ãŠæ å ±ãäžååãªç¶æ ã§åçãããŠããããšã«ãªããŸããå®èšåºã®çŸå Žã§ã¯æåæ å ±ã ãã§ãªããç»åãæç³»åçã®æ å ±ã䜿çšãããŸãããããããã«ãã¢ãŒãã«ãªæ å ±ãããšã«çŸå Žã®å€æããµããŒãã§ããã¢ãã«ã®éçºã¯ä»åŸã®èª²é¡ã§ãã
å»çãã¡ã€ã³ä»¥å€ã§ã®æ§èœã«ã€ããŠ
| Llama-3-Swallow-70B | Llama3-Preferred-MedSwallow-70B | |
| MMLU â | 0.7467 ± 0.0034 | 0.7477 ± 0.0034 |
| ARC-C â | 0.5939 ± 0.0144 | 0.6058 ± 0.0143 |
| HellaSwag â | 0.6638 ± 0.0047 | 0.6693 ± 0.0047 |
| Winogrande â | 0.7979 ± 0.0113 | 0.8114 ± 0.0110 |
| TruthfulQA â | 0.4296 ± 0.0173 | 0.4504 ± 0.0174 |
| GSM8K â | 0.8158 ± 0.0107 | 0.8294 ± 0.0104 |
衚2. LLM ã®äžè¬çãªèœåãæž¬ããã³ãããŒã¯ã§ã®çµæãã¢ãã«ã®ç·åçãªç¥èãåã MMLU [15]ãã¢ãã«ã®åžžèã»æšè«èœåãæž¬ã ARC-C [16]ã»HellaSwag [17]ã»Winogrande [18]ãã¢ãã«ã®èª å®ããæž¬ã TruthfulQA [19]ãç®æ°ã®æç« é¡ã§æ§æãããŠãã GSM8K [20] ã®èš6ã€ã®ææšã§ã®åŸç¹ããã³æšæºèª€å·®ã瀺ããŠããŸãããããã®ææšãåŸç¹ãé«ãã»ã©ç²ŸåºŠãè¯ãããšãæå³ããŸãã
LLM ã§ finetuning ãè¡ããªã¹ã¯ãšããŠç Žæ» çå¿åŽ (catastrophic forgetting) ãããè°è«ãããŸããç¹åããããã¡ã€ã³ä»¥å€ã§ã® LLM ã®ç Žæ» çå¿åŽãæãã€ã€ãç¹åå ã®ãã¡ã€ã³ç¥èã远å ã§åŠç¿ãããææ³ã®éçºã¯ LLM ã®å¿çšã«ãããŠéèŠãªèª²é¡ãšãªããŸããä»å㯠finetuning ååŸã§å»çãã¡ã€ã³ä»¥å€ã®ç¥èãèœåãã©ã®ããã«å€åããŠãããã調ã¹ãããã« open-llm-leaderboard v1 [21] ã§äœ¿çšãããŠããã¿ã¹ã¯ãçšã㊠finetuning ååŸã®ã¢ãã«ã®äžè¬çãªèœåã®æ¯èŒãè¡ããŸãã (衚2)ãè©äŸ¡ã«ã¯ EleutherAI ã® lm-evaluation-harness [22] ãçšããŸããã
çµæãèŠããšãåŠç¿åã® Llama-3-Swallow-70B ãš ç¶ç¶äºååŠç¿åŸã® Llama3-Preferred-MedSwallow-70B ã®éã§äžè¬çãªèœåãç¥èã«å·®ããªãããšãããããŸããå¿åŽãèµ·ããŠããªãèŠå ãšããŠã (i) å°ããããŒã¿ã»ããã§ iteration æ°å°ãªãåŠç¿ã§ããŠããããã«å¿åŽãæå°éã§æžãã§ããå¯èœæ§ [23] ãã (ii) full parameter finetuning ã§ã¯ãªããQLoRA ãçšããããšãå¿åŽãæããæ£ååãšããŠæ©èœããŠããå¯èœæ§ [24] ãªã©ãèããããŸããä»åŸãPFNã§ã¯ LLM ã® finetuning ã®ç ç©¶éçºãé²ããç Žæ» çå¿åŽã«ã察å¿ã§ãããã¡ã€ã³ç¹å LLM ã®ããã«å¹æçãªææ³éçºãç®æããŸãã
æåŸã«
ä»åãæ¢åã®ã¢ãã«ã«å¯ŸããŠç¶ç¶äºååŠç¿ãè¡ããå»çåã LLM ãšã㊠Llama3-Preferred-MedSwallow-70B ãéçºããŸãããLlama3-Preferred-MedSwallow-70B ã¯å»åž«åœå®¶è©Šéšã«ãããŠãHugging Face Hub äžã§å ¬éãããŠããã¢ãã«ã®äžã§æé«æ§èœãéæããŠãããQLoRA ãçšããç¶ç¶äºååŠç¿ãéåžžã«ããæ©èœããããšã確èªã§ããŸãããä»ååŸãããç¥èŠã¯ãä»åŸã®ç€Ÿå ã«ãããç ç©¶éçºããœãªã¥ãŒã·ã§ã³éçºã«ã掻çšããŠããäºå®ã§ãã
ãªãããã®èšäºã§ã¯æ¥æ¬å»åž«åœå®¶è©Šéšããã³ãããŒã¯ãšããŠå»çåéã«ãããã¢ãã«ã®æ§èœè©äŸ¡ãè¡ããŸããããå®éã«å»çã®çŸå Žã§æŽ»çšã§ããã¢ãã«ãäœãããã«ã¯ã䜿çšç®çã«ããå³ãããã³ãããŒã¯ãèšå®ããããã«åãããããŒã¿ã»ããã®æŽåãšã¢ãã«ã®ã«ã¹ã¿ãã€ãºãè¡ãå¿ èŠããããšèããŠããŸããPFN ã§ã¯æ§ã ãªãã¡ã€ã³ã«ããã LLM ã®æŽ»çšãç®æããç¬èªã®ãã³ãããŒã¯ãããŒã¿ã»ããã®æŽåãé²ããŠããŸããä»åã®åãçµã¿ã«èå³ãæã£ãŠããã ããäŒæ¥ã®æ¹ã¯ãã²ãåãåãããã©ãŒã ãããé£çµ¡ãã ããã
æåŸã«ãPreferred Networks ã§ã¯æ¡çšãè¡ã£ãŠãããŸããä»åã®èšäºã§èå³ãæã£ãŠããã ããæ¹ã¯ãã²ãå¿åãã ããã
- Machine Learningã»Optimizationã»Data Science Engineer/æ©æ¢°åŠç¿ã»æé©åã»ããŒã¿ãµã€ãšã³ã¹ãšã³ãžãã¢
- åºç€ã¢ãã«ãµãŒãã¹éçºãšã³ãžãã¢ïŒå€§èŠæš¡èšèªã¢ãã« / ç»ååºç€ã¢ãã«ïŒ
- Software Engineer – LLM Platform (Application, Infrastructure) / å€§èŠæš¡èšèªã¢ãã«ãã©ãããã©ãŒã éçºãšã³ãžã㢠(ã¢ããªã±ãŒã·ã§ã³ã»ã€ã³ãã©)
References
[1] Kasai, Jungo, et al. “Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations.” arXiv preprint arXiv:2303.18027 (2023). https://github.com/jungokasai/IgakuQA .
[2] “1,000åãã©ã¡ãŒã¿èŠæš¡ã®ç¬èªLLMãPLaMo-100Bãã®äºååŠç¿”. https://tech.preferred.jp/ja/blog/plamo-100b/
[3] Ouyang, Long, et al. “Training language models to follow instructions with human feedback.” Advances in neural information processing systems 35 (2022): 27730-27744.
[4] Chen, Zeming, et al. “Meditron-70b: Scaling Medical Pretraining for Large Language Models.” arXiv preprint arXiv:2311.16079 (2023).
[5] Gekhman, Zorik, et al. “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?.” arXiv preprint arXiv:2405.05904 (2024).
[6] “Llama 3 Swallow”. https://swallow-llm.github.io/llama3-swallow.ja.html.
[7] Dettmers, Tim, et al. “QLoRA: Efficient Finetuning of Quantized LLMs.” Advances in Neural Information Processing Systems 36 (2024).
[8] Dettmers, Tim, and Luke Zettlemoyer. “The case for 4-bit precision: k-bit inference scaling laws.” International Conference on Machine Learning. PMLR, 2023.
[9] Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” arXiv preprint arXiv:2106.09685 (2021).
[10] Aghajanyan, Armen, Luke Zettlemoyer, and Sonal Gupta. “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.” arXiv preprint arXiv:2012.13255 (2020).
[11] “Efficient finetuning of Llama 3 with FSDP QDoRA” https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html.
[12] https://www.mhlw.go.jp/seisakunitsuite/bunya/kenkou_iryou/iryou/topics/tp210416-01.html
[13] Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Advances in neural information processing systems 35 (2022): 24824-24837.
[14] Nori, Harsha, et al. “Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine.” arXiv preprint arXiv:2311.16452 (2023).
[15] Hendrycks, Dan, et al. “Measuring Massive Multitask Language Understanding.” arXiv preprint arXiv:2009.03300 (2020).
[16] Clark, Peter, et al. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.” arXiv preprint arXiv:1803.05457 (2018).
[17] Zellers, Rowan, et al. “HellaSwag: Can a Machine Really Finish Your Sentence?.” arXiv preprint arXiv:1905.07830 (2019).
[18] Sakaguchi, Keisuke, et al. “Winogrande: An adversarial winograd schema challenge at scale.” Communications of the ACM 64.9 (2021): 99-106.
[19] Lin, Stephanie, Jacob Hilton, and Owain Evans. “TruthfulQA: Measuring How Models Mimic Human Falsehoods.” arXiv preprint arXiv:2109.07958 (2021).
[20] Cobbe, Karl, et al. “Training Verifiers to Solve Math Word Problems.” arXiv preprint arXiv:2110.14168 (2021).
[21] https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard
[22] https://github.com/EleutherAI/lm-evaluation-harness
[23] Chang, Hoyeon, et al. “How Do Large Language Models Acquire Factual Knowledge During Pretraining?.” arXiv preprint arXiv:2406.11813 (2024).
[24] Howard, Jeremy, and Sebastian Ruder. “Universal Language Model Fine-tuning for Text Classification.” arXiv preprint arXiv:1801.06146 (2018).
* 2018â2022 幎ã®åæ Œç¹ã¯é ã«ã368ç¹ã369ç¹ã375ç¹ã369ç¹ã371ç¹ãšãªã£ãŠããŸãã [1]ããã
** ä»åéçºããã¢ãã«ãçŠå¿è¢ãéžãã§ããŸã£ãåé¡ã¯æ¬¡ã®4åã§ãã: 114B32ã115A75ã116B47ã116D48ã


