Alpaca-LoRAãšããå®¶åºçšGPUã§ãå€§èŠæš¡èšèªã¢ãã«ã®FineTuningãå¯èœãªã¢ãã«ãçºè¡šãããŸããã
æ¬èšäºã§ã¯ãlivedoorãã¥ãŒã¹ã³ãŒãã¹ã䜿çšããŠAlpaca-LoRAãFineTuningããŠãã¥ãŒã¹ã®ã¿ã€ãã«ãèãããããšããã¿ã¹ã¯ã«ææŠããŠã¿ãŸãã
æè¡ã®æŠèŠ
Alpacaãšã¯
Alpacaãšã¯ãå
æ¥Metaãçºè¡šããLLaMa 7Bãtext-davinci-003ã«ããself-instructã§çæãããããŒã¿ã䜿çšããŠFineTuningããèšèªã¢ãã«ã
çæããããŒã¿ã¯52Kåã§çæã³ã¹ãã¯500ãã«ä»¥äžãšäœã³ã¹ãã§ãã
人éã«ããäºåè©äŸ¡ã§ã¯7Bãšããæ¯èŒçå°ããªã¢ãã«ã«ãé¢ããããtext-davinci-003ã«äŒŒãæåã瀺ããšããå ±åããããããŠããŸãã
Alpaca-LoRAãšã¯
Alpaca-LoRAãšã¯Alpacaã§äœæããããŒã¿ã»ããã䜿çšããŠLLaMa 7Bã«low-rank adaptation(LoRA)ã䜿çšããŠã¢ãã«ãFineTuningããããšã§äœã³ã¹ãã§Alpacaãšåçã®åºåãåŸãããšãã§ããèšèªã¢ãã«ã§ãã
åŠç¿ã¯RTX4090äžå°ã§5æé以å
ã«å®äºãããšå ±åãããŠããŸãããŸãã13bã30bã65bãªã©ã®ã¢ãã«ã«ã容æã«æ¡åŒµãå¯èœãªããã§ãã
ãŸããæšè«ã¯Raspberry Pi(4GB)ã§ãåäœããããšã確èªãããŠããŸãã
以äžã¯ãAlpaca-LoRAã®ãªããžããªã«äžãã£ãŠãããtext-davinci-003ãAlpacaãAlpaca-LoRAã®åºåçµæãæ¯èŒããäŸãDeepLã«ããæ¥æ¬èªèš³ãããã®ã§ãã
ã¢ã«ãã«ã«ã€ããŠæããŠãã ããã
ã¢ã«ãã«ã¯ã©ã¯ãç§ã®åç©ã§ãåç±³ã®ã¢ã³ãã¹å±±èã«çæ¯ããŠããŸããåç±³ã¢ã³ãã¹å±±èã«çæ¯ããæãããé«çŽæã®ããããªãŒã¹ã§ç¥ãããè¡£é¡ãæ¯åžãªã©ã®ææãšããŠäœ¿ãããŠããŸããã¢ã«ãã«ã¯èé£åç©ã§ãèãªã©ãé£ã¹ãŠããŸããç€ŸäŒæ§ã®ããåç©ã§ãæå€§20é ã»ã©ã®çŸ€ãã§ç掻ããŠããŸãã
ã¢ã«ãã«ã¯ã©ã¯ãããªã£ãã«äŒŒãå°åã§ãµããµãããåç©ã§ãããã«ãŒãšããªãã¢ãåç£ã§ãçŽ5,000幎åã«åããŠå®¶çåãããŸãããããããªã©ã®è¡£æåã«äœ¿ãããäžè³ªã§æãããããªãŒã¹ãäž»ãªç®çã§é£ŒãããŠããŸããã¢ã«ãã«ã¯çŸ€ãã§ç掻ããåç©ã§ã幎é
ã®ç·æ§ãäžå¿ãšããå°ããªå®¶æéå£ã§æ®ãããŠããŸããç€ŸäŒæ§ã®é«ãåç©ã§ã飌ãäž»ãšåŒ·ãçµã§çµã°ããããšããããŸãã
ã¢ã«ãã«ã¯ãåç±³ã®ã©ã¯ãç§ã®åç©ã§ãã©ãã«äŒŒãå®¶çã®äžçš®ã§ããåç±³ã®ã¢ã³ãã¹å±±èã«çæ¯ããæãããé«çŽãªæ¯ãæã€ããšããå®¶çãšããŠé£ŒãããŠããŸããã¢ã«ãã«ã¯ç€ŸäŒæ§ã®ããåç©ã§ãæå€§20é ã®çŸ€ãã§ç掻ããŠããŸããäžè¬çã«ã¯2ïœ5é ã®å°ããªçŸ€ãã§é£Œè²ãããæ¯èŒçäžè©±ããããããšèšãããŠããŸããã¢ã«ãã«ã¯èé£åç©ã§ãèãå¹²ãèãªã©ã®æ€ç©ãé£ã¹ãŸãããŸããç©ããã§äººæã£ããæ§æ Œã§ç¥ãããŠãããããããšããŠã人æ°ããããŸãã
å®è£
æ¬æ¥ã¯Alpacaã§äœ¿çšããããŒã¿ãæ¥æ¬èªã«ç¿»èš³ããŠFineTuningãè¡ãæ¥æ¬èªã¢ãã«ãäœæãããã£ãã®ã§ããã翻蚳ã³ã¹ãã®åé¡ããããŒã¿ãçšæã§ããŸããã§ããã
ä»åã¯ã詊ããšããããšã§åé ã«ãè¿°ã¹ããšããlivedoorãã¥ãŒã¹ã³ãŒãã¹ã䜿çšããŠãã¥ãŒã¹ã®å 容ããã¿ã€ãã«ãèãããšããã¿ã¹ã¯ã«ç¹åããåŠç¿ãè¡ã£ãŠãããŸãã
çè ã®äœ¿çšGPUã¯RTX3080äžå°ã§ããããããµã€ãºã調æŽããã°GPUã¡ã¢ãªã12GB以äžã§ããã°åäžã®GPUã§FineTuningãå¯èœã ãšæããŸãã
livedoorãã¥ãŒã¹ã³ãŒãã¹ã®ååŠç
Alpacaã®ããŒã¿ã¯æ¬¡ã®ããã«instructãinputãoutputã®3ã€ã®èŠçŽ ããã§ããŠããŸãã
{
"instruction": "Give three tips for staying healthy.",
"input": "",
"output": "1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule."
}
ååŠçãšããŠlivedoorãã¥ãŒã¹ã³ãŒãã¹ã®ããŒã¿ãäžèšã®åœ¢ã«çœ®ãæããŸãã
ä»åã¯æ¬¡ã®ããã«instructionã¯åžžã«ã次ã®ãã¥ãŒã¹ã®ã¿ã€ãã«ãèããŠãã§åºå®ããinputããã¥ãŒã¹ã®å 容ãoutputãã¿ã€ãã«ãšããŠãããŸãã
{
"instruction": "次ã®ãã¥ãŒã¹ã®ã¿ã€ãã«ãèããŠ",
"input": "åå·ã²ãªã®ãããããªãšãããé·è°·å·äº¬åãããæ°å£çµè¡£ããããæå±ããŠããèžèœãããã¯ã·ã§ã³ã®æ ªåŒäŒç€Ÿã¬ãããšã³ã¿ãã€ã³ã¡ã³ããããæè¬ã蟌ããŠçæ§ã«ããããŒããè¿ãããããã£ãªãã£ãŒãããžã§ã¯ããã¬ããã»ãããã£ãªã»ãããžã§ã¯ãããéå§ããããŸãããæšå¹ŽããäŒç»æºåããŠãããšããåãããžã§ã¯ãã¯ãä»åã®å€§çœå®³ãçºçããããšã«ãããäŒç»å
容ã倿Žãèšç»ãæ©æ¥ã«é²ãããšã®ããšã§ãæå±ã¿ã¬ã³ãã»ã¹ã¿ããäžåã«ããç·é¡åèš3543äž9896åã®çŸ©æŽéã«ããæ¯æŽããæå±ã¿ã¬ã³ããšã¹ã¿ãããå°æ¹åå°ãå»»ããäžé·æçãªãã£ãªãã£ãŒæŽ»åãªã©ã宿œããäºå®ã ããã§ããâ ã¬ããã»ãããã£ãªã»ãããžã§ã¯ãæŠèŠ1.被çœè
ã®çæ§ãžçŸ©æŽéãå¯ä»ã¬ãããšã³ã¿ãã€ã³ã¡ã³ããã2000äžåãæå±ã¿ã¬ã³ãã»ã¹ã¿ããäžåãã1543äž9896åãåèš3543äž9896åãæ¥æ¬èµ€åå瀟ãéããŠå¯ä»èŽããŸãã2.æå±ã¿ã¬ã³ããã¹ã¿ããã«ãããã£ãªãã£ãŒæŽ»åã®å®æœæå±ã¿ã¬ã³ããšã¹ã¿ãããå°æ¹åå°ãå»»ããäžé·æçãªãã£ãªãã£ãŒæŽ»åã宿œããŸããåæã«è¢«çœå°ã®1æ¥ã§ãæ©ã埩èãé¡ã£ãŠãã¬ããåéããéèšããåºã矩æŽéãåããŸãã3.ãŠãã¯ããšã®ãã£ãªãã£ãŒã³ã©ãååçºå£²æ¬ãããžã§ã¯ãã«è³åããŠäžããäŒæ¥ãšååããæ¬ãããžã§ã¯ãã®çå¿µãæ³ããåºãäŒããŠãããŸããåéã»åçéã¯ãæ¥æ¬èµ€åå瀟ãç€Ÿå£æ³äººã»ãŒãã»ã¶ã»ãã«ãã¬ã³ã»ãžã£ãã³ãªã©ã«å¯ä»ãããŠé ããŸããæŽ»åã®è©³çްãç¢ºå®æ¬¡ç¬¬ãéæå
¬åŒãµã€ãçã«ãŠãäŒãããŠãŸãããŸããâ é¢é£ãªã³ã¯ã»æ ªåŒäŒç€Ÿã¬ãããšã³ã¿ãã€ã³ã¡ã³ã-å
¬åŒãµã€ã",
"output": "åå·ã²ãªã®ïŒããªãšãã«ããã¬ããã»ãããã£ãªã»ãããžã§ã¯ã"
}
ååŠççšã®ã³ãŒãã¯ä»¥äžã§ãã
import os
import json
def parse_file(file_path):
with open(file_path, 'r') as f:
title = ''
news = ''
for i, line in enumerate(f.readlines()):
if i < 2:
# urlãšæ¥ä»ã¯ç¡èŠ
continue
elif i == 2:
title = line.strip()
else:
news += line.strip()
return title, news
def main():
DATADIR = 'data/ldcc-20140209/text'
LICENCE_FILENAME = 'LICENSE.txt'
learn_datas = []
news_list = []
for item in os.listdir(DATADIR):
if not os.path.isdir(os.path.join(DATADIR, item)):
# ãã£ã¬ã¯ããªä»¥å€ã¯ç¡èŠ
continue
for filename in os.listdir(f'{DATADIR}/{item}'):
learn_data = {
"instruction": "次ã®ãã¥ãŒã¹ã®ã¿ã€ãã«ãèããŠ",
"input": "",
"output": ""
}
if filename == LICENCE_FILENAME:
continue
title, news = parse_file(f'{DATADIR}/{item}/{filename}')
news_list.append(news)
learn_data["input"] = news
learn_data["output"] = title
learn_datas.append(learn_data)
json_learn_data = json.dumps(learn_datas, indent=4, ensure_ascii=False)
with open('news_data_ja.json', 'w', encoding="utf-8") as f:
f.write(json_learn_data)
Alpaca-LoRAã«ããFineTuning
Alpaca-LoRAã®ãªããžããªãããšã£ãŠããfinetune.pyã®ãã€ããŒãã©ã¡ãŒã¿ãšå ¥åããŒã¿ã ãã倿Žãããã以å€ã¯å ã®ã³ãŒããã倿ŽããŠããŸããã
äžèšã¯RTX3080ã«åãããŠèª¿æŽãããã©ã¡ãŒã¿ã«ãªããŸãã
CUTOFF_LENã ãlivedoorãã¥ãŒã¹ã³ãŒãã¹ã§ã¯2200ã»ã©ã«ããã®ãé©åã§ãããGPUã«ã®ããªãããçããªãµã€ãºãèšå®ããŠããŸãã
MICRO_BATCH_SIZE = 1
BATCH_SIZE = 64
GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
EPOCHS = 5
LEARNING_RATE = 3e-4
CUTOFF_LEN = 512
LORA_R = 8
LORA_ALPHA = 16
LORA_DROPOUT = 0.05
VAL_SET_SIZE = 24
åºåçµæ
Alpaca-LoRAã§FineTuningããã¢ãã«ãšChatGPTã®åºåçµæãæ¯èŒããŠãããŸãã
å
¥åã®ãã¥ãŒã¹ã®å
容ã¯ä»¥äžããå
¥åé·ã®éœåã§äžéšãæç²ããŠããŸãã
https://news.yahoo.co.jp/articles/ad7a110b769c6d9154bb5c146d20095a805b8814
次ã®ãã¥ãŒã¹ã®ã¿ã€ãã«ãèããŠ
ã»ã®ãŒã®ãšå¹³åãªå
å®¹ãæ¯é±æ¥œãããã¢ãã¡ããµã¶ãšãããã§ãããæå€ãšæ³¥æ£ã«å
¥ããããšããç©éšãªãšããœãŒããå€ãã®ã¯ãåãã§ããããïŒã1992幎çºå£²ã®æ¬ã磯éå®¶ã®è¬ãã«ãããšãåäœãã³ã¬68å·»ã®ãã¡ã§ã磯éå®¶ãæ³¥æ£ã«å
¥ãããåæ°ã¯18åã«ãã®ãŒããšã®ããšã§ãã磯éå®¶ã®éåããèŠã ãããªåäœã«æ²¿ã£ãŠãã¢ãã¡ã§ãæ³¥æ£ã¯å®æçã«ãµã¶ãšãããã¡ã®å®¶ã«ãã£ãŠããŸãããã®åæ°ã®ç°æ§ãªå€ãã«ããããã§ã¯ãå®çªãšããœãŒãã«ããŠãå€ãããããšãå¿é
ãã声ãŸã§åºãŠããŸããŸãããããã«ãããããªæåäžä»£ã®è©±ãšããã©ããããã«é²ç¯æèã®äœããææããããªããŸãã ããããããµã¶ãšããå®
ã¯æ³¥æ£ãå¯ãä»ããªãé ãã«ã¯ãªã£ãŠããŸããã
æ³¥æ£ã«å
¥ããããµã¶ãšããã®ãšããœãŒããå€ããã
ããµã¶ãšãããã«ã¯æå€ãªç©éšãªãšããœãŒãã!? æ³¥æ£ãé »ç¹ã«èšªãã磯éå®¶ã«é²ç¯æèã®äœããææãã声ã
chatGPTã«ã¯å°ãå£ãããã«ãæããŸãããAlpaca-LoRAã®åºåãæªããªãæ¥æ¬èªãçè§£ããŠããããã«æããŸãã
æåŸã«
ä»å䜿çšããã³ãŒãã¯ä»¥äžã®ãªããžããªã«ãããŠããŸãã
https://github.com/tosiyuki/alpaca-lora-create-news-title
ä»åŸããã®ããã«å€§èŠæš¡èšèªã¢ãã«ãããŒã«ã«ç°å¢ã§ãããç°¡åã«åãããããã«ãªãæ°ãããã®ã§æ¥œãã¿ã§ãã