FlexGen ใง opt-66b ใๅใใใกใข
ๅไฝ็ฐๅข(256 GB ๆจๅฅจ(X670 ใชใฉใง 48GB x 4 ใง 192 GB ใใใใใใใฏใใใพใใ). 128 GB ใงใๅใใใฉไบๅใใใ้ ใใชใ)ใใใใฐ็นใซๅ้กใชใๅใใฎใงใใ, ไธ้จ huggingface transformers(ใใใใใๅๅใงใใใ, transformer ็ณป model ใใบใใฃใจใใฆใณใญใผใใใใ่ชญใใใใฎใฉใคใใฉใช) ใงใฎ(?)weight ใฎใใฆใณใญใผใใซไธ้ฝๅใใใใใงใ.
ใจใใพ่ฉฆใ
ใจใใพ chatbot ่ฉฆใใพใ!
- x299 256 GB CPU mem
- 3090 x 2(ๅฎ้ใฏไธๅใ ใไฝฟใใใ)
$ python chatbot.py --model facebook/opt-66b --percent 50 10 100 0 100 0 --compress-weight
(ใกใชใฟใซๆๆฐ FlexGen ใ ใจ chatbot.py ใตใณใใซใๆถใใใฆใพใ)
GPU ใฏ 20 GB, CPU ใฏ 25 GB ใใใใฎใกใขใชๆถ่ฒป้ใงใใ.
A chat between a curious human and a knowledgeable artificial intelligence assistant.
Human: Hello! What can you do?
Assistant: As an AI assistant, I can answer questions and chat with you.
Human: What is the name of the tallest mountain in the world?
Assistant: Everest.
ใใใงๆฑไบฌใจ NY ใฉใใใใ้ ใใ่ใใฆใฟใพใ!
Human: How far from Tokyo to New York?
Assistant: About 10,000 kilometers.
ๆญฃ่งฃใฏ 10,850 km ใใใใงใใ, About ~ ใง็ญใใฆใใใฎใงใจใท!
ใใกใใจ SI ๅไฝ็ณปใง็ญใใฆใใใจใฉใค!
Human: ใใใ ใใใฏใใใใใงใใ๏ผ
Assistant: ใใ! ใใใ ใใใฏใใใใใงใใ๏ผ
ๆฅๆฌ่ชใฏใ ใใ ใใงใใ...
่ฟ็ญใซใฏ 1 ๅใใใใใใใพใใ. FlexGen ใซใฏ opt-66b ใฎ benchmark ่จญๅฎใฏใชใใฎใงใใ, opt-175b ใฎ benchmark ็ตๆใใๆณๅฎใใใซใ ใใใ 1~2 token/sec ใใใใชๆใใงใใใใ.
opt-66b ใฏ chatbot ใจใใฆไฝฟใใฎใฏ็พ็ถ้ฃใใใซใ.
ใกใขใชใใๅ ดๅใฎๆนๅ
--percent 0 100 100 0 100 0 --compress-weight
ใง, CPU ใกใขใช 64 GB, GPU 4 GB ใใใใง่ฟ็ญๆ้ใฏ 20 ็งใใใใซใชใใพใใ! GPU ใใใพใไฝฟใใใปใใๆง่ฝใงใ...
(--compress-weight ใชใใฏใกใขใชไธ่ถณใงใใ(ๆญฃ็ขบใซใฏ้ ๅผตใฃใฆๅใใใใจ kernel ใใกใขใชๅง็ธฎใจใใใฆใใคใพใงใๅๆๅใ็ตใใใชใ. 384 GB ใจใใชใๅใใใใงใ)
ใงใ
Human: How far from Tokyo to New York? Assistant: About 12,000 miles.
ใจใใคใซๆ็ฎใง็ญใใฆใกใ!(ใใใ่ท้ข้้ใฃใฆใใ), ใจใชใใพใใ.
ใใ ่ฟ็ญ 20 ็งใ chatbot ็จ้ใงใฏใกใใฃใจใใณใใใซใ.
opt-30b (benchmark 7 token/sec) ใ ใจ 5~10 ็งใใใใง, ใชใใจใ่จฑๅฎนใงใใๆใ.
opt-6.7b ใฏ 2~3 ็งใง่ฟใฃใฆใใฆใใฃใใๆใใใพใใ.
opt-6b or opt-30b ใงใพใใฏใใฃใใใใฆ, ใใใงใใฆใผใถใผใๆบ่ถณใใชใใฃใใ, ่ฒฌไปป่ ใจใใฆ opt-66b ๅบใใฆๅใๅใใใใฆใใใๆใใง้็จใใใใงใใใใ...
ใใจใฏ, LLaMa ใใใใ GPT-neox ็ณปใซๆๅพ ใงใใใญ.
ใกใชใฟใซ opt-30b ใใใคใซใง่ฟ็ญใ, ใพใ่ท้ข้้ใฃใฆใใ..
opt-66b benchmark
FlexGen repo ใซใฏ opt-66b ใฎ benchmark ใฏ็กใใฎใงใใ, opt-30b ใจ opt-175b ใฎ่จญๅฎใๅ่ใซใใฃใฆใฟใพใใ. 1x1(1 node, single GPU)ใงใฎ็ตๆใงใ. compress ๆๅนใซใใฆใใพใ.
python -m flexgen.flex_opt --model facebook/opt-66b --path _DUMMY_ --percent 0 100 0 100 0 100 --gpu-batch-size 32 --num-gpu-batches 3
--debug fewer_batch --compress-cache
model size: 122.375 GB, cache size: 114.750 GB, hidden size (prefill): 0.896 GB
init weight...
warmup - generate
benchmark - generate
100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 20/20 [0
2:19<00:00, 6.96s/it]
/home/syoyo/miniconda3/envs/flexgen/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:262: UserWarning: torch.distribu
ted.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
warnings.warn(
TorchDevice: cuda:0
cur_mem: 1.2672 GB, peak_mem: 6.1617 GB
TorchDevice: cpu
cur_mem: 123.2760 GB, peak_mem: 0.0000 GB
model size: 122.375 GB cache size: 114.750 GB hidden size (p): 0.896 GB
peak gpu mem: 6.162 GB projected: True
prefill latency: 136.627 s prefill throughput: 359.753 token/s
decode latency: 511.038 s decode throughput: 5.823 token/s
total latency: 647.665 s total throughput: 4.743 token/s
python -m flexgen.flex_opt --model facebook/opt-66b --path _DUMMY_ --percent 0 100 0 100 0 100 --gpu-batch-size 12 --num-gpu-batches 4 --debug fewer_batch --compress-cache --prompt-len 1024
model size: 122.375 GB, cache size: 111.375 GB, hidden size (prefill): 0.870 GB
init weight...
warmup - generate
benchmark - generate
100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 20/20 [02:21<00:00, 7.09s/it]
/home/syoyo/miniconda3/envs/flexgen/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:262: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
warnings.warn(
TorchDevice: cuda:0
cur_mem: 1.2663 GB, peak_mem: 6.7893 GB
TorchDevice: cpu
cur_mem: 123.2759 GB, peak_mem: 0.0000 GB
model size: 122.375 GB cache size: 111.375 GB hidden size (p): 0.870 GB
peak gpu mem: 6.789 GB projected: True
prefill latency: 139.272 s prefill throughput: 352.921 token/s
decode latency: 501.070 s decode throughput: 2.970 token/s
total latency: 640.342 s total throughput: 2.399 token/s
CPU 123 GB ใใใไฝฟใ, 4.7 tokens/s ใงใใ. opt-30b ใ 7~8 tokens/sec, opt-175b ใ 1 tokens/sec ใงใใใๆๅคใจใใใใใใงใใใใ.
ใใ ใใใใตใคใบ็ญๅขใใใฆในใซใผใใใไธใใฆใฎ็ตๆใ ใจๆใใพใใฎใง, ใใใใ่จญๅฎใใใฃใฆใฌใคใใณใท้่ฆใซใใฆใ chatbot ็จ้ๅใใซใฌใคใใณใท็ญ็ธฎใฏใใใฃใฝใใใชๆฐใใใพใ.
ใขใใซใใฆใณใญใผใใฎไธ้ฝๅ
FlexGen ใงใฏ, huggingface ใง่ฝใจใใใขใใซใใผใฟใซๅคๆใใใใพใ.
ใใใใฃใฆ, ๅ
ใจใชใ huggingface ใฎ cache(~/.cache/huggingface/hub/)ใๅฃใใฆใใใจใใพใใใใพใใ.
(ใใจ, opt-66b ใ ใจ 14 ๅใใ weight ใใกใคใซใฎใใก, ๆๅพใฎ 1 ๅใ FlexGen ็ต็ฑใฎใใฆใณใญใผใใงใฏๆฌ ใใใฃใฝใ?)
huggingface ใฎใฉใคใใฉใชใงใฏ SHA ใใใทใฅใชใฉใงใใกใคใซใๅฃใใฆใใชใใใชใฉใฎใใงใใฏใฏใใฆใใใชใใใใงใ...
ใพใ, ใใกใคใซ่ชไฝใฏไฝใใใฎใใใทใฅใๅใฃใฆ blob ใซไฟๅญใชใฎใง, ใใกใคใซๅใฟใใ ใใงใฏใใใใพใใ.
ๅ ใใกใคใซๅใฏ
~/.cache/huggingface/hub/models--facebook--opt-66b/snapshots/7259969061237fe940036d22bea0fd349e4485e9
ใฎใใใซ snapshots ใซใใใพใ(ใใใซไธๆฎตไฝใใใใฎใใใทใฅใๅใฃใฆใใฉใซใไฝใฃใฆใใใใใใใ)
ใใใใ blob ใธใฎใทใณใใชใใฏใชใณใฏใฎๅฝขใงใใกใคใซใใใใพใใฎใง, ใใฆใณใญใผใๅคฑๆใใใฎใฏใใฎ symlink ใจ blob ใฎไธกๆนใฎใใกใคใซใๆถใใฆๅๅบฆใใฆใณใญใผใใใพใ.
(ใใฆใณใญใผใใใใใกใคใซๅใฏใชใซใๅฅ้ DB ใง็ฎก็ใใฆใใใใใงใฏใชใใฎใง, ๅใซไธ้ฝๅใฎใใใใกใคใซใๆถใใฐ OK ใ ใฃใ)
ใขใใซใฎใใฆใณใญใผใ่ชไฝใฏ, transformers ไฝฟใฃใ็ฐกๅใชในใฏใชใใใใใฆ,
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("facebook/opt-66b", torch_dtype=torch.float16).cuda()
ใใใใชๆใใง.
opt-175b ใฏ?
ไธๅฟ CPU mem 256 GB ใใใฐๅใใใใงใใ, opt-175b ใฏ่ฝใจใใฎใซใชใฏใจในใใๅฟ
่ฆใงใ.
้ใใ่จฑๅฏใใใใฐ่ฉฆใใฆใฟใใใจใใ.
Discussion