第6å é³å£°æžãèµ·ãããšããã³ããåŠçã飿ºãã
æ¬ãšã³ããªã¯Ubie çæAI Advent Calendar 2024ã®17æ¥ç®ãã瀟å çšçæAI Webã¢ããªã±ãŒã·ã§ã³ãã©ã®ããã«äœã£ãŠããããã®ç¬¬6åã§ãã
ååã¯ã第5å Webæ€çŽ¢æ©èœã«ãã£ãŠçæAIãšã®äŒè©±äžã®ç¥èã匷åããã«ã€ããŠèª¬æããŸãããä»åã¯ãé³å£°æžãèµ·ãããšããã³ããåŠçãšã®é£æºã«ã€ããŠè§£èª¬ããŸãã
æå€ãšãããGeminiã·ãªãŒãºã®é³å£°åŠç
é³å£°ã®æåèµ·ããã¯çæAIãå°é ãã以åãããœãªã¥ãŒã·ã§ã³ã沢山ååšããŠããŸãããçæAIã«ãã£ãŠå€èšèªãžã®å¯Ÿå¿ããã€ãºãã¢ã¯ã»ã³ãã®éãã«åŒ·ããªã©ã®äžå®ã®ãã¬ãŒã¯ã¹ã«ãŒãèµ·ããããã§ãããçè ã¯ãã®èŸºãã®æŽå²ã«ã¯ããŸã詳ãããªãã§ãã
äžæŠæ¢åã®ãœãªã¥ãŒã·ã§ã³ã¯çœ®ããŠãããŠãçæAIã®äžçã§ã®é³å£°åŠçã«çŠç¹ãåœãŠãå Žåã®éžæè¢ã«ã€ããŠèããŠã¿ãŸããäž»èŠãªãã®ãšããŠOpenAIã®WhisperãšGoogleã®Geminiã·ãªãŒãºããããŸãã
| API | ãã¡ã€ã«ãµã€ãºå¶é | 察å¿ãã©ãŒããã | é³å£°ã®æå€§ã®é·ã |
|---|---|---|---|
| Whisper | 25MB | mp3, m4a, wav, webm, mp4, mpeg, mpga | ãã¡ã€ã«ãµã€ãºã«äŸåããããæŠã30åçšåºŠ |
| Gemini Flash | å¶éç¹ã«ãªã | mp3, m4a, wav, webm, aiff, aac, ogg, flac | 8.4æé以äžãŸãã¯æå€§100äžããŒã¯ã³ |
| Gemini Pro | å¶éç¹ã«ãªã | mp3, m4a, wav, webm, aiff, aac, ogg, flac | 8.4æé以äžãŸãã¯æå€§100äžããŒã¯ã³ |
ã©ã¡ããæžãèµ·ããç²ŸåºŠã¯æªããªããæ§èœå·®ã¯ãããŸã§æããããŸããããã®ãŸãŸã§ãåå䜿ããŸããããã¯ãåºæåè©ãªã©ã«ã¯åŒ±ãå Žåãªã©ããããCloud Speech-to-Textãäºåã®èŸæžããŒã¿ãçµã¿åãããŠç²ŸåºŠãäžããäŸãªã©ãããããã§ãã
GeminiãšSpeech-to-Textã§å®çŸããé«ç²ŸåºŠãªæåèµ·ãã
WhisperãšGeminiãæ¯èŒããŠã¿ããšãæå€§ã®ãã¡ã€ã«ãµã€ãºã«å€§ããªå·®ããããŸããWhisperã¯1ãªã¯ãšã¹ãã§25MBãŸã§ãªã®ã§ãmp3çã§å§çž®ããŠã倧äœ30åçšåºŠã®é³å£°ãŸã§ãéçã§ããäžæ¹ãGeminiã¯1ãªã¯ãšã¹ãã§8.4æé以äžãŸãã¯æå€§100äžããŒã¯ã³ãªã®ã§ãæ®ã©ã®ãŠãŒã¹ã±ãŒã¹ã«çœ®ããŠååãªãµã€ãºã§ã¯ãªãããšæããŸãã
Dev Genius[1]ã®é³å£°æžãèµ·ããæ©èœã§ã¯ãGemini 1.5 Proã䜿çšããŠããŸãã
äºåã®ããã³ãããèšå®å¯èœã«ãã
é³å£°ãæžãèµ·ãããšãã£ãŠãæ§ã ãªç®çããããŸããåçŽã«æ£ç¢ºã«æžãèµ·ããããå Žåãããã°ãé³å£°ã®å 容ãèŠçŽãããå Žåãªã©ãããã§ãããã
ãã®ãããäºåã«ããã³ãããèšå®å¯èœã«ããŠããããšã§ãç®çã«åãããŠæè»ã«å¯Ÿå¿ã§ããããã«ããŠããŸãã

以äžã¯MP3ã®ãã¡ã€ã«ããã©ãŒã ããåãåã£ãŠãGemini 1.5 Proã§é³å£°ãæžãèµ·ããServer Actionsã®å®è£ äŸã§ãã
"use server";
import { VertexAI, GenerativeModel, InlineDataPart } from "@google-cloud/vertexai";
export const createTranscription = async (formData: FormData) => {
// formDataãããã¡ã€ã«ãšããã³ãããåãåã
const uploadFile = formData.get("file") as File | null;
const prePrompt = formData.get("prePrompt") as string | null;
if (!uploadFile || uploadFile.type !== "audio/mp3") {
console.error("Invalid file");
return;
}
// ã»ããšã¯å¥ã«base64ã«ããªããŠããããã
const arrayBuffer = await uploadFile.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
const base64 = buffer.toString("base64");
const vertexAI = new VertexAI({
project: process.env.VERTEX_PROJECT_ID,
location: "asia-northeast1",
});
if (!vertexAI) {
console.error("Vertex AI not found");
return;
}
const generativeModel = vertexAI.getGenerativeModel({
model: "gemini-1.5-pro-001",
});
const inlinePart: InlineDataPart = {
inlineData: {
data: base64,
mimeType: "audio/mp3",
},
};
const textPart = {
text: prePrompt || `é³å£°ãæžãèµ·ãããŠãã ãããèªã¿ãããããã«å¥èªç¹ãæ¹è¡ã远å ãèªã¿ãããããŠãã ããã`,
};
const request = {
contents: [{ role: "user", parts: [inlinePart, textPart] }],
};
const transcriptionPromise = generativeModel.generateContent(request);
return transcriptionPromise
.then(result => {
const text = result.response.candidates
?.map(candidate => candidate.content.parts.map(c => c.text))
.join("\n");
return text;
});
};
å®éã«è©ŠããŠã¿ããšã以äžã®ãããªçµæãåŸãããŸãããæžãèµ·ãã察象ã®åç»ã¯ https://www.youtube.com/watch?v=NIQDnWlwYyQ ãå©çšããŸããã

å
ã¯è±èªã ãã©æ¥æ¬èªã«ããŠããããã§ãã
äºåããã³ãããå€ãããšä»¥äžã®ããã«ãªããŸãã
å
容ãèŠçŽãç®æ¡æžãã«ããŠãã ããã

ããã¯ããã§ãããããã
ããªã»ãããšé£æºãã
ã»ãšãã©ã®å ŽåãäžæŠé³å£°ããã®ãŸãŸæžãèµ·ããããšã«ãªãã§ãããããã®åŸã®å å·¥ãèªç±ã«è¡ããããã§ãã
Dev Geniusã§ã¯ãæžãèµ·ããå 容ãç°¡åã«ãã£ããã§åŠçã§ããããã«ããŠããŸããããªã»ãããšçµã¿åãããããšã§ãç°¡åã«ç®çã®åºåãåŸãããŸãã

ChatGPTã®ãããªãšç»é¢å
±æã®æ©èœã¯æ°ã«ãªããŸãã
ããŒãã£ã³ã°ã®åç»çã®å Žåã§ããã°ãå 容ã®èŠçŽã®ã»ãã«ãèªåã®ãã¯ã¹ãã¢ã¯ã·ã§ã³ãæœåºãããããããã¯ãã®ã¢ã¯ã·ã§ã³ã«åããŠçžè«ããããã§ããŸãã
ããŸããffmpegããã©ãŠã¶ã§äœ¿ãããšã§åç»å€æãªãœãŒã¹ã忣ããã
Gemini 1.5 Proã¯åç»ãã¡ã€ã«ã®åŠçãµããŒãããŠããŸãããé³å£°ä»ãã®å Žåã¯50åãŸã§ãäžéãšãªããŸããåç»ã®å 容çè§£ãç®çãšããæ©èœãªã®ã§ãé³å£°æžãèµ·ããã«å©çšããã®ã¯ãããªãŒããŒã¹ããã¯ã§ããã€ã€ãåŠçå¯èœãªé·ãã¯ç©è¶³ããŸããã
ãªã®ã§ãåç»ãã¡ã€ã«ã®å Žåã¯é³å£°ãã¡ã€ã«ã«å€æããäžã§æžãèµ·ããã®åŠçãè¡ããããªããŸããåœåã¯ãµãŒãåŽã§ffmpegãã€ã³ã¹ããŒã«ãã倿åŠçãè¡ã£ãŠããŸãããçŸåšã§ã¯WebAssemblyãçšããŠãããã³ããšã³ãã§ffmpegãåãããåç»ãé³å£°ã«å€æããããšã§ãªãœãŒã¹ã忣ãããããã«ããŠããŸãã以äžã¯éžæãããã¡ã€ã«ãmp3ã«å€æããäŸã§ãã倿ãããã¡ã€ã«ãFormDataã«è¿œå ããŠPOSTããããšã§ããµãŒãåŽã§ã¯åžžã«é³å£°ãã¡ã€ã«ã ããæèããã°è¯ããªããŸãããã®å®è£ ã¯@yukukotaniãè¡ã£ãŠãããŸããã
import { FFmpeg } from "@ffmpeg/ffmpeg";
import { fetchFile, toBlobURL } from "@ffmpeg/util";
import { ProgressEventCallback } from "@ffmpeg/ffmpeg/dist/esm/types";
const ffmpeg = new FFmpeg();
const FFMPEG_CORE_URL = "https://unpkg.com/@ffmpeg/core@0.12.6/dist/umd";
async function convertVideoToMp3(file: File, onProgress: (progress: number) => void): Promise<File> {
await ffmpeg.load({
coreURL: await toBlobURL(
`${FFMPEG_CORE_URL}/ffmpeg-core.js`,
"text/javascript",
),
wasmURL: await toBlobURL(
`${FFMPEG_CORE_URL}/ffmpeg-core.wasm`,
"application/wasm",
),
});
await ffmpeg.writeFile(file.name, await fetchFile(file));
const progressCallback: ProgressEventCallback = (e) => {
onProgress(Math.floor(e.progress * 100));
};
ffmpeg.on("progress", progressCallback);
await ffmpeg.exec(["-i", file.name, "-f", "mp3", "output.mp3"]);
const mp3Data = await ffmpeg.readFile("output.mp3");
const mp3File = new File([mp3Data], "output.mp3", {
type: "audio/mp3",
});
ffmpeg.off("progress", progressCallback);
ffmpeg.deleteFile(file.name);
return mp3File;
}
ãŸãšã
瀟å
ã®çæAI Webã¢ããªã±ãŒã·ã§ã³ã«ãããé³å£°æžãèµ·ãããšããã³ããåŠçã®é£æºãããŸãã§WebAssemblyãçšããŠããã³ããšã³ãã§é³å£°å€æããåŠçã«ã€ããŠèª¬æããŸãããã»ãŒAPIãå©çšããŠããã ããªã®ã§ã·ã³ãã«ãªäœãã§ããä»åŸã¯æžãèµ·ããããŒã¿ã®å
±æèšå®ããäºåããã³ããã§ã®ããªã»ããå©çšãªã©ããã€ãã®æ¹åãèããããŸããé³å£°æžãèµ·ããã¯å€§æµã¯ã³ã·ã§ããã§çµãããããã®åŸèŠçŽãã¬ããŒãã£ã³ã°ãæ
å ±æœåºãªã©ãå¿
èŠãšããå Žåãå€ãã§ãããããã®ç¹ã«ã€ããŠã¯ã¯ãŒã¯ãããŒã«é¢ãããšã³ããªã§ãŸãè§ŠããããšæããŸãã
-
Ubieã®ç€Ÿå ã§å 補ããŠããçæAI Webã¢ããªã±ãŒã·ã§ã³ã§ããæ®æ®µã¯devçºãšåŒã°ããŠããŸãã â©ïž
Ubieæ ªåŒäŒç€Ÿã®ããã¯ããã°ã§ãã æ¡çšæ å ±ïŒrecruit.ubie.life
Discussion