Version 0.17.0 makes repeated prompts faster, adds new vision and audio models, and brings more ways to tune the Engine for different GPUs.

Highlights

• Faster repeated prompts with prefix caching - The dynamic Engine can now reuse matching prompt prefixes across requests. Prefix caching is enabled automatically for supported paged KV-cache layouts.
• Runtime profiles for different GPUs - A single model package can select Engine settings and model variants based on available GPU memory. New capability APIs also make it easier for applications to discover Engine limits.
• New LFM2 models - Added support for LFM2-VL and LFM2.5-VL vision models, LFM2-MoE, and LFM2-Audio and LFM2.5-Audio with speech input and output.
• Moonshine streaming speech recognition - Added Moonshine small and tiny streaming ASR models, designed for fast transcription with low CPU use.
• Faster DFlash decoding - DFlash now supports CUDA graph capture, prepacked weights, optional projection fusion, sampled speculative decoding, and direct Qwen 3.8 package export.
• Lower GPU memory pressure - Embedding lookups can be offloaded to CPU while keeping CUDA graph capture enabled for decoding.
• Less work during generation - Paged decoders can compute logits only for the token rows needed for generation and speculative verification.
• Safer conversation editing - Engine requests can rewind to the start of an earlier turn and continue from there without creating a new request.
• A better Model Builder configuration - The new structured configuration organizes target, drafter, speculative decoding, runtime, and search settings in one validated format.
• More Engine flexibility - Added configurable paged block sizes, improved speculative draft controls, and APIs for maximum request length.
• Reliability improvements - Includes fixes for WebGPU graph capture, decoder pipeline KV-cache shapes, paged prefill settings, legacy execution providers, and prefill logits.

This is a big release! Version 0.16.0 brings major upgrades for high-throughput serving, faster decoding, and the latest generation of Qwen models.

Highlights

• A much more capable Engine - Continuous batching now supports chunked prefill, paged attention, multi-turn conversations, mixed workloads, and smarter token scheduling. This improves throughput while making better use of GPU memory.
• Faster generation with speculative decoding - Added support for draft models, n-gram speculation, MTP self-speculation, and DFlash2 and DSpark block drafting.
• Tool calling and constrained decoding - The Engine can generate tool calls, continue after tool results without replaying the full prompt, and enforce structured output with guidance.
• Better multi-turn control - Requests can retain their KV cache between turns. Sampling, token limits, seeds, stop strings, and guidance can be changed for each turn.
• More Qwen support - The model builder now supports Qwen2.5-VL, Qwen3-VL, and Qwen3.5 dense and MoE models, with additional export and speculative-decoding support for Qwen3.6 and Qwen3.8.
• More efficient KV caches - Added paged, windowed, sparse, and quantized KV-cache support, including INT4, INT8, and FP8 paths.
• New quantized model formats - Added native support for NVFP4 and FP8 checkpoints from NVIDIA Model Optimizer and compressed-tensors, plus the quant_auto format.
• New hardware support - Added the AMDGPU execution provider and WebGPU paged-attention exports.
• New model support - Added Granite models and improved Marian batching, multimodal prefill, Nemotron audio, and multi-image vision handling.
• Stronger reliability - This release includes broad validation and recovery improvements across the Engine, speculative decoding, vision and audio inputs, cache management, and process shutdown.

What's Changed

• Add per_layer_inputs support for Gemma4 and fix pixel_position_ids by Akshay Sonawane (@apsonawane) in #2188
• Enable Qwen3.5 35B-A3B MoE text export for TRT-RTX EP by Yen-Shi Wang (@yen-shi) in #2186
• Multilingual model ids expansion in the sample/example based on new Nvidia's README by Nenad Banfic (@nenad1002) in #2190
• Fix webgpu shared buffer issue by Akshay Sonawane (@apsonawane) in #2191
• Fix nightly build by kunal-vaishnavi in #2193
• Clean up base constructor in model builder by kunal-vaishnavi in #2192
• Fix CUDA illegal memory access crash in TopK when inputs contain NaN by David Fan (@jiafatom) in #2195
• Fix: Skip CPU provider in SetProviderSessionOptions by David Fan (@jiafatom) in #2179
• Add README file for testing folder by kunal-vaishnavi in #2205
• Add real-model integration test pipeline by Baiju Meswani (@baijumeswani) in #2198
• Update Extensions for Harmony/GPT-OSS Tool Calling Support by Sayan Shaw (@sayanshaw24) in #2200
• Forward user provider options to allocator init session by Jiajia Qin (@qjia7) in #2177
• Add grid dimension validation by Akshay Sonawane (@apsonawane) in #2219
• Support Hunyuan Dense V1 export by Sunghoon Choi (@hanbitmyths) in #2189
• Fix model builder for gpt-oss by Tianlei Wu (@tianleiwu) in #2228
• Update main to version 0.15.0 by Baiju Meswani (@baijumeswani) in #2183
• Build onnxruntime-genai-cuda packages on win-arm64 by Baiju Meswani (@baijumeswani) in #2199
• Bump torch from 2.7.1+cpu to 2.12.0+cpu in /test/python/directml/torch by Dependabot (@dependabot)[bot] in #2230
• Fix BFloat16 logits returned as garbage in Logits::Get() by Justin Chu (@justinchuby) in #2203
• Fix DefaultKeyValueCache: per-layer num_kv_heads for Gemma 4 dual/MQA attention by shreyshah-microsoft in #2214
• Fix security vulnerabilities by Akshay Sonawane (@apsonawane) in #2224
• Fix divide by zero error by Akshay Sonawane (@apsonawane) in #2226
• Load models from ONNX Runtime model packages by Jambay Kinley (@jambayk) in #2227
• fix(ci): use new identity client ID by Sanaa Hamel (@sanaa-hamel-microsoft) in #2246
• Use Linux A10 Agents for Integration Tests by Baiju Meswani (@baijumeswani) in #2248
• Fix OrtGlobals::Allocator destruction order by Edward Chen (@edgchen1) in #2249
• Add validation for input data by Akshay Sonawane (@apsonawane) in #2220
• Add upper bounds to dependencies by Akshay Sonawane (@apsonawane) in #2221
• Add check for malformed audio files by Akshay Sonawane (@apsonawane) in #2225
• Route init-session provider-option shaping through DeviceInterface by Jiajia Qin (@qjia7) in #2232
• Release captured graph resources when generator is destroyed by Jiajia Qin (@qjia7) in #2106
• feat: add audio content blocks for Gemma multimodal prompts in model_mm by Chang Liu (@cliu1003) in #2240
• Fix tied embeddings by kunal-vaishnavi in #2237
• Enable zero-copy for QNN GPU by qti-mattsinc in #2105
• fix(ci): use GHCR instead of ACR for workflow docker caches by Sanaa Hamel (@sanaa-hamel-microsoft) in #2264
• Fix new-delete-type-mismatch in C API opaque handle destroy functions by Akshay Sonawane (@apsonawane) in #2271
• Add regression tests for top_k exceeding vocab_size by David Fan (@jiafatom) in #2265
• Add bounds checking to OgaSequencesGetSequence C API functions by David Fan (@jiafatom) in #2260
• webgpu: fix RecurrentState graph capture with shared buffer aliasing by Jiajia Qin (@qjia7) in #2244
• Reject vocab_size < 2 for beam search to fix OOB in BeamSearch_Cpu::SelectTop by David Fan (@jiafatom) in #2272
• Update trust_remote_code by Akshay Sonawane (@apsonawane) in #2256
• Validate eos_token_id against vocab_size to prevent OOB write in ApplyMinLength by David Fan (@jiafatom) in #2266
• Fix use-after-free by making GeneratorParams co-own its Model by David Fan (@jiafatom) in #2270
• Fix arbitrary dll load via json injection by Akshay Sonawane (@apsonawane) in #2269
• Resolve model package paths and path-valued session options through ONNX Runtime by Jambay Kinley (@jambayk) in #2255
• Fall back to the system D3D12 runtime when the Agility SDK device factory cannot create a device by Gianluca Mazza (@gianlucamazza) in #2280
• Support UpdatePositionIds in webgpu's interface by Jiajia Qin (@qjia7) in #1952
• Fix UAF in adapter lifecycle: serialize Adapters ops with a mutex by Akshay Sonawane (@apsonawane) in #2274
• Fix AppendNextTokensToSequences heap overflow by Akshay Sonawane (@apsonawane) in #2111
• hack(ci): temporarily disable GHCR cache by Sanaa Hamel (@sanaa-hamel-microsoft) in #2297
• Enable re-init after OgaShutdown by Scott McKay (@skottmckay) in #2292
• Add int8 precision support to model builder by danielsongmicrosoft in #2275
• DML: allow opting out of graph capture via provider option by Gianluca Mazza (@gianlucamazza) in #2300
• Detect model packages by manifest.json without a root genai_config.json by Jambay Kinley (@jambayk) in #2298
• Add multi_rotary_cache_concat_offset for Phi-4 model in TRT-RTX EP by Yen-Shi Wang (@yen-shi) in #2276
• LFM2: add shared-buffer KV cache support to LFM2Cache by Tachion (@SanjayAMD) in #2284
• Update model builder for gpt-oss by Tianlei Wu (@tianleiwu) in #2234
• Support transformers v5 rope_parameters in model builder by Tianlei Wu (@tianleiwu) in #2311
• TurboQuant for ORT WebGPU along with model_bechmark updates to measure GPU memory by Sushanth Rajasankar (@sushraja-msft) in #2084
• Use lintrunner to format all files by Tianlei Wu (@tianleiwu) in #2317
• Bump torch from 2.7.1+cpu to 2.12.1+cpu in /test/python/cpu/torch by Dependabot (@dependabot)[bot] in #2305
• Bump torch from 2.12.0+cpu to 2.12.1+cpu in /test/python/directml/torch by Dependabot (@dependabot)[bot] in #2303
• [WebGPU] implements webgpu support in benchmarking script by Prathik Rao (@prathikr) in #2318
• Roll onnxruntime-extensions to pick up Gemma4LogMel OOB fix (#1092) by Sushanth Rajasankar (@sushraja-msft) in #2320
• Fix DML readback size by Sushanth Rajasankar (@sushraja-msft) in #2319
• Update torch package versions for CIs by kunal-vaishnavi in #2322
• chore(ci): use new integration test pool on Linux by Sanaa Hamel (@sanaa-hamel-microsoft) in #2254
• Implement no_repeat_ngram_size for CPU search by Mustapha Jaber (@mustjab) in #2242
• Add GetMemoryInfo to DeviceInterface by qti-mattsinc in #2259
• Support both FP16 and FP32 I/O for Nemotron by Anatol Liu (@p3achyjr) in #2279
• Drop CWD fallback in custom_ops_library resolution by Akshay Sonawane (@apsonawane) in #2267
• Redesign unified packaging pipeline by Baiju Meswani (@baijumeswani) in #2310
• Bump requests from 2.32.3 to 2.33.0 by Dependabot (@dependabot)[bot] in #2323
• [WebGPU] append WebGPUExecutionProvider to config by Prathik Rao (@prathikr) in #2326
• Add address validation feature by Akshay Sonawane (@apsonawane) in #2223
• Update prechecks in model builder by kunal-vaishnavi in #2312
• Remove onnxruntime-genai-cuda.dll from foundry package win-arm64 rid by Baiju Meswani (@baijumeswani) in #2329
• Add Tool Cal...

What's Changed

• Fix WhisperProcessor divide-by-zero when single prompt is provided by @Copilot in #2068
• Fix lm_head tensor loading order dependency in quantized model builder by Thiago Pereira Rocha (@thpereir) in #2061
• Fail to build Whisper model by Xiaofei Han (@xiaofeihan1) in #2075
• Rename NemotronCacheConfig to NemotronConfig and add blank penalty to the decoder by Nenad Banfic (@nenad1002) in #2042
• Fix YaRN RoPE bugs in model builder and add parity tests by Ti-Tai Wang (@titaiwangms) in #2076
• Add Transformers v5 Support by Sayan Shaw (@sayanshaw24) in #2089
• macOS ARM64 ADO pipeline by @Copilot in #2091
• Reduce CPU-side per-token overhead in GenerateNextToken and SampleTopP by Sunghoon Choi (@hanbitmyths) in #2085
• Add onStageComplete by Akshay Sonawane (@apsonawane) in #2074
• [WebGPU] Support continuous decoding (RewindTo) with graph capture by Jiajia Qin (@qjia7) in #2083
• [Mistral3] Add VLM support with multi-image inference by Ti-Tai Wang (@titaiwangms) in #2077
• Add k_quant_linear mixed-precision quantization for hybrid attention … by Akshay Sonawane (@apsonawane) in #2100
• Removes QNN packaging from onnxruntime-genai pipelines by Baiju Meswani (@baijumeswani) in #2109
• Add Gemma4 multimodal support (vision + audio) by Akshay Sonawane (@apsonawane) in #2103
• Update GUIDs during az login by kunal-vaishnavi in #2122
• Add CODEOWNERS file for repository ownership by kunal-vaishnavi in #2119
• Qwen3.5: drop fp32 cast around RMSNorm in builder by Xiaofei Han (@xiaofeihan1) in #2101
• Add support for LFM2 in ORT GenAI by Joshua Lochner (@xenova) in #1979
• Enable CUDA graph capture for CUDA EP to improve decode throughput by Akshay Sonawane (@apsonawane) in #2070
• [Qwen3.5] dedup position ids by Jianhui Dai (@daijh) in #2102
• Address win-cuda pipeline errors by Baiju Meswani (@baijumeswani) in #2154
• Update Extensions Commit to Fix Id2Token Bugs by Sayan Shaw (@sayanshaw24) in #2159
• Limit the CUDA cmake architectures to 86 for CI builds by Baiju Meswani (@baijumeswani) in #2161
• Gate leaked-object error reporting in Shutdown() to debug builds or when logging is enabled by Baiju Meswani (@baijumeswani) in #2162
• Update Copilot instructions for reviewing model builder by kunal-vaishnavi in #2164
• Fix DecoderState input_ids check regression introduced in #2103 by Ti-Tai Wang (@titaiwangms) in #2148
• Fix memory leaks by Scott McKay (@skottmckay) in #2153
• [Qwen3.5] Use LpNormalization for L2-norm in linear-attention Q/K by Xiaofei Han (@xiaofeihan1) in #2127
• Fix: Win32 build failure when paths contain spaces by NSubaru (@nsubaru) in #2053
• Fix CUDA build with MSVC by enabling /Zc:preprocessor for nvcc host compilation on VS 16.5 or greater by NSubaru (@nsubaru) in #2054
• Apply linear rope_scaling in model builder for Neutts/nano by Vishal Jain (@VishalX) in #2142
• Fix Quark/AWQ weight loading for Qwen3-VL-4B text model by anilmartha in #2143
• Fix WebGPU inference crash in embedding and multi-modal feature allocation by Fei Chen (@feich-ms) in #2163
• Support Visual Studio 18 2026 build by @Copilot in #2017
• Add QNN EP documentation to OGA including Genie note by Kyle Romero (@qti-kromero) in #2158
• Use windowsml package and make winml usage simpler by Baiju Meswani (@baijumeswani) in #2155
• Cleanup TensorObject created by OrtxTensorResultGetAt by Scott McKay (@skottmckay) in #2168
• Fix nemotron leaks by Scott McKay (@skottmckay) in #2169
• [RyzenAI] make speech sub-model optional in PhiMultiModalProcessor by manasablrm in #2167
• Enable graph capture for WebGPU models and DML continuous decoding tests by Jiajia Qin (@qjia7) in #2099
• [Qwen3] Allow packed QKV MatMul under QK-Norm via post-MatMul Split by Xiaofei Han (@xiaofeihan1) in #2137
• Enable Linux ARM64 builds and packaging by Baiju Meswani (@baijumeswani) in #2107
• Add gemma4 unit tests by Akshay Sonawane (@apsonawane) in #2151
• Auto-detect fixed kv-cache shape in DefaultKeyValueCache by Oleksandr Kholodnyi (@akholodnamdcom) in #2166
• Add text-only mode support for Qwen 3.5 model builder by Akshay Sonawane (@apsonawane) in #2157
• Fix heap overflow issue by Akshay Sonawane (@apsonawane) in #2110
• [Benchmark] Add --use_random_tokens flag to C benchmark by Vishal Jain (@VishalX) in #2170
• Add HunYuan Dense V1 (hunyuan_v1_dense) model support by anilmartha in #2144
• Nvidia Parakeet Tdt ASR support by Nenad Banfic (@nenad1002) in #2150
• Multilingual Streaming Nemotron ASR + CUDA support by Nenad Banfic (@nenad1002) in #2171
• Add Csharp binding for Multi-lingual ASR by Rui Ren (@rui-ren) in #2176
• Add VideoChat-Flash (OpenGVLab) language model support by anilmartha in #2147
• Update Nemotron ASR docs by Rui Ren (@rui-ren) in #2178
• Validate sliding window size before creating KV cache by Baiju Meswani (@baijumeswani) in #2181
• Fix external weights loading for in-memory models without changing cwd by Baiju Meswani (@baijumeswani) in #2180
• Enable Qwen3.5 TRT-RTX EP path with CUDA graph by Yen-Shi Wang (@yen-shi) in #2139
• Add Qwen3.5-MoE (35B-A3B) model support by tanzeel-amd in #2146
• Update ort-extensions commit by Baiju Meswani (@baijumeswani) in #2182

New Contributors

• Ti-Tai Wang (@titaiwangms) made their first contribution in #2076
• NSubaru (@nsubaru) made their first contribution in #2053
• Vishal Jain (@VishalX) made their first contribution in #2142
• anilmartha made their first contribution in #2143
• Fei Chen (@feich-ms) made their first contribution in #2163
• Kyle Romero (@qti-kromero) made their first contribution in #2158
• manasablrm made their first contribution in #2167
• Yen-Shi Wang (@yen-shi) made their first contribution in #2139
• tanzeel-amd made their first contribution in #2146

Full Changelog: v0.13.1...v0.14.0