推論系統
TensorRT Model Connect Enters Public Preview: Package Hugging Face Models for Native C++ Inference Without ONNX
NVIDIA has open-sourced TensorRT Model Connect, which uses two commands to turn supported Hugging Face or local checkpoints into versioned bundles that can be handed off to C++ applications. It shortens the model-porting path, but remains a reference implementation; support and performance must be validated for each model, precision, and hardware configuration.
NVIDIA has released TensorRT Model Connect in an effort to close the engineering gap between training PyTorch models and reliably shipping them in native applications. The quickstart uses Qwen3-0.6B as an example: `trtmc build` reads a Hugging Face checkpoint directly and produces a `.bundle` containing the TensorRT engine, model configuration, and runtime assets; `trtmc run` can test it immediately, while the same bundle can also be loaded from C++ with `trtmc::load()`. This workflow does not require an intermediate ONNX export, avoiding common failure points such as unsupported operators, dynamic shapes, and exporter version mismatches.
The project is more than a CLI wrapper for language models. Its architecture provides model-family-specific builders, runtime pipelines, auxiliary kernels, and validation contracts, while task APIs for text generation, transcription, segmentation, embeddings, and image or video generation isolate applications from the underlying implementation. For teams handing off Python prototypes to low-latency C++, edge devices, or existing services, versioned bundles are also easier to integrate into release workflows than scattered weights, plugins, and build parameters. The repository additionally includes agent-specific guidance and skills intended to help coding agents add and test support for new models.
However, NVIDIA explicitly labels Model Connect as a reference implementation, not a universal compiler capable of importing any model. Exact checkpoint, precision, quantization method, and hardware eligibility remain subject to the support matrix. For optimized edge deployment of LLMs and VLMs, NVIDIA also recommends evaluating TensorRT Edge-LLM first. Engineering teams should next monitor how quickly new architectures are added, cross-version bundle compatibility, real-world performance benchmarks, and the supply-chain security of custom operators and third-party native libraries.