Quickstart#

Embedl Deploy keeps deployment preparation inside PyTorch. transform() rewrites your model’s graph into a deployable form using the active backend’s patterns. If a single backend is installed it is selected automatically; if several are present you must call set_backend() first or transform() will raise an error.

Usage#

import torch
from embedl_deploy import transform
from embedl_deploy.quantize import quantize
from torchvision.models import resnet18 as Model

# 1. Load a standard PyTorch model
model = Model().eval()
example_input = torch.randn(1, 3, 224, 224)

# 2. Transform — fuse and optimize for TensorRT in one call
res = transform(model, (example_input,))
print("Model")
res.model.print_readable()
print("Matches", "\n".join([str(match) for match in res.matches]))


# 3. Quantize (PTQ)
def forward_loop(model: torch.fx.GraphModule):
    model.eval()
    for _ in range(100):
        model(torch.randn(1, 3, 224, 224))


quantized_model = quantize(
    res.model, (example_input,), forward_loop=forward_loop
)
quantized_model.eval()

# 4. Export as usual (dynamo exported models may have compilation issues)
torch.onnx.export(
    quantized_model, (example_input,), "model.onnx", dynamo=False
)

# 5. Quantization-aware training with a training loop
qat_model = quantized_model.train()
# Freeze BatchNorm, or apply other QAT utilities as needed
# train(qat_model)

Compile#

Compilation can be done with TensorRT’s trtexec tool, which can take the ONNX model and compile it for inference. The exported layer info and profile can be used for debugging, optimization and visualization.

Note that the ONNX model might need to be simplified with onnx-simplifier to make trtexec compile it. Dynamo exported models may have compilation issues, so it’s recommended to export with dynamo=False.

onnxsim model.onnx model.onnx
/usr/src/tensorrt/bin/trtexec --onnx=model.onnx --fp16 --int8 --useCudaGraph

Optionally you can get the layer profile with the following flags:

--exportLayerInfo=layer_info.json
--exportProfile=profile.json
--profilingVerbosity=detailed

Next steps#