Operator Fusions#

Fusions combine sequences of operators into single fused modules that map directly to hardware-accelerated kernels. After fusion, the model graph contains modules like FusedConvBNAct instead of separate Conv2d, BatchNorm2d, and ReLU layers.

Fused modules:

Are numerically equivalent to the original operator sequence (no weight folding at this stage — that’s handled by the hardware compiler).
Export cleanly to ONNX for downstream compilation.

Each FusedModule subclass declares inputs_to_quantize — the set of positional input indices where the quantization pipeline places QDQ stubs. See Quantization for the full table.

Convolution fusions#

ConvBNPattern#

Matches: Conv2d → [BatchNorm2d]

Produces: FusedConvBN

The most basic convolution fusion. The BatchNorm2d is optional — a bare Conv2d is also matched (useful for ensuring QDQ stub placement even on convolutions without batch normalization).

ConvBNActPattern#

Matches: Conv2d → [BatchNorm2d] → Activation

Produces: FusedConvBNAct

Fuses convolution, optional batch normalization, and an activation function. The pattern accepts any activation in ActivationLike: ReLU, ReLU6, LeakyReLU, ELU, GELU, SiLU, Mish, Hardswish, Hardsigmoid, PReLU, Sigmoid, Tanh.

StemConvBNActMaxPoolPattern#

Matches: Conv2d(3in, 7×7) → [BatchNorm2d] → Activation → MaxPool2d

Produces: FusedConvBNActMaxPool

Captures the classification network stem found in ResNet and similar architectures. The convolution is constrained to in_channels=3, kernel_size=(7,7) so only the actual stem is matched.

ConvBNAddActPattern#

Matches: Conv2d → BatchNorm2d → add(·, residual) → Activation

Produces: FusedConvBNAddAct

Captures the tail of ResNet-style bottleneck blocks where the convolution path merges with a skip connection before the final activation. This is a branching pattern — it matches a Fork topology with two inputs feeding into an operator.add node.

Linear fusions#

LinearActPattern#

Matches: Linear → Activation

Produces: FusedLinearAct

LinearPattern#

Matches: standalone Linear

Produces: FusedLinear

LayerNormPattern#

Matches: LayerNorm

Produces: FusedLayerNorm

ActAddPattern#

Matches: Fork(Activation, residual) → operator.add

Produces: FusedActAdd

Captures the inverted residual pattern where an activation precedes the skip-connection add (e.g. in transformer feed-forward blocks). Like ConvBNAddActPattern, this is a branching pattern matching a Fork topology.

MulAddPattern#

Matches: Fork(operator.mul, residual) → operator.add

Produces: FusedMulAdd

Captures the layer-scale residual tail used by ConvNeXt-style blocks: a runtime tensor scaled by a constant (a parameter or a scalar literal), then added to a skip connection. Fully dynamic multiplies (e.g. SE-style gating) and adds whose other operand is a weight are not matched. The fused module declares no quantized inputs, keeping the layer-scale residual in floating point — INT8 Q/DQ on this operation measurably hurts ConvNeXt accuracy while the latency cost of floating point is negligible.

Attention fusions#

These patterns match the sub-modules produced by the DecomposeMultiheadAttentionPattern and DecomposeSwinAttentionPattern conversions.

MHAInProjectionPattern#

Matches: MHAInProjection

Produces: FusedMHAInProjection

ScaledDotProductAttentionPattern#

Matches: ScaledDotProductAttention

Produces: FusedScaledDotProductAttention

SwinAttentionPattern#

Matches: SwinAttention

Produces: FusedSwinAttention

Wraps the Swin Transformer window-attention module produced by DecomposeSwinAttentionPattern. The fused module carries shared spatial state (SwinSpatialState) for window partition/reverse geometry.

Pooling fusions#

AdaptiveAvgPoolPattern#

Matches: AdaptiveAvgPool2d

Produces: FusedAdaptiveAvgPool2d

Tip

In mixed-precision workflows the AdaptiveAvgPoolPattern is often omitted entirely from the pattern list so the pool is not fused and stays as a plain AdaptiveAvgPool2d in the graph. See Custom Patterns for details.

Fusion summary by architecture#

ResNet50#

Fused module	Count	Description
`FusedConvBNActMaxPool`	1	Stem: Conv(7×7) + BN + ReLU + MaxPool
`FusedConvBNAddAct`	16	Bottleneck residual blocks
`FusedConvBNAct`	16	Main-path Conv + BN + ReLU
`FusedConvBN`	17	Conv + BN without activation
`FusedAdaptiveAvgPool2d`	1	Global average pool
Total	51

Conversions applied: FlattenLinearToConv1x1Pattern converts the Flatten → Linear classifier into Conv2d(1×1) → Flatten.

ConvNeXt (Tiny/Base/Large)#

Fused module	Count	Description
`FusedConvBN`	23/41/41	Depthwise + downsampling convolutions
`FusedLayerNorm`	23/41/41	LayerNorm layers
`FusedLinearAct`	18/36/36	Linear + GELU chains
`FusedLinear`	18/36/36	Standalone linear layers
`FusedMulAdd`	18/36/36	Layer-scale residual tails (kept in FP)
`FusedAdaptiveAvgPool2d`	1/1/1	Global average pool

Counts increase with model depth (Base and Large share the block count and differ in channel width).

Conversions applied:

RemoveIdentityAdaptiveAvgPoolPattern removes identity pooling ops.
FlattenLinearToConv1x1Pattern converts the classifier head.

ConvNeXt uses depthwise separable convolutions extensively. The default pattern set quantizes all convolutions equally, but mixed-precision (see Custom Patterns) skips depthwise convolutions for better latency.

Vision Transformer (ViT-B/16)#

Fused module	Count	Description
`FusedConvBN`	1	Patch embedding Conv2d
`FusedMHAInProjection`	12	Q/K/V projections
`FusedScaledDotProductAttention`	12	Attention computation
`FusedLinear`	36+	Out-proj + MLP linear layers
`FusedLinearAct`	12	MLP hidden → GELU chains
`FusedLayerNorm`	25	Pre/post-norm layers

Conversions applied: DecomposeMultiheadAttentionPattern decomposes all 12 attention layers into explicit sub-modules.

Running fusions only#

from embedl_deploy import transform
from embedl_deploy.tensorrt import TENSORRT_FUSION_PATTERNS

result = transform(model, (example_input,), patterns=TENSORRT_FUSION_PATTERNS)

To inspect what was fused:

from collections import Counter

fused_counts = Counter(
    type(m).__name__
    for m in result.model.modules()
    if type(m).__name__.startswith('Fused')
)
for name, count in sorted(fused_counts.items()):
    print(f"  {name}: {count}")