Md. Asif Uddin
    Problem I.11.B05

    What a depthwise factorisation saves

    complexity▲▲△

    Exact integers and fractions where possible; six decimals otherwise.

    STATEMENT

    Compare a standard convolution with a depthwise-plus-pointwise replacement.

    GIVEN

    One example has 32 input channels, 64 output channels and a 2828 by 2828 output grid. The standard kernel is 33 by 33. The replacement uses one depthwise 33 by 33 filter per input channel followed by a 11 by 11 pointwise layer to 64 channels. Ignore biases and nonlinearities in both cases.

    FIND

    Weights and multiply-accumulates for both designs, the cost ratio, and a reason why equal shapes do not make their function classes equal.

    STRATEGY

    Count the depthwise and pointwise weights separately, then multiply by the number of output positions.

    SOLUTION

    The standard layer has 9⋅32⋅64=184329\cdot32\cdot64=18432 weights. The factorised version has 9⋅32=2889\cdot32=288 depthwise weights and 32⋅64=204832\cdot64=2048 pointwise weights, for a total of 23362336.

    Since 282=78428^2=784, the main-product counts are

    Mstandard=18432(784)=14450688,Mfactorised=2336(784)=1831424.\begin{aligned} M_{\mathrm{standard}}&=18432(784)\\ &=14450688,\\ M_{\mathrm{factorised}}&=2336(784)\\ &=1831424. \end{aligned}

    The ratio is

    233618432=164+19≈0.126736.\frac{2336}{18432}=\frac1{64}+\frac19\approx0.126736.

    That is about 7.897.89 times fewer main-product operations in this counting model, not a measured latency speedup.

    Without an intervening nonlinearity, the effective kernel has the form Ko,c,u,v=Po,cDc,u,vK_{o,c,u,v}=P_{o,c}D_{c,u,v}. For each input channel, all output channels use scalar multiples of its same spatial filter. A general convolution does not impose this restriction.

    Answer

    Standard: 18,432 weights and 14,450,688 MACs. Factorised: 2,336 weights and 1,831,424 MACs. The smaller operation is a constrained model, not an exact replacement for every standard kernel.

    Check — sanity

    Both weight counts and MAC counts have the same ratio because the two designs use the same spatial grid. Biases and intermediate activations were explicitly excluded.

    Where this breaks

    Real runtime also depends on memory traffic, kernel efficiency and hardware. Adding a nonlinearity between the layers changes the function again; it does not remove the need to evaluate accuracy after replacement.

    Variation

    For spatial width kk and output width CoutC_{\mathrm{out}}, the corresponding ratio is 1/Cout+1/k21/C_{\mathrm{out}}+1/k^2. At k=1k=1 the extra depthwise layer does not create the same saving.

    Draws on