Learning Sparse Networks Using Targeted Dropout

Aidan N. Gomez 1,2,3 Ivan Zhang 2

Siddhartha Rao Kamalakara 2 Divyam Madaan 2

Kevin Swersky 1 Yarin Gal 3 Geoffrey E. Hinton 1

1 Google Brain 2 for.ai 3 Department of Computer ScienceUniversity of Oxford

Abstract

Neural networks are easier to optimise when they have many more weights thanare required for modelling the mapping from inputs to outputs. This suggests atwo-stage learning procedure that first learns a large net and then prunes away con-nections or hidden units. But standard training does not necessarily encourage netsto be amenable to pruning. We introduce targeted dropout, a method for training aneural network so that it is robust to subsequent pruning. Before computing thegradients for each weight update, targeted dropout stochastically selects a set ofunits or weights to be dropped using a simple self-reinforcing sparsity criterion andthen computes the gradients for the remaining weights. The resulting network isrobust to post hoc pruning of weights or units that frequently occur in the droppedsets. The method improves upon more complicated sparsifying regularisers whilebeing simple to implement and easy to tune.

1 Introduction

Neural networks are a powerful class of models that achieve the state-of-the-art on a wide range oftasks such as object recognition, speech recognition, and machine translation. One reason for theirsuccess is that they are extremely flexible models because they have a large number of learnableparameters. However, this flexibility can lead to overfitting, and can unnecessarily increase thecomputational and storage requirements of the network.

There has been a large amount of work on developing strategies to compress neural networks. Oneintuitive strategy is sparsification: removing weights or entire units from the network. Sparsity canbe encouraged during learning by the use of sparsity-inducing regularisers, like L1 or L0 penalties. Itcan also be imposed by post hoc pruning, where a full-sized network is trained, and then sparsifiedaccording to some pruning strategy. Ideally, given some measurement of task performance, wewould prune the weights or units that provide the least amount of benefit to the task. Findingthe optimal set is, in general, a difficult combinatorial problem, and even a greedy strategy wouldrequire an unrealistic number of task evaluations, as there are often millions of parameters. Commonpruning strategies therefore focus on fast approximations, such as removing weights with the smallestmagnitude [12], or ranking the weights by the sensitivity of the task performance with respect tothe weights, and then removing the least-sensitive ones [22]. The hope is that these approximationscorrelate well with task performance, so that pruning results in a highly compressed network whilecausing little negative impact to task performance, however this may not always be the case.

Our approach is based on the observation that dropout regularisation [16, 32] itself enforces sparsitytolerance during training, by sparsifying the network with each forward pass. This encourages the

Preprint. Under review.

arX

iv:1

905.

1367

8v5

[cs

.LG

] 9

Sep

201

9

network to learn a representation that is robust to a particular form of post hoc sparsification – in thiscase, where a random set of units is removed. Our hypothesis is that if we plan to do explicit posthoc sparsification, then we can do better by specifically applying dropout to the set of units that we apriori believe are the least useful. We call this approach targeted dropout. The idea is to rank weightsor units according to some fast, approximate measure of importance (like magnitude), and then applydropout primarily to those elements deemed unimportant. Similar to the observation with regulardropout, we show that this encourages the network to learn a representation where the importance ofweights or units more closely aligns with our approximation. In other words, the network learns to berobust to our choice of post hoc pruning strategy.

The advantage of targeted dropout as compared to other approaches is that it makes networksextremely robust to the post hoc pruning strategy of choice, gives intimate control over the desiredsparsity patterns, and is easy to implement 1, consisting of a two-line change for neural networkframeworks such as Tensorflow [1] or PyTorch [28]. The method achieves impressive sparsity rateson a wide range of architectures and datasets; notably 99% sparsity on the ResNet-32 architecture fora less than 4% drop in test set accuracy on CIFAR-10.

2 Background

In order to present targeted dropout, we first briefly introduce some notation, and review the conceptsof dropout and magnitude-based pruning.

2.1 Notation

Assume we are dealing with a particular network architecture. We will use θ ∈ Θ to denote thevector of parameters of a neural network drawn from candidate set Θ, with |θ| giving the numberof parameters. Ωθ denotes the set of weight matrices in a neural network parameterised by θ,accordingly, we will denote W ∈ Ωθ as a weight matrix that connects one layer to another in thenetwork. We will only consider weights, ignoring biases for convenience, and note that biases arenot removed during pruning. For brevity, we will use the notation wo ≡W·,o to denote the weightsconnecting the layer below to the oth output unit (i.e. the oth column of the weight matrix), Ncol(W)to denote the number of columns in W, and Nrow(W) to denote the number of rows. Each columncorresponds to a hidden unit, or feature map in the case of convolutional layers. Note that flatteningand concatenating all of the weight matrices in Ωθ would recover θ.

2.2 Dropout

Our work uses the two most popular Bernoulli dropout techniques, Hinton et al.’s unit dropout[16, 32] and Wan et al.’s weight dropout (dropconnect) [39]. For a fully-connected layer with inputtensor X , weight matrix W, output tensor Y , and mask M ∼ Bernoulli(1 − α) we define bothtechniques below:

Unit dropout [16, 32]:Y = (X M)W

Unit dropout randomly drops units (often referred to as neurons) at each training step to reducedependence between units and prevent overfitting.

Weight dropout [39]:Y = X(W M)

Weight dropout randomly drops individual weights in the weight matrices at each training step.Intuitively, this is dropping connections between layers, forcing the network to adapt to a differentconnectivity at each training step.

2.3 Magnitude-based pruning

A popular class of pruning strategies are those characterised as magnitude-based pruning strategies.These strategies treat the top-k largest magnitude weights as important. We use argmax-k to returnthe top-k elements (units or weights) out of all elements being considered.

1Code available at: github.com/for-ai/TD , as well as in Tensor2Tensor [38]

2

Unit pruning [26, 6]: considers the units (column-vectors) of weight matrices under the L2-norm.

W(θ) =

argmax-kwo

1≤o≤Ncol(W)

‖wo‖2∣∣∣∣W ∈ Ωθ

(1)

Weight pruning [12, 26]: considers the entries of each feature vector under the L1-norm. Note thatthe top-k is with respect to the other weights within the same feature vector.

W(θ) =

argmax-kWio

1≤i≤Nrow(W)

|Wio|∣∣∣∣ 1 ≤ o ≤ Ncol(W),W ∈ Ωθ

(2)

While weight pruning tends to preserve more of the task performance under coarser prunings[11, 35, 6], unit pruning allows for considerably greater computational savings [41, 24]. In particular,weight pruned networks can be implemented using sparse linear algebra operations, which offerspeedups only under sufficiently sparse conditions; while unit pruned networks execute standardlinear algebra ops on lower dimensional tensors, which tends to be a much faster option for given afixed sparsity rate.

3 Targeted Dropout

Consider a neural network parameterized by θ, and our importance criterion (defined above inEquations (1) and (2))W(θ). We hope to find optimal parameters θ∗ such that our loss E(W(θ∗)) islow, and at the same time |W(θ∗)| ≤ k, i.e. we wish to keep only the k weights of highest magnitudein the network. A deterministic pruning implementation would select the bottom |θ| − k elementsand drop them out. However, we would like for low-valued elements to be able to increase their valueif they become important during training. Therefore, we introduce stochasticity into the process usinga targeting proportion γ and a drop probability α. The targeting proportion means that we select thebottom γ|θ| weights as candidates for dropout, and of those we drop the elements independently withdrop rate α. This implies that the expected number of units to keep during each round of targeteddropout is (1−γ ·α)|θ|. As we will see below, the result is a reduction in the important subnetwork’sdependency on the unimportant subnetwork, thereby reducing the performance degradation as a resultof pruning at the conclusion of training.

3.1 Dependence Between the Important and Unimportant Subnetworks

The goal of targeted dropout is to reduce the dependence of the important subnetwork on its com-plement. A commonly used intuition behind dropout is the prevention of coadaptation betweenunits; that is, when dropout is applied to a unit, the remaining network can no longer depend on thatunit’s contribution to the function and must learn to propagate that unit’s information through a morereliable channel. An alternative description asserts that dropout maximizes the mutual informationbetween units in the same layer, thereby decreasing the impact of losing a unit [32]. Similar toour approach, dropout can be used to guide properties of the representation. For example, nesteddropout [29] has been shown to impose ‘hierarchy’ among units depending on the particular drop rateassociated with each unit. Dropout itself can also be interpreted as a Bayesian approximation [8].

A more relevant intuition into the effect of targeted dropout in our specific pruning scenario canbe obtained from an illustrative case where the important subnetwork is completely separated fromthe unimportant one. Suppose a network was composed of two non-overlapping subnetworks, eachable to produce the correct output by itself, with the network output given as the average of bothsubnetwork outputs. If our importance criterion designated the first subnetwork as important, and thesecond subnetwork as unimportant (more specifically, it has lower weight magnitude), then addingnoise to the weights of the unimportant subnetwork (i.e. applying dropout) means that with non-zeroprobability we will corrupt the network output. Since the important subnetwork is already ableto predict the output correctly, to reduce the loss we must therefore reduce the weight magnitudeof the unimportant subnetwork output layer towards zero, in effect “killing” that subnetwork, andreinforcing the separation between the important subnetwork and the unimportant one.

3

These interpretations make clear why dropout should be considered a natural tool for application inpruning. We can empirically confirm targeted dropout’s effect on weight dependence by comparinga network trained with and without targeted dropout and inspecting the Hessian and gradient todetermine the dependence of the network on the weights/units to be pruned. As in LeCun et al. [22],we can estimate the effect of pruning weights by considering the second degree Taylor expansion ofchange in loss, ∆E = |E(θ − d)− E(θ)|:

∆E = | − ∇θE>d+ 1/2d>Hd+O(‖d‖3)| (3)

Where di = θi if θi ∈ W(θ) (the weights to be removed) and 0 otherwise. ∇θE are the gradients ofthe loss, andH is the Hessian. Note that at the end of training, if we have found a critical point θ∗,then ∇θE(θ∗) = 0, leaving only the Hessian term. In our experiments we empirically confirm thattargeted dropout reduces the dependence between the important and unimportant subnetworks by anorder of magnitude (See Fig. 1, and Section 5.1 for more details).

4 Related Work

The pruning and sparsification of neural networks has been studied for nearly three decades andhas seen a substantial increase in interest due to their implementation on resource limited devicessuch as mobile phones and ASICs. Early work such as optimal brain damage [22] and optimal brainsurgeon [13], as well as more recent efforts [26, 34], use a second order Taylor expansion of theloss around the weights trained to a local minimum to glean strategies for selecting the order inwhich to prune parameters. Han et al. [11] combine weight quantisation with pruning and achieveimpressive network compression results, reducing the spatial cost of networks drastically. Donget al. [5] improve the efficiency of the optimal brain surgeon procedure by making an independenceassumption between layers. Wen et al. [41] propose using Group Lasso [42] on convolutional filtersand are able to remove up to 6 layers from a ResNet-20 network for a 1% increase in error.

A great deal of effort has been put towards developing improved pruning heuristics and sparsifyingregularizers [22, 13, 11, 3, 26, 5, 24, 17, 34]. These are generally comprised of two components: thefirst is a regularisation scheme incorporated into training to make the important subnetworks easilyidentifiable to a post hoc pruning strategy; the second is a particular post hoc pruning strategy whichoperates on a pre-trained network and strips away the unimportant subnetwork.

The two works most relevant to our own are L0 regularisation [24] and variational dropout [25].Louizos et al. [24] use an adaptation of concrete dropout [9] on the weights of a network and regularisethe drop rates in order to sparsify the network. Similarly, Molchanov et al. [25] apply variationaldropout [19] to the weights of a network and note that the prior implicitly sparsifies the parametersby preferring large drop rates. In addition to our methods being more effective at shrinking thesize of the important subnetwork, targeted dropout uses two intuitive hyperparameters, the targetingproportion γ and the drop rate α, and directly controls sparsity throughout training (i.e., attainsa predetermined sparsity threshold). In comparison, Louizos et al. [24] uses the Hard-Concretedistribution which adds three hyperparameters and doubles the number of trainable parameters byintroducing a unique gating parameter for each model parameter, which determines the Concretedropout rate; while Molchanov et al. [26] adds two hyperparameters and doubles the number oftrainable parameters. In our experiments we also compare against L1 regularization [12] which isintended to drive unimportant weights towards zero.

Another dropout-based pruning mechanism is that of Wang et al. [40], where a procedure is used toadapt dropout rates towards zero and one (similar to Louizos et al. [24] and [25]). We recommendGale et al. [10]’s rigorous analysis of recently proposed pruning procedures for a complete pictureof the efficacy of recent neural network pruning algorithms; in particular, it challenges some of therecent claims suggesting pruning algorithms perform about as well as random pruning procedures[4, 23].

Targeted dropout itself is reminiscent of nested dropout [29] which applies a structured form ofdropout: a chain structure is imposed on units, and children are deterministically dropped whenevertheir parent is dropped. In effect, each child unit gets a progressively higher marginal drop rate,imposing a hierarchy across the units; similar to both meProp [33] and excitation dropout [44]. Rippelet al. [29] demonstrate the effect using an autoencoder where nested dropout is applied to the code;the result is a model where one can trade off reconstruction accuracy with compute by droppinglower priority elements of the code. Standout [2] is another similar variant of dropout; in standout,

4

the activation value of a unit determines the drop rate, where high activations values lead to a higherkeep probability and vice versa.

The Lottery Ticket Hypothesis of Frankle and Carbin [6] demonstrates the existence of a subnetworkthat – in isolation, with the rest of the network pruned away – both dictates the function found bygradient descent, and can be trained to the same level of task performance with, or without, theremaining network. In our notation, a prediction of this “winning lottery ticket” isW(θ); and theeffectiveness of our method suggests that one can reduce the size of the winning lottery ticket byregularising the network.

5 Experiments

Our experiments were performed using the original ResNet [15], Wide ResNet [43], and Transformer[37] architectures; applied to the CIAFR-10 [20], ImageNet [30], and WMT English-German Trans-lation datasets. For each baseline experiment we verify our networks reach the reported accuracyon the appropriate test set; we report the test accuracy at differing prune percentages and comparedifferent regularisation strategies. In addition, we compare our targeted dropout to standard dropoutwhere the expected number of dropped weights is matched between the two techniques (i.e. the droprate of standard dropout runs is set to γ · α, the proportion of weights to target times the dropout rate).

For our pruning procedure, we perform the greedy layer-wise magnitude-base pruning described inSection 2.3 to all weight matrices except those leading to the logits. In our experiments we comparetargeted dropout against the following competitive schemes:

L1 Regularization [12]: Complexity cost θ = ‖θ‖1 is added to the cost function. The hope beingthat this term would drive unimportant weights to zero. In our table we denote this loss by L1

β whereβ is the cost-balancing coefficient applied to the complexity term.

L0 Regularization [24]: Louizos et al. apply an augmentation of Concrete Dropout [9], called Hard-Concrete Dropout, to the parameters of a neural network. The mask applied to the weights follows aHard-Concrete distribution where each weight is associated with a gating parameter that determinesthe drop rate. The use of the Concrete distribution allows for a differentiable approximation to the L0

cost, so we may directly minimise it alongside our task objective. When sparisfying these networksto a desired sparsity rate, we prune according to the learned keep probabilities (σ(log(α)) from [24]),dropping those weights with lowest keep probabilities first.

Variational Dropout [19, 25]: Similar to the technique used for L0 regularisation, Molchanov et al.[25] apply Gaussian dropout with trainable drop rates to the weights of the network and interprets themodel as a variational posterior with a particular prior. The authors note that the variational lowerbound used in training favors higher drop probabilities and experimentally confirm that networkstrained in this way do indeed sparsify.

Smallify [21]: Leclerc et al. use trainable gates on weights/units and regularise gates towards zerousing L1 regularisation. Crucial to the technique is the online pruning condition: Smallify keeps amoving variance of the sign of the gates, and a weight/unit’s associated gate is set to zero (effectivelypruning that weight/unit) when this variance exceeds a certain threshold. This technique has beenshown to be extremely effective at reaching high prune rates on VGG networks [31].

Specifically, we compare the following techniques:

dropoutα

: Standard weight or unit dropout applied at a rate of α.

targetedα,γ

: Targeted dropout (the weight variant in ‘a)’ tables, and unit variant in ‘b)’ tables) applied

to the γ · 100% lowest magnitude weights at a rate of α.

variational: Variational dropout [19, 25] applied with a cost coefficient of 0.01/50, 000.

L0β : L0 regularisation [24] applied with a cost coefficient of β/50, 000.

L1β : L1 regularisation [12] applied with a cost coefficient of β.

smallifyλ

: Smallify SwitchLayers [21] applied with a cost coefficient of λ, exponential moving

average decay of 0.9, and a variance threshold of 0.5.

5

Figure 1: A comparison between a network without dropout (left) and with targeted dropout (right)of the matrix formed by θ> H θ. The weights are ordered such that the last 75% are the weightswith the lowest magnitude (those we intend to prune). The sum of the elements of the lower righthand corner approximates the change in error after pruning (Eqn. (3)). Note the stark differencebetween the two networks, with targeted dropout concentrating its dependence on the top left corner,leading to a much smaller error change after pruning (given in Table 1).

REGULARISATION |∆E| UNPRUNED ACCURACY PRUNED ACCURACY

NONE 0.120698 38.11% 26.13%TARGETED DROPOUT 0.0145907 40.09% 40.14%

Table 1: Comparison of the change in loss (|∆E| of Equation (3)) for dense networks.

5.1 Analysing the Important Subnetwork

In order to analyze the effects of targeted dropout we construct a toy experiment with small densenetworks to analyse properties of the network’s dependence on its weights. The model we consideris a single hidden layer densely connected network with ten units and ReLU activations [27]. Wetrain two of these networks on CIFAR-10; the first unregularised, and the second with targeteddropout applied to the γ = 75% lowest-magnitude weights at a rate of α = 50%. The networksare both trained for 200 epochs at a learning rate of 0.001 using stochastic gradient descent withoutmomentum.

We then compute the gradient and Hessian over the test set in order to estimate the change inerror from Equation 3 (see Table 1). In addition, we compute the Hessian-weight product matrixformed by typical element [θ> H θ]ij = θiHijθj as an estimate of weight correlations andnetwork dependence (see Figure 1). This matrix is an important visualisation tool since summing theentries associated with weights you intend to delete corresponds to computing the second term inEquation (1) – this becomes the dominant term towards the end of training, at which time the gradientis approximately zero.

Figure 1 makes clear the dramatic effects of targeted dropout regularisation on the network. Inthe Figure, we reorder the rows and columns of the matrices so that the first 25% of the matrixrows/columns correspond to the 25% of weights we identify as the important subnetwork (i.e. highestmagnitude weights), and the latter 75% are the weights in the unimportant subnetwork (i.e. lowestmagnitude weights). The network trained with targeted dropout relies nearly exclusively on the 25%of weights with the largest magnitude at the end of training. Whereas, the network trained withoutregularisation relies on a much larger portion of the weights and has numerous dependencies in theparameters marked for pruning.

5.2 ResNet

We test the performance of targeted dropout on Residual Networks (ResNets) [14] applied to theCIFAR-10 dataset, to which we apply basic input augmentation in the form of random crops, randomhorizontal flipping, and standardisation. This architectural structure has become ubiquitous incomputer vision, and is gaining popularity in the domains of language [18], and audio [36]. Our

6

Weight Dropout/Pruning

prun

epe

rcen

tage

none dropoutα=0.25

targetedα=0.5,γ=0.5

targetedα=0.33,γ=0.75

targetedα=0.66,γ=0.75

targetedα=0.75,γ=0.90

variational L10.1 L0

0.1

0 % 93.71 93.62 93.03 89.88 92.64 92.53 92.09 92.80 88.8310% 93.72 93.63 93.04 89.80 92.62 92.55 92.00 92.72 90.6620% 93.77 93.66 93.02 89.93 92.63 92.48 92.02 92.84 88.6430% 93.59 93.58 92.98 89.89 92.66 92.53 92.07 92.63 87.1640% 93.09 93.45 93.03 89.75 92.70 92.63 92.12 92.80 85.3150% 92.20 93.07 92.99 89.72 92.65 92.54 91.84 92.29 80.9460% 90.46 90.81 92.66 89.84 92.70 92.55 91.48 91.20 69.4870% 81.88 72.29 92.22 89.80 92.66 92.56 90.23 86.30 46.1980% 32.02 19.84 84.03 85.80 91.86 92.54 83.44 63.00 23.7190% 14.63 10.05 28.27 27.04 67.58 92.48 15.16 21.08 12.55

Unit Dropout/Pruning

prun

epe

rcen

tage

none dropoutα=0.25

targetedα=0.5,γ=0.5

targetedα=0.33,γ=0.75

targetedα=0.66,γ=0.75

targetedα=0.90,γ=0.75

variational L10.01 L0

0.01

0 % 93.69 92.43 92.21 90.46 89.38 89.78 93.14 93.31 93.3510% 90.05 67.52 91.96 88.44 89.48 90.18 92.91 91.03 83.0120% 80.34 25.05 91.63 83.55 88.89 89.79 90.38 85.63 54.5930% 59.94 13.47 91.30 69.82 88.84 89.88 86.38 72.19 21.3440% 35.40 10.02 89.89 54.42 87.54 89.98 83.59 46.41 10.8250% 12.63 9.97 88.41 28.88 84.86 90.05 65.79 26.72 15.0460% 10.65 9.99 26.55 18.55 81.98 90.08 41.05 12.11 9.4670% 11.70 10.01 17.41 17.84 75.47 90.03 19.36 11.81 10.0280% 9.99 9.95 10.63 10.87 28.99 34.18 9.56 14.73 14.8890% 9.85 9.98 9.30 10.29 9.97 10.04 10.41 10.22 9.98

Table 2: ResNet-32 model accuracies on CIFAR-10 at differing pruning percentages and underdifferent regularisation schemes. The top table depicts results using the weight pruning strategy,while the bottom table depicts the results of unit pruning (see Sec. 2.3)

baseline model reaches over 93% final accuracy after 256 epochs, which matches previously reportedresults for ResNet-32 [14].

Our weight pruning experiments demonstrate that standard dropout schemes are comparatively weakcompared to their targeted counterparts; standard dropout performs worse than our no-regularisationbaseline. We find that a higher targeted dropout rate applied to a larger portion of the weights resultsin the network matching unregularised performance with only 40% of the parameters.

Variational dropout seems to improve things marginally over the unregularised baseline in both weightand unit pruning scenarios, but was still outperformed by targeted dropout. L0 regularisation wasfairly insensitive to its complexity term coefficient; we searched over a range of β ∈ [10−6, 101] andfound that values above 10−1 failed to converge, while values beneath 10−4 tended to show no signsof regularisation. Similarly to variational dropout, L0 regularisation does not prescribe a methodfor achieving a specific prune percentage in a network, and so, an extensive hyperparameter searchbecomes a requirement in order to find values that result in the desired sparsity. As a compromise,we search over the range mentioned above and select the setting most competitive with targeteddropout; next, we applied magnitude-based pruning to the estimates provided in Equation 13 ofLouizos et al. [24]. Unfortunately, L0 regularisation seems to force the model away from conformingto our assumption of importance being described by parameter magnitude.

In Table 3 we present the results of pruning ResNet-102 trained on ImageNet. We observe similarbehaviour to ResNet applied to CIFAR-10, although it’s clear that the task utilises much more of thenetwork’s capacity, rendering it far more sensitive to pruning relative to CIFAR-10.

5.3 Wide ResNet

In order to ensure fair comparison against the L0 regularisation baseline, we adapt the authors owncodebase2 to support targeted dropout, and compare the network’s robustness to sparsification underthe provided L0 implementation and targeted dropout. In Table 4 we observe that L0 regularisationfails to truly sparsify the network, but has a strong regularising effect on the accuracy of the network(confirming the claims of Louizos et al.). This further verifies the observations made above, showingthat L0 regularisation fails to sparsify the ResNet architecture.

2the original L0 PyTorch code can be found at: github.com/AMLab-Amsterdam/L0_regularization

7

WeightDropout/Pruning

prun

epe

rcen

tage

none targetedα=0.5,γ=0.5

L110−5

0 % 75.9 75.7 70.610% 75.9 75.7 70.420% 74.9 75.3 69.830% 71.9 74.4 65.740% 64.4 73.5 62.150% 45.0 68.8 53.460% 8.6 50.5 38.370% 0.7 14.8 17.680% 0.2 0.4 1.490% 0.1 0.1 0.4

UnitDropout/Pruning

prun

epe

rcen

tage

none targetedα=0.5,γ=0.5

L10.001

0 % 75.7 74.3 75.710% 34.5 67.2 66.620% 1.8 59.4 12.630% 0.4 33.0 0.440% 0.1 6.4 0.250% 0.1 0.6 0.160% 0.1 0.2 0.170% 0.1 0.1 0.180% 0.1 0.1 0.190% 0.1 0.1 0.1

Table 3: ResNet-102 model accuracies on ImageNet. Accuracies are top-1, single crop on 224 by224 pixel images.

Unit Dropout/Pruning

prun

epe

rcen

tage

none targetedα=0.33,γ=0.75

L010−6

0 % 92.21 92.24 94.1510% 89.76 92.09 88.0520% 82.37 91.55 65.0330% 52.20 90.09 13.3440% 18.48 87.47 10.0150% 10.53 82.09 10.0060% 10.04 69.58 10.0070% 10.00 44.05 10.0080% 10.00 16.94 10.0090% 10.00 10.43 10.00

Table 4: Wide ResNet [43] model classification accuracy on CIFAR-10 test set at differing prunepercentages.

5.4 Transformer

Weight Dropout/Pruning

prun

epe

rcen

tage

none targetedα=0.66,γ=0.75

targetedα=0.66,γ=0.90

0 % 26.01 26.52 25.3210% 26.05 26.44 25.3220% 25.90 26.48 25.1930% 25.91 26.30 25.2740% 25.81 26.20 24.9750% 25.08 26.03 24.9360% 23.31 25.62 24.2770% 8.89 24.07 22.4180% 0.24 12.39 10.5790% 0.01 0.07 0.64

(a) Transformer model uncased BLEU score.

Weight Dropout/Pruning

prun

epe

rcen

tage

none targetedα=0.66,γ=0.75

targetedα=0.66,γ=0.90

0 % 62.29 58.31 57.4110% 62.54 59.00 58.1020% 62.21 59.39 58.5230% 62.33 58.66 57.8640% 61.81 59.39 58.6750% 60.82 57.71 57.0860% 58.13 58.42 57.9670% 48.40 55.39 54.8580% 25.80 47.09 46.6390% 6.90 21.64 27.02

(b) Transformer model per-token accuracy.

Table 5: Evaluation of the Transformer Network under varying sparsity rates on the WMT new-stest2014 EN-DE test set.

The Transformer network architecture [37] represents the state-of-the-art on a variety of NLP tasks.In order to evaluate the general applicability of our method we measure the Transformer’s robustnessto weight-level pruning without regularisation, and compare this against two settings of targeteddropout applied to the network.

The Transformer architecture consists of stacked multi-head attention layers and feed-forward(densely connected) layers, both of which we target for sparsification; within the multihead attention

8

layers, each head of each input has a unique linear transformation applied to it, which are the weightmatrices we target for sparsification.

Table 5a details the results of pruning the Transformer architecture applied to the WMT newstest2014English-German (EN-DE). Free of any regularisation, the Transformer seems to be fairly robust topruning, but with targeted dropout we are able to increase the BLEU score by 15 at 70% sparsity, and12 at 80% sparsity; further confirming target dropout’s applicability to a range of architectures anddatasets.

5.5 Scheduling the Targeting Proportion

Weight Dropout/Pruning

prun

epe

rcen

tage

targetedα=0.66,γ=0.75

smallifyλ=0.00001

smallifyλ=0.00001

ramp targα=0.99,γ=0.99

ramp targα=0.99,γ=0.99

0 % 92.64 90.16 90% 90.20 89.03 98.5% 89.0310% 92.62 90.13 91% 90.33 89.16 98.6% 89.0820% 92.63 90.16 92% 90.30 89.14 98.7% 89.0030% 92.66 90.06 93% 90.27 89.03 98.8% 89.0540% 92.70 90.17 94% 89.46 89.05 98.9% 88.9950% 92.65 90.20 95% 89.41 89.05 99.0% 89.1060% 92.70 90.12 96% 88.55 89.02 99.1% 88.3570% 92.66 90.10 97% 86.35 89.05 99.2% 79.8880% 91.86 90.15 98% 59.27 89.05 99.3% 77.3590% 67.58 90.16 99% 13.83 88.97 99.4% 16.55

Unit Dropout/Pruning

prun

epe

rcen

tage

targetedα=0.66,γ=0.75

smallifyλ=0.0001

ramp targα=0.99,γ=0.90

0 % 90.55 90.20 85.9810% 90.83 90.33 86.1220% 89.88 90.30 86.0130% 87.35 90.27 86.1040% 85.39 89.46 85.9850% 80.84 89.41 86.1360% 71.97 88.55 86.0270% 55.98 86.35 86.0880% 10.02 59.27 85.9590% 10.07 13.83 85.99

Table 6: Comparing Smallify to targeted dropout and ramping targeted dropout. Experiments onCIFAR10 using ResNet32.

Upon evaluation of weight-level Smallify [21] we found that, with tuning, we were able to out-perform targeted dropout at very high pruning percentages (see Table 6). One might expect that asparsification scheme like Smallify – which allows for differing prune rates between layers – would bemore flexible and better suited to finding optimal pruning masks; however, we show that a variant oftargeted dropout we call ramping targeted dropout is capable of similar high rate pruning. Moreover,ramping targeted dropout preserves the primary benefit of targeted dropout: fine control over sparsityrates.

Ramping targeted dropout simply anneals the targeting rate γ from zero, to the specified final γthroughout the course of training. For our ResNet experiments, we anneal from zero to 95% of γover the first forty-nine epochs, and then from 95% of γ to 100% of γ over the subsequent forty-nine.In a similar fashion, we ramp α from 0% to 100% linearly over the first ninety-eight steps.

Using ramping targeted dropout we are able to achieve sparsity of 99% in a ResNet32 with accuracy87.03% on the CIFAR-10 datatset; while the best Smallify run achieved intrinsic sparsity of 98.8% atconvergence with accuracy 88.13%, when we perform pruning to enforce equal pruning rates in allweight matrices, the network degrades rapidly (see Table 6).

5.6 Fixed Filter Sparsity

We also propose a variation of Ramping targeted dropout (Section 5.5), where each layer is assigneda γ such that only a fixed number of weights are non-zero by the end of training (for example, three

9

parameter per filter). We refer to this as Xtreme dropout. ResNet32 when trained with Xtreme-3 (3weights per filter are non-zero) was able to achieve an accuracy of 84.7% on the CIFAR-10 datatset ata sparsity level of 99.6% while Xtreme-4 was able to achieve 87.06% accuracy at a sparsity level of99.47%. An interesting observation of Xtreme pruning is that when trained on ResNet18, it achieves82% accuracy at a sparsity level of 99.8%. When translated to the number of parameters, it has only29,760 non-zero parameters (includes BatchNorm) which is less than the number of parameters in anetwork consisting of a single dense layer with 10 output units.

6 Exploring Recent Discussions and Concerns

A line of work [4, 23] has suggested that post hoc pruning with fine-tuning is not as effective as itcould be. They propose using the sparsity patterns derived from a pruned model to define a smallernetwork (where the remaining, unpruned weights are reinitialised randomly) which is then trainedfrom scratch, yielding better final task performance than fine-tuning the pruned model’s weights.

A similar question arose in our own work as we pondered how early in the training procedurethe important subnetwork could be decided. In the ideal case, the important subnetwork would bearbitrary and we could blindly select any subnetwork at the beginning of training, delete the remainingnetwork, and recover similar accuracy to a much more complicated pruning strategy. While in theworst case, the important subnetwork would be predestined, and would remain difficult to identifyuntil the very end of training.

While Crowley et al. [4], Liu et al. [23] rely on sparsity patterns derived from pruned models, inthis paper we are concerned with pruning schemes that achieve sparsity in a single execution of thetraining procedure; and so, in order to evaluate the more general claim that training smaller networksfrom scratch can match (or even out-perform) pruning, we compare the following two methods:

• Random-pruning: Before training, prune away a random subnetwork.

• Targeted Dropout (Ramping TD): Apply ramping targeted dropout throughout the courseof training.

Prune %Type 50% 75% 90% 99%

Random-prune 92.58 92.32 90.66 80.86Ramping TD 93.29 92.72 92.51 88.80

(a) Comparison of weight-level pruning methods us-ing ResNet-32 trained on CIFAR-10.

Prune %Type 75% 85% 90% 95%

Random-prune 90.50 88.52 84.98 79.09Ramping TD 90.84 88.59 86.45 80.65

(b) Comparison of unit-level pruning methods usingResNet-32 trained on CIFAR-10.

Prune %Type 75% 85% 90% 95%

Random-prune 48.98 (0.62) 45.58 (1.25) 40.50 (2.03) 31.44 (1.64)Ramping TD 52.64 (0.61) 49.20 (0.10) 45.03 (0.83) 30.15 (1.72)

(c) Comparison of unit-level pruning methods using VGG-16 trained on CIFAR-100. Results are the average offive independent training runs followed by one standard deviation reported in brackets.

Table 7: Comparison between random pruning at the beginning of training and regularising withtargeted dropout throughout the course of training, followed by post hoc pruning.

The results of our experiment are displayed in Table 7. It is clear that – although Crowley et al.[4], Liu et al. [23]’s results show that knowing a good sparsity pattern in advance allows you toachieve competitive results with pruning – simply training a smaller subnetwork chosen at randomdoes not compete with a strong regularisation scheme used over the course of training. Similarobservations that contradict the conclusions of Crowley et al. [4], Liu et al. [23] have been made inboth Frankle et al. [7] and Gale et al. [10].

10

7 Conclusion

We propose targeted dropout as a simple and effective regularisation tool for training neural networksthat are robust to post hoc pruning. Among the primary benefits of targeted dropout are the simplicityof implementation, intuitive hyperparameters, and fine-grained control over sparsity - both duringtraining and inference. Targeted dropout performs well across a range of network architectures andtasks, demonstrating is broad applicability. Importantly, like [29], we show how dropout can be usedas a tool to encode prior structural assumptions into neural networks. This perspective opens the doorfor many interesting applications and extensions.

Acknowledgements

We would like to thank Christos Louizos for his extensive review of our codebase and help withdebugging our L0 implementation, his feedback was immensely valuable to this work. Our thanksgoes to Nick Frosst, Jimmy Ba, and Mohammad Norouzi who provided valuable feedback andsupport throughout. We would also like to thank Guillaume Leclerc for his assistance in verifyingour implementation of Smallify.

11

References[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro,

Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser,Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray,Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar,Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, PeteWarden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2015. URL http://download.tensorflow.org/paper/whitepaper2015.pdf.

[2] Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In Advancesin Neural Information Processing Systems, pages 3084–3092, 2013.

[3] Mohammad Babaeizadeh, Paris Smaragdis, and Roy H Campbell. Noiseout: A simple way toprune neural networks. arXiv preprint arXiv:1611.06211, 2016.

[4] Elliot J Crowley, Jack Turner, Amos Storkey, and Michael O’Boyle. Pruning neural networks:is it time to nip it in the bud? arXiv preprint arXiv:1810.04622, 2018.

[5] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pages4860–4874, 2017.

[6] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neuralnetworks. arXiv preprint arXiv:1803.03635, 2018.

[7] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The lotteryticket hypothesis at scale, 2019.

[8] Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.

[9] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural InformationProcessing Systems, pages 3584–3593, 2017.

[10] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.

[11] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net-works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149,2015.

[12] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connectionsfor efficient neural network. In Advances in neural information processing systems, pages1135–1143, 2015.

[13] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brainsurgeon. In Advances in neural information processing systems, pages 164–171, 1993.

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for imagerecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,pages 770–778, 2016.

[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for imagerecognition. In CVPR, 2016.

[16] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut-dinov. Improving neural networks by preventing co-adaptation of feature detectors. arXivpreprint arXiv:1207.0580, 2012.

[17] Qiangui Huang, Kevin Zhou, Suya You, and Ulrich Neumann. Learning to prune filters inconvolutional neural networks. arXiv preprint arXiv:1801.07365, 2018.

[18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Ko-ray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099,2016.

12

[19] Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the localreparameterization trick. In Advances in Neural Information Processing Systems, pages 2575–2583, 2015.

[20] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.2009.

[21] Guillaume Leclerc, Manasi Vartak, Raul Castro Fernandez, Tim Kraska, and Samuel Madden.Smallify: Learning network size while training. arXiv preprint arXiv:1806.03723, 2018.

[22] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neuralinformation processing systems, pages 598–605, 1990.

[23] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the valueof network pruning. arXiv preprint arXiv:1810.05270, 2018.

[24] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networksthrough l_0 regularization. arXiv preprint arXiv:1712.01312, 2017.

[25] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deepneural networks. arXiv preprint arXiv:1701.05369, 2017.

[26] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutionalneural networks for resource efficient inference. 2016.

[27] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines.In Proceedings of the 27th international conference on machine learning (ICML-10), pages807–814, 2010.

[28] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation inpytorch. 2017.

[29] Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nesteddropout. In International Conference on Machine Learning, pages 1746–1754, 2014.

[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, ZhihengHuang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision(IJCV), 115(3):211–252, 2015. doi: 10.1007/s11263-015-0816-y.

[31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scaleimage recognition. arXiv preprint arXiv:1409.1556, 2014.

[32] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.Dropout: A simple way to prevent neural networks from overfitting. The Journal of MachineLearning Research, 15(1):1929–1958, 2014.

[33] X. Sun, X. Ren, S. Ma, and H. Wang. meProp: Sparsified Back Propagation for AcceleratedDeep Learning with Reduced Overfitting. ArXiv e-prints, June 2017.

[34] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze predictionwith dense networks and fisher pruning. arXiv preprint arXiv:1801.05787, 2018.

[35] Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural networkcompression. arXiv preprint arXiv:1702.04008, 2017.

[36] Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, AlexGraves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generativemodel for raw audio. arXiv preprint arXiv:1609.03499, 2016.

[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762,2017.

13

[38] Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, StephanGouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. Tensor2tensor forneural machine translation. Vol. 1: MT Researchers’ Track, page 193, 2018.

[39] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization ofneural networks using dropconnect. In International Conference on Machine Learning, pages1058–1066, 2013.

[40] Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured probabilistic pruning forconvolutional neural network acceleration. arXiv preprint arXiv:1709.06994, 2017.

[41] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity indeep neural networks. In Advances in Neural Information Processing Systems, pages 2074–2082,2016.

[42] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables.Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49–67, 2006.

[43] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprintarXiv:1605.07146, 2016.

[44] A. Zunino, S. Adel Bargal, P. Morerio, J. Zhang, S. Sclaroff, and V. Murino. Excitation Dropout:Encouraging Plasticity in Deep Neural Networks. ArXiv e-prints, May 2018.

14