AlexNet: ImageNet Win, Two-GPU Training & Architecture

Two gaming cards instead of a cluster

Krizhevsky, Sutskever and Hinton, all at the University of Toronto, trained the network on two NVIDIA GTX 580 cards with 3GB of memory each, NVIDIA's high-end gaming card released in November 2010, not datacenter hardware. [3][5]

The 60-million-parameter network was too large for one card's memory, so the authors split it across both GPUs, which communicated only at certain layers to keep the exchange small relative to the computation. That split alone cut the top-1 and top-5 error rates by 1.7 and 1.2 points versus a single-GPU network half the size. [3]

Two tricks that made a bigger network trainable

The network used rectified linear units, f(x) = max(0, x), instead of the tanh neurons then standard. In the authors' own CIFAR-10 test, a network with ReLUs reached 25% training error six times faster than the same network with tanh units. [3]

In the first two fully-connected layers, dropout zeroed each hidden unit's output with probability 0.5 on every pass, so the network could not rely on any one unit. That roughly doubled the iterations needed to converge, but without it the network overfit substantially. [3]

Atlas interpretation: Neither trick originated here; both came from earlier Toronto work, including the layer-by-layer unsupervised pretraining that had made deep networks trainable six years earlier. This paper's move was to skip that unsupervised stage and train all eight layers by ordinary supervised backpropagation, now fast enough thanks to ReLUs, dropout and two GPUs. The paper also reports that removing any one convolutional layer cost about 2 points of top-1 accuracy, its evidence that depth itself, not just scale, was doing the work. [3]

The number everyone quotes is an ensemble's score

The win came two months before the paper. ILSVRC2012 closed submissions on September 30, 2012; the organizers presented the results at the PASCAL workshop at ECCV in Florence on October 12 and released them in full the next day, with the Toronto team, entered as SuperVision, at 0.153 top-5 error against runner-up ISI's 0.262. The paper explaining the network came out at NIPS that December. [4][1][3]

The 15.3% top-5 test error that beat the runner-up's 26.2% was not one network's score. It came from averaging seven CNNs, two of which were first trained on the full 15-million-image ImageNet Fall 2011 release before fine-tuning. The single eight-layer network the paper otherwise describes scored 18.2% top-5 on the validation set. [3]

Atlas interpretation: The summary's "almost in half" holds for the contest-winning score: 15.3 is roughly 58% of 26.2. But retrospectives that credit that number to "AlexNet," meaning the single architecture the paper spends most of its pages on, are describing a seven-model ensemble with extra pretraining, not that network alone. [3]

Ten years to a Test of Time award

NeurIPS gave the paper its Test of Time award in 2022, describing it as the first CNN to win the ImageNet challenge and crediting it with lasting impact on the field's methods. [6]

Atlas interpretation: That decade of hindsight is the honest basis for calling this era-defining. None of it was available to the researchers who read the 2012 scoreboard and, per the summary above, changed what they worked on that same year. [6]