University of Toronto in AI: Hinton, Deep Belief Nets & AlexNet

A lab that kept working on neural networks

University of Toronto was founded in 1827. Geoffrey Hinton joined its faculty in 1987, left for University College London between 1998 and 2001, and returned, remaining affiliated ever since as a professor and later professor emeritus. For most of that period neural networks were a minority approach in machine learning research, and Hinton's lab was one of a small number that kept building on them through the period when the field's attention was elsewhere. [2]

In 2018 Hinton shared the ACM A.M. Turing Award with Yoshua Bengio and Yann LeCun for conceptual and engineering work that made deep neural networks central to computing. In 2017 he was also a founding scientific figure behind the Vector Institute, an independent, not-for-profit AI research institute launched with $135 million in funding commitments from Canadian governments and more than forty industry partners, built on leadership from University of Toronto faculty and working closely with the university without being part of it. [2][3]

Making deep networks trainable again

The 2006 deep belief nets paper, by Hinton with Simon Osindero and Yee-Whye Teh, trained a deep network one layer at a time and got an untuned version of it to beat a plain backpropagation network and match the tuned methods researchers were using at the time on handwritten digit recognition. It mattered less for the specific technique, layer-wise pretraining, than for the demonstration that deep networks could be trained reliably at all, at a moment when much of the field had concluded they could not. [4]

The paper that changed what people worked on

Six years later, Alex Krizhevsky, Ilya Sutskever and Hinton entered AlexNet in the 2012 ImageNet competition, a convolutional network trained on two consumer gaming GPUs that cut the competition's error rate by nearly half against the next-best entry. Krizhevsky and Sutskever were Hinton's doctoral students at the University of Toronto. The margin was large enough that researchers who saw that year's results changed what they worked on, and deep learning stopped being treated as a fringe position in computer vision. [5][2]

The three founded DNNresearch Inc. in 2012 around the AlexNet work, and Google acquired it in 2013 for a price reported at $44 million. That commercial event followed directly from a university lab result, and it is one instance of a pattern the university's other event on the timeline also shows: a small group publishing methods years before the compute and data existed to make the full case for them. [2]