What’s in a name? The ‘deep learning’ debate


Monday’s historic debate between machine learning luminary Yoshua Bengio and machine learning critic Gary Marcus spilled over into a tit for tat between the two in the days following, mostly about the status of the term “deep learning.”

The history of the term deep learning shows that the use of it has been opportunistic at times but has had little to do in the way of advancing the science of artificial intelligence. Hence, the current debate will likely not go anywhere, ultimately. 

Monday night’s debate found Bengio and Marcus talking about similar-seeming end goals, things such as the need for “hybrid” models of intelligence, maybe combining neural networks with something like a “symbol” class of object. The details were where the two argued about definitions and terminology. 

In the days that followed, Marcus, in a post on Medium, observed that Bengio seemed to have white-washed his own recent critique of shortcomings in deep learning. And Bengio replied, in a letter on Google Docs linked from his Facebook account, that Marcus was presuming to tell the deep learning community how it can define its terms. Marcus responded in a follow-up post by suggesting the shifting descriptions of deep learning are “sloppy.” Bengio replied again late Friday on his Facebook page with a definition of deep learning as a goal, stating, “Deep learning is inspired by neural networks of the brain to build learning machines which discover rich and useful internal representations, computed as a composition of learned features and functions.” Bengio noted the definition did not cover the “how” of the matter, leaving it open. 

Also: Devil’s in the details in Historic AI debate

The term “deep learning” has emerged a bunch of times over the decades, and it has been used in different ways. It’s never been rigorous, and doubtless it will morph again, and at some point it may lose its utility. 

Jürgen Schmidhuber, who co-developed the “long-short term memory” form of neural network, has written that the AI scientist Rina Dechter first used the term “deep learning” in the 1980s. That use was different from today’s usage.  Dechter was writing about methods to search a graph of a problem, having nothing much to do with deep networks of artificial neurons. But there was a similarity: she was using the word “deep” as a way to indicate the degree of complexity of a problem and its solution, which is what others started doing in the new century.

The same kind of heuristic use of deep learning started to happen with Bengio and others around 2006, when Geoffrey Hinton offered up seminal work on neural networks with many more layers of computation than in past. Starting that year, Hinton and others in the  field began to refer to “deep networks” as opposed to earlier work that employed collections of just a small number of artificial neurons.



From Yoshua Bengio’s slides for the AI debate with Gary Marcus, December 23rd.

Yoshua Bengio

So deep learning emerged as a very rough, very broad way to distinguish a layering approach that makes things such as AlexNet work. 

In the meantime, as Marcus suggests, the term deep learning has been so successful in the popular literature that it has taken on a branding aspect, and it has become a kind-of catchall that can sometimes seem like it stands for anything. Marcus’s best work has been in pointing out how cavalierly and irresponsibly such terms are used (mostly by journalists and corporations), causing confusion among the public. Companies with “deep” in their name have certainly branded their achievements and earned hundreds of millions for it. So the topic of branding is in some sense unavoidable.

Bengio’s response implies he doesn’t much care about the semantic drift that the term has undergone because he’s focused on practicing science, not on defining terms. To him, deep learning is serviceable as a placeholder for a community of approaches and practices that evolve together over time. 

Also: Intel’s neuro guru slams deep learning: ‘it’s not actually learning’

Probably, deep learning as a term will at some point disappear from the scene, just as it and other terms have floated in and out of use over time. 

There was something else in Monday’s debate, actually, that was far more provocative than the branding issue, and it was Bengio’s insistence that everything in deep learning is united in some respect via the notion of optimization, typically optimization of an objective function. That could be a loss function, or an energy function, or something else, depending on the context. 

In fact, Bengio and colleagues have argued in a recent paper that the notion of objective functions should be extended to neuroscience. As they put it, “If things don’t ‘get better’ according to some metric, how can we refer to any phenotypic plasticity as ‘learning’ as opposed to just ‘changes’?”

That’s such a basic idea, it seems so self-evident, that it almost seems trivial for Bengio to insist on it. 

But it is not trivial. Insisting that a system optimizes along some vector is a position that not everyone agrees with. For example, Mike Davies, head of Intel’s “neuromorphic” chip effort, this past February criticized back-propagation, the main learning rule used to optimize in deep learning, during a talk at the International Solid State Circuits Conference.

Davies’s complaint is that back-prop is unlike human brain activity, arguing “it’s really an optimization procedure, it’s not actually learning.” 

Thus, deep learning’s adherents have at least one main  tenet that is very broad but also not without controversy.

The moral of the story is, there will always be something to argue about. 

About the author

E-Crypto News was developed to assist all cryptocurrency investors in developing profitable cryptocurrency portfolios through the provision of timely and much-needed information. Investments in cryptocurrency require a level of detail, sensitivity, and accuracy that isn’t required in any other market and as such, we’ve developed our databases to help fill in information gaps.

Related Posts

E-Crypto News Executive Interviews

Crypto Scams

Cryptosoft Trading Bot Review
June 27, 2022
The Largest Crypto Scams Of 2022 (So Far)
The Largest Crypto Scams Of 2022 (So Far)
June 14, 2022
How Do Scammers Entice Their Prey?
May 10, 2022
Beanstalk Farms Loses $80M In A Massive DeFi Governance Flash-Loan Hack
Beanstalk Farms Loses $80M In A Massive DeFi Governance Flash-Loan Hack
April 23, 2022
Joon Pak Head of Crypto at Prove talks to Us about Crypto Fraud And More
April 11, 2022

Automated trading with HaasBot Crypto Trading Bots

Blockchain/Cryptocurrency Questions and Answers

Is The Crypto Market Combating A Lehman Brothers Moment?
Is The Crypto Market Combating A Lehman Brothers Moment?
June 30, 2022
Roundtable Interview-What is the Effect of The Russia-Ukraine War on Cryptocurrency Prices?
March 4, 2022
How Does Bitcoin Casino Work + 2021 Beginner’s Guide
November 8, 2021
How to Buy and Sell Cryptocurrency
November 8, 2021
What Are Bitcoin Futures And How Will They Work In 2022?
November 4, 2021

CryptoCurrencyUSDChange 1hChange 24hChange 7d
Bitcoin19,091 0.94 % 0.88 % 11.31 %
Ethereum1,061.4 1.48 % 1.31 % 14.70 %
Tether1.002 0.02 % 0.06 % 0.14 %
USD Coin1.003 0.05 % 0.13 % 0.01 %
BNB217.96 0.40 % 0.68 % 9.15 %
Binance USD1.002 0.50 % 0.08 % 0.00 %
Cardano0.4486 1.23 % 0.31 % 9.96 %
XRP0.3131 0.99 % 0.43 % 14.77 %
Solana32.71 1.97 % 0.49 % 22.84 %
Dogecoin0.06620 1.30 % 1.04 % 3.45 %

Bitcoin (BTC) $ 19,114.90
Ethereum (ETH) $ 1,065.43
Tether (USDT) $ 1.00
USD Coin (USDC) $ 1.00
BNB (BNB) $ 218.40
Binance USD (BUSD) $ 1.00
Cardano (ADA) $ 0.449189
XRP (XRP) $ 0.313808
Solana (SOL) $ 32.91
Dogecoin (DOGE) $ 0.066484