Rendered at 11:30:48 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jcattle 4 hours ago [-]
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.
And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.
When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.
But still, 30 years later here we are, standing on top of the shoulders of this useless research.
yorwba 3 hours ago [-]
Like half of what Schmidhuber is always complaining about is that (except for LSTMs) people aren't standing on the shoulders of his research very much. They try to solve some of the same problems people have always wanted to solve, try some of the same approaches people always tend to try, and then tinker until it works. At no point do they consult Schmidhuber's decade-old papers where he tried something kind of similar but didn't get very impressive results, and hence they also do not think to cite him. Then he comes out of the woodwork to assert priority.
suddenlybananas 2 hours ago [-]
You can be influenced downstream by papers you haven't personally read.
bonzini 2 hours ago [-]
Shane Legg was in Schmidhuber's lab at IDSIA before being one of the founders of DeepMind, so he probably read the papers personally and knows what influenced him or not...
gillesjacobs 58 minutes ago [-]
Of course, but if you haven't read them you also shouldn't cite them.
And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.
psb217 24 minutes ago [-]
"if you haven't read them you also shouldn't cite them" -- this is wildly incorrect in an academic context. If I'm using ResNets, I should cite the original ResNet paper, even if I haven't read it. If I'm using Transformers, I should cite the original Transformer paper, even if I haven't read it. If my work is a direct extension of method B, and method B is a direct extension of method A, I should cite the source of A, even if I haven't read it.
You can't claim independence from past work simply because you didn't look directly at it. The job of an academic researcher is to know the landscape of relevant ideas, where they come from, where they're going, and to hopefully contribute a few new good ones.
Citation chains should extend back from your work, along a reasonable line conceptual inheritance, back to a reasonable point of origin. Schmidhuber has different definitions for both of these reasonables than the bulk of the ML research community, to a point that makes him difficult to satisfy.
dividedbyzero 31 minutes ago [-]
> Of course, but if you haven't read them you also shouldn't cite them.
But if you build on them you should have read them. I don't know about the specifics and I don't know if Schmidhuber is out of line or not, and citations and impact factors are a terrible mess, but generally speaking, you are responsible for finding and reading and citing any related work that needs to be cited, and if you work on neural networks in an academic context you probably have been forced to read that particular one at some point. Citation obligations don't just disappear because you don't want to do the research.
elorant 50 minutes ago [-]
I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life easier. So while the bedrock is based on academic research and we thank them for that, scaling anything requires a lot of work in uncharted territories.
jhbadger 34 minutes ago [-]
But that isn't the purpose of academia -- the purpose of it is to discover new phenomena not to make products. It is true that there is a lot of work to turn a new advance into a product whether it is software or turning biological knowledge into a drug, but without discovery of new phenomena new products will come to a halt. While it is true that some corporate labs, most famously Bell Labs in its heyday, but also for example IBM's T.J. Watson and Xerox's PARC did do basic research besides product-focused work, this is pretty rare because it is hard to justify the cost of something that may only be practical in decades and often help your competitors as much as yourself.
ACCount37 2 hours ago [-]
Where is "this crowd" that you are talking about?
The closest to that that I've seen is that traditional academia approaches are too far removed from practical applications for highly applied fields like software engineering, or too slow for fast-moving fields like modern day ML (thus, all the preprints).
tcp_handshaker 2 hours ago [-]
I think most of criticism of academia is about the rampant fraud and unreproducible results, due to the way the incentives are structured.
wolfi1 59 minutes ago [-]
and you still need tons of money
MrBuddyCasino 1 hours ago [-]
This is a straw-man if I ever saw one.
Practically no one is against hard science research, properly conducted. The issues are rampant fraud / p-hacking / unreproducible garbage mixed with an unhealthy dose of ideological monoculture and indoctrination, garnished with rising tuition prices while sitting on huge endowments in case of the Ivy Leagues.
eru 3 minutes ago [-]
> Practically no one is against hard science research, properly conducted.
As long as you do that with your own money (or money got freely given from other people), sure.
If you use taxpayer money, that's a different game.
jcattle 50 minutes ago [-]
Yes all good points showing issues that academia has at the moment.
However I often see this going from "there's issues" to discounting academia altogether and positioning private labs as a good or only alternative.
After all, most people in the open science collaboration which published the seminal paper kicking off the replication crisis were from academia.
MrBuddyCasino 29 minutes ago [-]
Yes there is no substitute for academia. Monopolist's research labs get close (Bell Labs etc), but they tend to be more "applied".
It's sad that he is the only one speaking out about Hinton. This whole Hinton glorification seems like it's being pushed by an agenda. I'm not sure if he would receive this much attention if he held a different view (closer to LeCun or Ng), rather than these Effective Altruism takes on current AI.
Hoasi 1 hours ago [-]
Not that surprising since the whole LLM ecosystem is based on plagiarism.
letssaythat 41 minutes ago [-]
"Research made in Ukraine.."
No, the research was made in USSR, however much Scmidhuber likes to think of "occupied Ukraine".
I mean, if one thinks it is his mission to establish the truth..
The truth, Scmidhuber, was never in your fuhrer's hands. Nor it is in the hands of the western fuhrers of today.
Just for the context, today is Russia's Commemoration Day of the victims of the Velikaya Otechestvennaya Voyna (your translations always feel wrong, sorry) of 1941-1945. (Yes, 1941, when the western fascist coalition of 5 million soldiers, invaded Soviet Union.)
26 and more million people of USSR perished, 13 million civilians. Of all nationalities.
snowpid 31 minutes ago [-]
well, so you think, all parts and peoples of USSR were voluntarily part of USSR?
trashburger 14 minutes ago [-]
This article, too, was originally discovered by Jürgen Schmidhuber in 1991!
practal 2 hours ago [-]
TU Munich and Nipkow, Makarius et.al. are also at the center of the influential Isabelle theorem prover. TU Munich is cool :-)
Which work has more value: the abstract description of a catalogue of potential model architectures or their validated application trained on real data?
In the Schmidhuber case their is 20 years and a chain of countless other works in between the two.
jacknews 4 hours ago [-]
Surely the roots, if we skip over the early preceptron work', are in backpropagation and Hinton, and the work going on at Edinburgh and elsewhere in the 80s.
Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was.
No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.
h8hawk 4 hours ago [-]
Hinton did not invent backpropagation.
related paragraph from Wikipedia:
Modern backpropagation was first published by Seppo Linnainmaa as "reverse mode of automatic differentiation" (1970)[26] for discrete connected networks of nested differentiable functions.[27][28][29]
In 1982, Paul Werbos applied backpropagation to MLPs in the way that has become standard.
ogrisel 2 hours ago [-]
Paul Werbos did not apply backprop to MLPs as cleanly described in Hinton's paper, but rather to some kind of autoregressive non-linear parametrized functions with a much more specific application scope.
Both papers are direct applications of the chain rule applied to estimate the gradient of a multivariate function.
hyttioaoa 3 hours ago [-]
That's what bugs me about him. So much work has gone into today's models that calling his contributions "the root" isn't really warranted. He's always complaining that Hinton, LeCun, and Bengio get more credit than they deserve, and now he's over-claiming himself.
BoredPositron 2 hours ago [-]
Both can be right.
emil-lp 3 hours ago [-]
Surely the roots go back to Turing, Gödel, Hilbert, Frege, Leibniz, Aristoteles.
sagex 1 hours ago [-]
I believe invention of Transformers and especially Attention mechanism do have influence from past research but its not definitely only the Schmidhuber's work. Said that, if we remove the papers mentioned by Schmidhuber from history, I am quite certain that there will be no influence in the discovery of Transformers, hence his works can not be the root. He has to grow up and accept that work and equations can appear similar, looking at inverse squared law and saying Newton stole that from someone is being dishonest.
And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.
When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.
But still, 30 years later here we are, standing on top of the shoulders of this useless research.
And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.
You can't claim independence from past work simply because you didn't look directly at it. The job of an academic researcher is to know the landscape of relevant ideas, where they come from, where they're going, and to hopefully contribute a few new good ones.
Citation chains should extend back from your work, along a reasonable line conceptual inheritance, back to a reasonable point of origin. Schmidhuber has different definitions for both of these reasonables than the bulk of the ML research community, to a point that makes him difficult to satisfy.
But if you build on them you should have read them. I don't know about the specifics and I don't know if Schmidhuber is out of line or not, and citations and impact factors are a terrible mess, but generally speaking, you are responsible for finding and reading and citing any related work that needs to be cited, and if you work on neural networks in an academic context you probably have been forced to read that particular one at some point. Citation obligations don't just disappear because you don't want to do the research.
The closest to that that I've seen is that traditional academia approaches are too far removed from practical applications for highly applied fields like software engineering, or too slow for fast-moving fields like modern day ML (thus, all the preprints).
Practically no one is against hard science research, properly conducted. The issues are rampant fraud / p-hacking / unreproducible garbage mixed with an unhealthy dose of ideological monoculture and indoctrination, garnished with rising tuition prices while sitting on huge endowments in case of the Ivy Leagues.
As long as you do that with your own money (or money got freely given from other people), sure.
If you use taxpayer money, that's a different game.
However I often see this going from "there's issues" to discounting academia altogether and positioning private labs as a good or only alternative.
After all, most people in the open science collaboration which published the seminal paper kicking off the replication crisis were from academia.
No, the research was made in USSR, however much Scmidhuber likes to think of "occupied Ukraine".
I mean, if one thinks it is his mission to establish the truth..
The truth, Scmidhuber, was never in your fuhrer's hands. Nor it is in the hands of the western fuhrers of today.
Just for the context, today is Russia's Commemoration Day of the victims of the Velikaya Otechestvennaya Voyna (your translations always feel wrong, sorry) of 1941-1945. (Yes, 1941, when the western fascist coalition of 5 million soldiers, invaded Soviet Union.)
26 and more million people of USSR perished, 13 million civilians. Of all nationalities.
In the Schmidhuber case their is 20 years and a chain of countless other works in between the two.
Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was.
No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.
related paragraph from Wikipedia:
Modern backpropagation was first published by Seppo Linnainmaa as "reverse mode of automatic differentiation" (1970)[26] for discrete connected networks of nested differentiable functions.[27][28][29]
In 1982, Paul Werbos applied backpropagation to MLPs in the way that has become standard.
Both papers are direct applications of the chain rule applied to estimate the gradient of a multivariate function.