I had to make a few simplifications to spell out the differences clearly and avoid making the text infinitely long. It's true that most current gradient descent algorithms are stochastic because they are computed in batch mode, and that sophisticated evolution strategies approximate the gradient. I still think the differences are significant, in that evolution updates less often and the direction of the update is much less (if at all) dependent on the feedback.
Now, your point about to what extent this is really about neural networks is a good one. Could a network learn F=ma, even if we could not interpret it? Maybe. With the right data, represented the right way.
"a systematic enterprise that builds and organizes knowledge"
That is exactly my point. Doing your scholarship is doing your part to organize knowledge. Systematically building knowledge means building on others, and knowing what you build on.
So the quote (which you don't attribute to anyone other than "wiki" - a citation here would be useful) really proves my point. If you're doing not doing your scholarship, you are not doing science, or research, you are tinkering.
Interestingly enough, when I have found research papers on the subject that I am looking at, most are so opaque that it was not worth reading in the first place. More helpful information has been presented by others who you would classify as tinkerers. They have been clearer in their explanations and much easier to build upon.
In the sense of presenting the research in a manner that other people can digest, academia seems to be more of a ancient guild than an organisation for expanding knowledge.
Academic papers are written for an audience of experts. If you wish to understand them, you’ll need to be an expert in the field (you can sometimes get away with a bit less, depending on the topic). They are not intended for general dissemination.
>Academic papers are written for an audience of experts.
You and parent are saying the same thing. Just because they are written for experts does not mean they are not written in an opaque manner.
And as someone who was once in academia, they really are written for 2 reasons:
1. To get past the peer review process.
2. To be written in the minimal time possible.
Enlightening peers comes a distant 3rd.
Example: It took days for a grad student/professor to derive a formula that is included in the paper.
Professor insists the derivation not be included in the paper. Insists the student not even mention in a few sentences the steps to get to it. Claims "any expert should be able to do this. No need to add it to the paper."
It took that expert days to do it. Unless it turns out to be a seminal paper, I guarantee that in most cases, no reader of the paper will even try. An error in the derivation? No one would catch it. Clearly, the professor is not even writing for his peers.
I do agree with the parent - the practices approach that of a guild more than any objective measure of explaining things.
One could also argue its a lack of distillation. Yes, an expert in field could/should understand, but does that mean it should be accepted that the papers are inaccessible to without same or greater level of expertise? Even experts struggle with opaque nature of many papers. Not saying that they should be dumbed down or using less precise language in order to appeal to wider audience, but more than a little more effort could increase value to all.
Good point, I could have used "hacker" instead, though that's more specific to writing software. I didn't perceive "tinkerer" as a word with negative connotations when I wrote it.
Come to think of it, many outside the hacker community would perceive "hacker" more negatively than "tinkerer", because many people still think of hackers as criminal. At least that's my perception. I guess words have different values in different contexts.
> Science (from Latin scientia, meaning "knowledge")[2][3]:58 is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[a]
Incidentally, the "wiki" as a system separates "scholarship" from "authoring". "scholarship" is done collaboratively post factum, given substance "authored" by various overlapping sources.
I'm the author, and I'm somewhat amused by the negative tone of many of the comments. Seems there are plenty of people with resentments towards research here ;)
I think it's because it treats hackers as deficient academics who can't be bothered to cite their sources properly or do any research, while there's an equal and opposite argument that academics are deficient hackers who can't be bothered to put their code on github or write any documentation.
In reality, neither is really deficient, they're just aiming at different audiences which need different things.
Oh yes, I'm a terrible hacker, I rarely write code anymore. An unfortunate byproduct of academia is that you get "promoted out of the job", and all the hands-on work is done by your lab members.
I didn't mean to imply that tinkering was inferior to research - the whole premise was just to tease out how they're different, with different audiences, as you say. Interestingly, the discussion here has been dominated by people who think that I look down on them. People who've discussed it in other fora have not read the post that way.
which cites two [1][2] of your papers among others :
yet you say :
"
it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up.
"
a bit harsh perhaps ? At least mentioning the foundational NEAT paper is not a complete lack of scholarship.
I feel you have overlooked that Seth does try and provide good citations to his audience and it would kind if you mentioned him by name in your article rather than " some guy "
He did inspire other twitch streamers to experiment with neuroevolution and neural nets and benchmark many SNES games.[3] although downstream this follow up does goes uncited by yourself.
My main criticism with your otherwise excellent article is the use of the word "tinkerer". "Tinker" has a pejorative meaning: "to busy oneself with a thing without useful results", "to work unskillfully or clumsily at anything." to take a couple of definitions from a random dictionary search.
I'm old enough to remember "shared source" and the classification of "hobbyists" in Microsoft's licenses. In that case the intent was to try to create a division between "professional" programmers, who would pay for useful access to source code and "hobbyists", who supposedly had no need for useful access to source code (because they aren't "professional").
I'm quite certain that's not your intent (and you say so several times in your article), but the implication still lingers. My area of interest is in language design and practitioners have made significant contributions to the field. Researchers are enabled because the endless experimentation of practitioners have narrowed the search space, even though it may have been done in an inefficient manner (by reinventing the wheel many more times than necessary). In turn practitioners have benefited greatly from the exhaustive knowledge and documentation of the researchers.
Both scholarship and practice are useful, but it is understandable that one group may emphasise one over the other and get different kinds of useful results.
I don't think you will find it against research per se, but against the arrogance that many academic researchers exhibit towards those who are not in academia. Once one has a PhD, it seems to be a mental requirement that one will have or show disdain to those who do not.
I have met humble academics, I have met many more who are arrogant know-it-alls. It behooves all to understand that each one of us has a limited set of knowledge. Especially when one becomes more "expert" in some field.
I think one reason that various groups of people has a distaste for science and academics is the fairly common "I'm better than you because I've earned a degree or two or three and know more than you do" attitude displayed.
One aspect of research and publication which has been highlighted many times is that replication of experiments that have been published don't get any funding or recognition.
In that regard, what seems to happen in research is that instead of further investigating the slight anomalies that are found, it is often just let go because it is not within the scope of what was initially being investigated. It seems to be a rendition of the attitude of "Move along, there's nothing to see here".
In case it was unclear, I personally am not at all resentful toward academic research. It is just that for structural reasons related to social organization and funding model, it has both advantages and disadvantages, and the latter shouldn’t be ignored.
Your post did a nice job explaining some of the reasons that academics do the type of work they do, and how that differs from hobby side projects. I think it’s more interesting to discuss the trade-offs involved instead of just passively agreeing with the OP. [Also, many non-academic projects are more organized than someone’s weekend hobby hack (large scale volunteer efforts, funded by donations and sponsorships, directly corporate-run, etc.), so it is worth discussing those as well.]
My comment is probably also considered "negative" but I actually come from an academic background and appreciate the value and importance of good research.
In my opinion, the major problem with your post is that your view crystallises the ivory tower mentality instead of really discussing the difference between "researchers" and "tinkerers". In reality, statuses and titles aside, there is only a fine line between a seriously good "tinkerer" and a seriously good "researcher", and the assertions you have made are simply shallow and narrow.
Your attitude towards opposing, but non-hostile opinions is also rather off-putting—you are simply making light of people who wish to add to the discussion and dismissing them instead of addressing them. It doesn't seem to be that of a reasonable person who is willing to consider different views and explore different possibilities, and definitely not one of a good researcher.
It's hard to have a conversation when so many have angst or animosity toward academia. how many times have I read denigrating opinions of the validity of research here on HN
I suspect that some people here have had to take software written by academics and convert it to production quality code. From experience it's a pretty thankless task :)
This is all true. However, the large publishers (IEEE, ACM, Springer) these days all allow you to self-archive on your own webpage and typically also on ArXiv. So you can (and should) make your papers freely available. I have done that since the start of my career. It's just a matter of looking.
He mentions NEAT, which is good, but not any of the work on playing Mario, much less any work on evolving neural nets to play Mario. There's quite a few papers on the topic, not just mine.
Note that I don't say that SethBling is wrong. Seen as tinkering, this is perfectly fine, and he did a great job with the video. No hard feelings. I just use it as an example of how it is not research, because of the lack of scholarship.
Like basically all researchers in CS these days, I keep all my papers freely downloadable from my webpage. Easily findable via Google Scholar, or just a regular Google search, for anyone who cares to look.
Yet, before today, I had never heard of you or your work. I download and read from a broad selection of papers and fields.
Mayhaps, you need to do a bit more "youtubing" to get your stuff out there and more accessible. A couple of my current favourites on youtube is 3Blue1Brown and Mathologer. They are both entertaining as well as being informative and have set me on investigatory paths for other research in various fields.
Why should they be familiar to you? There are hundreds of thousands of academics around the world working in thousands of distinct fields. Promotion on YouTube can be useful, but not nearly so much as you're thinking.
I am an AI researcher and faculty member at a large and famous university. I probably know less math than that Reddit poster. Math is important if you are specifically interested in the math of AI. If you are interested in inventing algorithms and solutions you mostly don't need the math.
Glad you liked it! We have done some work on trying to generate complete games - an overview of it can be found in this book chapter:
http://pcgbook.com/wp-content/uploads/chapter06.pdf
The most similar to what you describe is probably the ASP approach to game generation.
Thanks for link anyway. Yeah, it's a subset of my approach but a good one for clear intro to subject. As they were describing it, my mind drifted back to attempts at automated programming. I've seen recent work under banner program synthesis. In any case, I was thinking how they specified facts & constraints to force generation of results could be applied to software-rewriting. You start with unsafe code expressed with certain meaning/semantics, extract those as abstract rules, and then use them + safety heuristics to synthesize equivalent program that's safe.
Now, your point about to what extent this is really about neural networks is a good one. Could a network learn F=ma, even if we could not interpret it? Maybe. With the right data, represented the right way.