As usual, vague terms like "90% accuracy" are thrown around without specifying what exactly they mean. Is that a 10% false positive rate? A 10% false negative rate? 90% accuracy over a representative sample of the general population?
Because the last one is what would seem to be implied, but I have a much simpler program, written in pure JavaScript, that is also 90% accurate at diagnosing Americans with depression: https://gist.github.com/osuushi/5297823
Since, according to the CDC[1], about 10% of Americans are depressed, my script will be accurate 90% of the time.
This is a good point. The parent article doesn't link to the paper but in it they give all of their hypothesis with p values <0.05, so the can reject the null hypothesis and give statistical confidence to theirs:
90% isn't all that great of a result considering only about 6.7% of the US population suffers from depression each year[1] and the study was done in California. Just returning false would give you better accuracy.
Still Eliza on steroids is pretty cool. Can't wait till they integrate it into emacs.
They give their hypothesis in the paper, and each is shown with a p value < 0.05, so is statistically significant (and better than just returning false :)
p values are a measure of the accuracy of the numbers. They are measuring how close the sample mean is to the population mean. In the case of the paper, the p values represent how close the sample mean of the head gaze, eye gaze, smile intensity and smile duration measurements are to the population means of the same values.
I've only skimmed it, but the vision work looks sound, and it looks like it makes pretty essential use of the depth map from the Kinect. But I don't think there's any speech recognition going on, so that part is just acting (from both the humans and the virtual platform). I'll bet an untrained user could get the system to break pretty quickly...
Another thing is that this needs a properly trained interviewer, and it's not based on the Kinect's measurements alone but also on the questions asked (and the reaction is then measured). The behaviour of the interviewer is a great confounding factor in this.
Reminds me of a certain someone asking another certain someone why he flipped a tortoise in the desert...
I think the problem isn't with the published paper - it looks like they used a trained human interviewer and recorded both humans to evaluate their descriptors. The real problem is with the PR video, which looks really impressive, but seriously oversells the capabilities of the system. Not that the problem is unique to this situation - I think a lot of the videos in AI oversell the research. It's a pity, because the research is usually very good, but by itself isn't "exciting enough for general consumption," so they add bells and whistles that have nothing to do with the science.
You're right, posting the paper and not that blog-post would have led to a much more interesting discussion.
One of the strangest things in that SimSensei is the automated interviewer - when I talk to a recorded voice instead of a real person, I behave enormously different! Why should I fidget around in answering when no-one really listens anyway? Why should I give proper answers? Why should I exhibit signs of shame when I talk to no-one about myself?
I'll address your second point first. I agree that it would be very interesting to see the methodology. You're right that it's the algorithm that determines it, but it's the Kinect that's powering the CVA. Also, sorry I opted for a shorter, less accurate title. (ps. I'm not affiliate to the blog or product. I just thought it was interesting)
This is interesting to me because... is there even a scientific definition of depression yet? Such that, if you have a defined list of symptoms, you are definitely depressed, and if you don't, you are definitely not depressed?
If not, we're at best talking about a magnitudinal thing or a probability thing. Like a Bayesian thing where each symptom's presence updates the probability of the person being depressed, which is turn based off of a database of previous cases of diagnosed depression and what symptoms they had.
But even that may be circular, ultimately. It could just as well be that depression is a vague category of similar symptoms that each come from a wide variety of scientific causes.
This is not to be mistaken for saying it doesn't exist or that it's all in someone's head - it's more just to say that if someone has depression, they have something that may deserve treatment and sensitivity, but that it's something that we still do not know exactly how to describe or diagnose, and so therefore we will just call it "depression" in the meantime. But what if we've sort of collectively forgotten the "in the meantime" part?
> This is interesting to me because... is there even a scientific definition of depression yet? Such that, if you have a defined list of symptoms, you are definitely depressed, and if you don't, you are definitely not depressed?
It depends a bit on what you mean. In the U.S. at least, diagnosis of depression is generally based on the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders (DSM).[1] That lists a number of disorders that would be categorized under "depression", including "Major Depressive Disorder", "Dysthymia", and the catchily named "Depressive Disorder Not Otherwise Specified".
In the DSM, these disorders are all treated as syndromes, meaning they are defined entirely by symptoms. They may have organic causes (troubles with neurotransmitters, etc.), but the definition of each disorder does not address that issue. This is in contrast to something like influenza, which refers to a particular organic cause (infection by a certain kind of virus).
As for the actual definitions, they vary in how specific they are. Major Depressive Disorder is quite specific, requiring certain kinds of episodes to occur with specified frequencies and lengths. On the other end of the scale is DD-NOS, which is essentially defined as a depression-ish thing that doesn't fit into any of the other categories. (Okay, it's a little more precise than that, but, honestly, not much.)
> This is not to be mistaken for saying it doesn't exist or that it's all in someone's head - it's more just to say that if someone has depression, they have something that may deserve treatment and sensitivity, but that it's something that we still do not know exactly how to describe or diagnose, and so therefore we will just call it "depression" in the meantime.
I'd say that's pretty much on target. However, biology & biochemistry are proceeding forward at a breakneck pace these days. We do have some understanding of the causes for some kinds of depression, and this understanding seems to be improving significantly each year.
MRI's are expensive (in the US). Specialist mental health care is not covered under most standard health insurance plans. How about you just fill out the questionnaire[1], and we'll match you with a nice (newly patented) pill?
> What about correlations with patterns in MRI scans?
An excellent question, about which I know absolutely nothing.
Something to keep in mind, though: medical tests & whatnot are only useful insofar as they affect the way we handle treatment. Diagnosing depression really isn't that tough; if someone has a debilitating mood disorder, then it is reasonable to get them some kind of help. (I imagine that, for some of these people, the really hard problem is getting them to figure out they have troubles that the medical establishment can help with.)
So I don't think we need MRIs to tell whether someone is depressed. OTOH, can MRIs tell us what treatments are likely to be successful? That sounds like a very interesting question. Hopefully, someone is looking into it.
I studied psych in college a few years back and read about a variety of studies on how individuals with slight-moderate depression respond to certain stimuli. One of the most interesting studies (unrelated to the topic) had to deal with individuals looking into a mirror after failing a task. Individuals with >= slight depression would look themselves in the eyes, while happily ignorant individuals would avoid eye contact with themselves. They attributed this with a depressed person's desire to figure out what is wrong with themselves and internalize the failure, where as the happy people would avoid internalizing the failure.
> They attributed this with a depressed person's desire to figure out what is wrong with themselves and internalize the failure, where as the happy people would avoid internalizing the failure.
I've also read that depressed people tend to have a more accurate picture of reality. Basically, happy people are walking Dunning-Kruger cases waiting to happen?
I believe I know the study you are referring to and you are sort of right. on a scale of accuracy, slightly depressed people are pretty accurate in self assessments, happy people tend to over exaggerate their abilities, and moderately-severely depressed individuals rate their abilities lower than their actual level. So if the person's average ability was a 5/10, happy people tended to rate themselves a 6/10 and moderately depressed people a 4/10 on average iirc.
Edit: My favorite example of Dunning-Kruger happens to be found in religion. The people that know God the most tend to talk about how little they know God, where as the proselytizers that claim the most to know God, really have no clue about the teachings of christ, mohammed, etc.
"SimSensei can work out whether you’re exhibiting signs that indicate depression"
Im not sure I totally buy it - Im sure it can pick up the signs with 90% accuracy but how often do the signs actually correspond to a clinically depressed person. Just because your in a down mood at one moment does not mean you are depressed - depression is just a lot more complex than that. Im not sure that this would be of much use in accurate diagnosis although it seems like pretty interesting tech.
Depends on the kind of questions they're asking, really.
As someone with relatively mild depression, you'd never know it in normal conversation. But if you ask directly about symptoms and I give honest answers, both the answers themselves and my body language will probably give you a pretty good indication.
Doctors tried something similar with paper forms in the 60s. It was pretty disastrous, given depression is such a complex, "3d" kind of issue. Adam Curtis explored this theme in perhaps his best series of documentaries, "The Trap" - http://www.youtube.com/watch?v=WbRApO3k_Jo
If it's as accurate as the team says it is, this is pretty astounding IMO. They'd have to be capturing some sort of baseline in the beginning of the conversation, and recording relative shifts of variables (ie; smile, eye shift/location) to come up with how "depressed" the person is. Just writing some software that says "oh this person is looking down" probably wouldn't cut it-- because shy people look down too. I noticed in the demo it seemed as if it started monitoring these things after certain questions were asked, which would be the correct way to measure a shift from baseline. Elicit a predictable period of what could be change and record.
Actually, I'm looking to do something like this in related applications. Just need to find time to go through the kinect capabilities....my own unit has been gathering dust on my shelf for a while. Anyone in southern california (LA area or within driving distance) want to talk about this?
Because the last one is what would seem to be implied, but I have a much simpler program, written in pure JavaScript, that is also 90% accurate at diagnosing Americans with depression: https://gist.github.com/osuushi/5297823
Since, according to the CDC[1], about 10% of Americans are depressed, my script will be accurate 90% of the time.
[1] http://www.cdc.gov/features/dsdepression/