I apologize to anyone who thought our post was callous; definitely not what we intended. Let me try to better articulate what we were trying to convey...
If you post a developer job on Monster, you're likely to get over 500 resumes. Over 95% of them (I'm using that number literally, not figuratively) are from people who shouldn't have applied if they'd read the job description carefully.
Reading through those resumes takes a vast amount of time. It makes it harder to identify the qualified applicants because they're more likely to get lost in the shuffle. It isn't fair to them. We have only three full-time developers at SeatGeek. There's a finite number of hours per week we can dedicate to reading resumes.
It isn't that I think people incapable of passing our test are "worthless", as samd says below, that's a gross mis-characterization. But we want to focus our hiring efforts on folks who are highly capable applicants, and if we read the resume of everyone who responded to a typical Monster post, then we wouldn't have the resources to do that.
I think we all agree that it can be very useful to filter applicants by making them solve some puzzle, and there's nothing wrong with that.
What bothered me was that it sounded like you were mocking people for trying but failing to solve your puzzle, and relishing in the fact that you didn't have to consider their application. Trying is a good thing, even if it ends in failure, and it should be encouraged.
It might even turn out that the sort of person who spends an afternoon trying to solve a challenging puzzle makes a great employee after a bit of training.
I think it's the job of the future employer to "waste their time" finding and filtering people rather than getting applicants to waste their time applying. I've solved quite few puzzles in my day. What irked me was getting chosen for my solutions and then getting rejected for other, arbitrary criteria.
An employer generally has more leverage in a transaction like this, let them put in some effort in compensation.
The main problem is that the spread of tints is too small.
If you exclude black and white (and gray scale) you will normally have around 8-10 colors to work with if you don't want your wheel to look too Schizophrenic.
You can choose between:
Dark Blue
Light Blue
Dark Red
Light Red
Dark Yellow
Light Yellow
Dark Green
Light Green
Orange
Violet
Beige
Brown
Pink
You could also use textures (just like in the old IBM matrix printers did) to represent the variations.
I would though normally stay away from more then 5-6 color variations in a pie chart.
Note that the pie and the key are in the same order, so you can walk clockwise around the pie, starting at 12:00, and read the country names off the key, from top to bottom.
because that's what's often offered as default by chart drawing software? people with money love it ("the pie is getting larger!!!"), so it must be default.
I made it through, but had no intention of actually applying - the space just isn't very interesting, I'm not sure why I'd want to work for this startup in particular, and they're not sophisticated enough to work remotely.
Still, the challenge was fun and had a hint of the weebly job challenge to it (though that one was considerably tougher), to the point I was thinking about trying to come up with a series of these just as a weekly exercise to post on hn. Anyone else have ideas for this?
Many spent an entire afternoon working on this thing without success. I hope they liked the challenge, but I’m also glad we didn’t have to use our internal resources reviewing their resume.
How fucking pretentious can you get? Maybe they were busy, maybe they had kids to take care of, maybe they were documenting their steps vs. flailing around. Glad I don't work in the valley if this is the kind of attitudes "rockstar" employers have.
If they had kids to take care of at the time, I hope they did not spend an entire afternoon working on it without success. And if they took so much time documenting their steps that it made the project take so long they could not complete it, I would say that is a negative indicator.
In the end, IIRC it wasn't that hard a test for the sort of person who would be good at the job, so it seems like a reasonable filter. The only good reasons for not successfully completing it were 1) You didn't have time for it, or 2) You simply aren't ready at this point in your career.
I'd like to note that I have yet to pass the test - my first and only attempt was a curl one-liner that I gave up on when I realized I had work to complete :D.
I don't understand the outrage. Some completed this in 7-8 minutes. That's quite a difference from spending hours without a result. Add a few of these problems up (which I hope are representative of actual skills) and you're looking at minutes vs. days. Sure, the aside on resources comes off a bit flippant -- but it doesn't mean it's untrue.
1. Time to complete the task is a terrible metric for white-collar, knowledge work. There are myriad factors to explain the time discrepancies, especially since there wasn't first-hand observation. If this was a job for building a widget on a factory line, then time might be a reasonable metric.
2. Using time alone as a selection criteria borders on discrimination given that some users might have sight or other physical disabilities that require longer to complete the task.
3. Do you really want to work for someone that has that kind of attitude? Oh, you're worthless because you took a few hours longer than John over there. We're not even going to consider your other talents, abilities, capabilities, or creativity that you bring to the table.
To clarify - time it took to complete the challenge was never factored into the application process. We simply pulled this data because we thought it would make for an interesting metric to display in the post.
I just did it for the challenge, took me about 30 minutes, primarily because I didn't look at the cookies first and spend about 15 minutes wiring up greasemonkey to add an input field into the form. Turns out I needed this for the _csrf and cookie stuffing so it wasn't entirely a waste.
"Many spent an entire afternoon working on this thing without success. I hope they liked the challenge, but I’m also glad we didn’t have to use our internal resources reviewing their resume."
What a great attitude. People who work hard and challenge themselves are worthless, they don't even deserve the time it takes to reject their application. A person like that would never be of any value to your company. I mean, you can't expect people to learn new things and grow professionally while on the job. If they don't already know everything then they are idiots that you want nothing to do with.
Or perhaps they consider the skills needed to pass the test easily to be the bare minimum needed for the job, and anyone who can't pass it quickly isn't qualified (but hey, if you learned something from it, good for you!).
Could it be that the issue here is not whether they used a test to filter applicants, but the tone in which they describe their satisfaction with the result?
But I've noticed a pattern, which is that folks who talk a lot about using these tests talk in this rather offensive way. It makes me think that companies that start off with these test have somewhat abusive attitude towards their employees - "we don't mind making you jump through hoops so we don't have to". Might not be a signal this is a great place to work.
Maybe the company could promise to solve a puzzle I submitted if I solved their puzzle? It might seem fairer.
"If they don't already know everything then they are idiots that you want nothing to do with."
Well, why should a startup want anything to do with someone who needs training? A startup needs to get as much as possible out of its resources, including individual employees and time. Given that, and all else being equal, it just makes more sense to hire the people with a decent understanding of HTTP for a position related to web development.
It might not have been the most tactful way to say it, but I don't disagree with their premise.
I agree, the ideal candidate, particularly for a startup, would have a wealth of knowledge and experience in addition to the drive to challenge themselves and the ability to learn new things. The problem that startups face is that they can't usually hire those people because someone like Google or Facebook is paying them with truckloads of gold. Not always of course, sometimes startups can attract those people and it's great for them. But when they can't they must compromise; and if you have to choose between someone with knowledge and experience, and someone with drive and the ability to learn you're better served in the long run taking the latter. It will require some sacrifice in the beginning to bring them up to speed, but eventually they will overtake the candidate who stands still content with what they already know.
The attitude that I see coming through in that post is that they value what people already know more than what people can learn, which I think is backwards. On top of that they just came across sounding like assholes. I could be wrong though, it's hard to read people's intention through text.
In many architectural and engineering companies where CAD drafting is a core part of the job, the interview will often end with a drafting test.
You are given a plan or even a set of requirements and are given typically 15-30 minutes to complete/get as far as you can.
This is important because someone may speak well etc - but when it comes to actually completing work (which is typically done by billable hours) some people just wont have the speed or skill to produce the work product.
I would feel much better with a test at the end of an interview, once I know the interviewer has put in some effort seeing if I'm a fit in other ways, than a test merely to get the interview.
Sure, tests are very logical as a filter if people are just resources to be filtered. If you expect to have a personal relationship with me, as a person, though, starting that relationship with indication that I'm just a resource is not good.
This seems especially important in startup where you want to like your fellow teammates. Reciprocation is an important part of liking - psychologists say so!
When this posting was originally made here on HN, I noted that it really reminded me of a site I loved years ago that basically taught you web security by getting you to hack your way through various levels. The site (quiz.ngsec.com) is now defunct, but for those of you who enjoy this kind of challenge I have found several copycat sites which started around the same time and still exist.
One of the more fun ones which is still around is http://www.try2hack.nl/ so if you enjoyed the job application process you'll probably have fun with this.
I just played with that test for fun, it is pretty easy (OK, so HN technical users probably aren't the target here) but I do like it.
Without giving it away and spoiling it... it does well at covering the general things you would want web devs to intuitively know.
I'm more interested in how you monitored this stuff... setting up the test would've been pretty trivial, but getting good data out must've been harder.
Are you recording failed attempts too? So that should someone fail on the last step but have succeeded at the prior 2 steps that you still have the ability to contact them if their CV is good enough too?
I like this approach a lot. Assuming the idea is just to make it a teeny bit more difficult to apply than firing off an email to jobs@wherever.com, this is a great way to do it. It filters out those with no real interest in the company and establishes a competency floor (albeit a rather low one), and selects for people who are interested in solving problems in the most elegant way possible -- by actually giving them a problem to solve.
I didn't try this particular challenge, but it reminds me of Reddit's recent job postings (which I was able to complete in a couple minutes using Google and the command line on my Ubuntu machine) in spirit. Maybe the tone with which they communicated it was a little smug, but if you had your choice of people who spent as little as 7-8 minutes on this versus folks who needed to work on it for a whole afternoon, who would you choose to hire?
Alright, I'll be the idiot that asks for those that are quiet: could someone do a step by step guide as to how everything was done? Even reading the comments didn't get me any farther than changing the user agent (and having the form show up).
Here's the thing though, none of the test proved anything about a person's creativity. In my opinion, the most important part of being a developer. Proper technique can be taught creativity is an instinct. I look for creative people that have creative ideas over technical challenge. Because in the end, I want to train them to work in the way that my team works anyway. I understand the need to filter out the passionless but I think a purely academic exercise whether trivial or difficult does not accomplish the goal of finding a truly creative developer that can bring insight to an organization.
One of the best markers that I have found about a person and their ability to develop and conceive of creative solutions and business strategies, is their hobbies. Through a person's hobbies you can see their passion and their creativity. I look for developers who have hobbies that include antique car restoration, airplane building, hobbyist rockets, electronics embedded or otherwise, fine arts and other markers of creativity. I've found that these people, the creative types make the best developers.
This test wasn't the only metric they used, it was just the initial filter. A good developer needs to be able to come up with creative solutions, but also to just get down in the dirt and analytically solve a problem. Someone that is good at both makes the best developer.
Also, you apply your cheapest/most scalable filter when your volume is the highest. If you're worried about false negatives, you open it up a little wider and deal with the false positives at a later stage.
I didn't even know about the challenge until today, but trying it just now made me realize that Chromium's built in dev tools are lacking some must-have functionality.
1) You are saying too much. Saying stuff about looking for the best is obviously your purpose, so no need to say that. Now, you are left defending yourself about being Jerk. For somethings, it is best just to give the facts.
2) Why aren't you hiring developers from your network?
3) You should try to come off as "We are awesome, and we want you to be awesome too.", not "We are awesome and you are not".
I needed to Google around and like 1 hint and got it in about 26 minutes. Does that make me a good web developer? I feel like this is sysadmin stuff. I've personally never done any of that stuff before...is this what happens when you work for an enterprise company and you hide behind Javascript and C#?
I am curious to know if there is a site that aggregates companies that use puzzles and tests like this as a filter for job applicants. I think that these sort of filters are potentially quite useful for the job-seeker as well...
If you post a developer job on Monster, you're likely to get over 500 resumes. Over 95% of them (I'm using that number literally, not figuratively) are from people who shouldn't have applied if they'd read the job description carefully.
Reading through those resumes takes a vast amount of time. It makes it harder to identify the qualified applicants because they're more likely to get lost in the shuffle. It isn't fair to them. We have only three full-time developers at SeatGeek. There's a finite number of hours per week we can dedicate to reading resumes.
It isn't that I think people incapable of passing our test are "worthless", as samd says below, that's a gross mis-characterization. But we want to focus our hiring efforts on folks who are highly capable applicants, and if we read the resume of everyone who responded to a typical Monster post, then we wouldn't have the resources to do that.