How We Write Interview Questions
In this article we are going to demonstrate how we write interview questions for our programming tests so hopefully you can learn from our techniques and build your own interview questions.
What matters to us most is developing high-quality questions. We’ll explain in detail how we do this.
Ok, let’s jump right in!
We divide the whole process of building a test into three parts:
(HINT — The last is the most important.)
1. Searching for talents
The key point here is to have several professionals (programmers, in our case) work on a test. We’ll explain why in part #3.
It’s great if you have enough stellar employees at your company that you can attract them to write interview questions, but in our case, because we create such a wide variety of programming tests, we always hire developers from outside.
Chances are good that you can find great talent on the market because a) you search all over the world, and b) candidates don’t need to abandon their current job because this is short-term, side project.
Of course, you should pay enough to keep the best talent interested; in exchange, you will have quality interview questions which you can count on in your hiring process.
So where can you find talent?
We mainly use 3 sources: StackOverflow, GitHub and Twitter.
StackOverflow is our first choice because it’s a programming community, users have a rating, and they also have sub-rating for specific programming disciplines. Plus you can inspect all the answers given by a developer and find out how experienced, skilled and intelligent he/she is.
GitHub allows you to see the projects which eligible candidates participate in, their contribution to a project, and even their code commits.
It’s tough to search for talent though. GitHub is better for evaluating developers rather than finding them.
Twitter allows you to search for relevant topics (programming disciplines, in our case) and find talented professionals.
And once you find one, there’s a “You may also like” section leading to many other candidates.
2. Writing the questions
Once we have hired the talent, we provide a platform for building interview questions. It looks like this:
(By the way, if you are interested in building your own interview test, you can try the platform yourself: we are offering a 30-day free trial).
What are our requirements for interview questions? We have developed a set of basic rules that we ask developers to follow when they write programming questions for us.
Here are the full instructions our developers see before they begin writing questions:
Hi. Please read this carefully.
We’ve just created the test for you below. Please click on it and try to create new questions (but first, please finish reading!).
Here is the basic idea of what kind of questions we’d like to have:
Imagine you are hiring a developer for your own team to write a “billion dollar” app. What questions would you ask in a live interview?
Here are the requirements:
- The test should be written for mid-level developers, not juniors and not team-leads. Don’t make the test too tricky or too simple.
- Please use code examples in your questions and answers instead of asking theoretical questions. This is mandatory!
- In addition to writing the questions, we need to define the categories. Each question will belong to a defined category.
- For each question we need to have a brief explanation describing why the question is worthy of inclusion in the test. Please note that the explanation is not for candidates; it is for us and for other developers who will review your questions.
-
We need to AVOID the following types of questions:
- Syntax questions. Knowledge of syntax and the ability to write good applications are different things. The right syntax could be easily found in Google.
- Tricky questions. We want to test developers’ skills; we don’t want to force them to make a mistake.
- General questions. If you enter the text of your question in Google and find the answer on the first page, it means that’s a bad question. Better to use code examples.
- Please, no plagiarism. Otherwise, we will end our partnership immediately.
Have any other questions? Don’t hesitate to contact us. It’s always better to ask more questions in the beginning rather than having to fix mistakes later.
Good luck, and thank you in advance!
3. Reviewing the questions
As I said in part #1, we always hire several developers to work on a single programming test. Usually, it’s 4 or 5 developers per test.
Why?
When we started our service a long time ago, we thought to ourselves “Ok, now we are going to hire a guru and he/she will write us the best programming test imaginable.” And then it did not work…
Why not? For a simple reason: We all are human beings and we all treat information subjectively.
What do I mean by that?
All developers have their own unique working experience. Any given topic that a particular developer thinks is important to ask about in an interview, may, in fact, be not so popular in the community.
Also we encountered many examples of misleading questions and answers (such as those which might be both right or wrong, depending on the situation). Even if a developer is a guru, it does not mean they expresses their thoughts clearly or have good writing skills.
That’s why we count on a team when we write questions — team members leave their reviews and point out if a question or an answer is misleading or might be subjectively treated.
Here is the message we show everyone who participates in a review:
Hi. Now we are ready to start the review process!
You will need to check each question carefully and rate it with 1 to 5 marks in terms of value for testing candidates.
Please try to be as objective as possible.
What should you pay attention to?
- We asked developers to avoid syntax, tricky, too basic, and easily Googled questions. If you see this kind of question, the rank should be decreased.
-
Check the correctness of a question and its answers. Here, I mean simple mistakes, such as an incorrect answer marked as correct, or a code sample has errors, etc.
We have a special checkbox for these kinds of mistakes and the question’s rating should be 1 only. - Finally, beware of misleading questions. Questions should be clear for the candidate. If a question can have different correct answers in different situations or the correct answer is too subjective, then it’s a bad question.
The purpose of this review is to determine nice questions. Imagine if you were a candidate yourself; would you like a particular question? Does the question help demonstrate that you are a good programmer? Does it validate your programming skills? Is it interesting?
Many people will be tested with the final version of this test. Sometimes, the results will be life-changing for them. So, we need to choose the best questions and make the test perfect.
Good luck, and thank you in advance!
After the review is done, it’s time to determine the best questions for the final version of a test — i.e., the one we release.
We used to have Excel sheets for that:
Now we’ve built a platform especially for the review process designed to guide us in selecting the best questions.
This is how a reviewer reviews a question:
The overall rating and all reviews for a question:
Thank you for reading! If you have any ideas for how to improve the process of writing interview questions, please leave them in the comments below.
Coding skills assessment
If you're looking to hire software engineers and want to ensure that you only interview qualified candidates, we've created coding tests that can significantly simplify your hiring process: