For many engineers, thinking about spending countless hours of studying for coding interviews leaves them exhausted before they even get started. At Cockroach Labs, we have open sourced the interview process on Github to create familiarity for candidates and removed resumes and utilized exercise-based interviews to reduce bias. Beginning in mid-2017, we engaged with our engineering team on how we can make our engineering interview process less intimidating and more fair, resulting in better hiring decisions.
Traditional Engineering Interviews
Traditional engineering interviews focus heavily on real-time coding questions. At Cockroach Labs, our past interview process included five coding questions and one system design question for each engineering candidate. Our interviewers assessed the candidate’s skill with coding and debugging, algorithms, data structures, and overall code structure and design. These coding interviews were done in a “live coding” format either over Google Hangouts or in-person using CoderPad or the candidate’s preferred editor/IDE. Each interviewer asked their own coding question, and a handful of our engineers were prepared to ask a systems design question.
While all of the coding questions focused on technical concepts that our engineers work on within their roles, only the engineer owning that question was versed on the rubric for the assessment. For our one design interview, the assessments that resulted in ‘strong yes’ or ‘strong no’ led to the most confidence in the signal to hire or not. Our interviewer assessments that were on the fence were often disregarded because we could not find meaning in the signal on how to proceed. When this occurred, the only assessments that were meaningful were the interviews that focused on coding and debugging, algorithms, and data structures.
To make better hiring decisions, we wanted to make the engineering interview process more reflective of real-world engineering, focusing on assessments past coding and debugging, algorithms, and data structures and creating stronger hiring signals for a candidate’s experience with large-scale system design and design decisions.
Where to Start
Iterating on an interview process that our engineers have been using throughout their careers was extremely challenging. We anticipated that we would be able to roll out a new process in 1 quarter. However, the reality of structuring a new process and running betas to test the quality of assessments from interviews is a massive time investment, which we have worked on for the past 3 quarters.
Before we got started on changing the process, we first had to define our goals for our effort of iterating the engineering interview process.
For Cockroach Labs:
Increase fairness of interviews by creating familiarity for the candidates.
Get stronger signals from our interviews, and thus make better hiring decisions.
For Candidates:
Build a more fair interview process by offering familiarity and interviews that reflect their day-to-day roles.
Make sure that all of our questions apply to problems we solve in our daily work.
For Interviewers:
Create consistency and better calibration by using a small repository of questions that every engineer can ask, solve, and grade.
Create stronger signals from our interviews, resulting in more confident hiring decisions.
After discussion, we determined that we wanted to make a few changes to the interview process, specifically around coding and our design portion of the interview. To address both these areas, we added the following:
Take-home Exercise
Reduced Question Repository
Choose Your Own Design
Take-home Exercise
We decided that we wanted to let candidates get started at their own pace, in an environment that they choose, which mirrors more closely to what they do in their day-to-day work. Engineers can do the exercise whenever works for their schedule. While we let candidates know that the exercise usually takes one to two hours to complete, we ask candidates to let us know if that’s not the case for them so that we can make sure our expectations are reasonable.
We have simplified the process by asking the candidate to use a unique CoderPad link that contains the coding question, instructions, tips, and how to start test cases. Once the candidate has completed the question, they submit the link via our applicant tracking system for a member of our engineering team to review. The intent behind the take-home exercise is to make it as close to an on-the-job simulation as possible to create a familiar environment in which engineers can do their best work.
After reviewing a take-home exercise, assuming the submitted solution looks good to the reviewer, we schedule a follow-up with the candidate where they work with the interviewer, usually over a Google Hangout and CoderPad session, to extend their submission with additional or changed functionality. This we think is more representative of our day to day work -- where we modify existing code to add features or handle changed requirements, rather than starting with an blank slate every day. Additionally, lets the candidate demonstrate their ability while working with they wrote, which hopefully gives them a familiar and more comfortable starting point jumping into an interactive interview that could otherwise be more intimidating.
Reduced Question Repository
We want to see how candidates apply their skills to technical challenges that our engineers solve in their daily work and give candidates the opportunity to see what type of work they would be doing at Cockroach Labs. Previously, we had our interviewers ask coding questions that they defined and that only they understood the rubric for the assessment. These questions varied, with some being more or less challenging with regards to different skill areas. For example, some questions pushed on data structures, while other pushed on algorithms. The areas that each interviewer assessed was left up to who the recruiter decided to put on the interview slate.
By reducing the number of coding questions, we can ensure that every candidate sees similar questions, regardless of who is interviewing them. Also, every engineer on our team can ask, solve, and grade both take home challenges and onsite interviews. We believe that our interviewers should use the same, objective standards to assess a candidate’s skill set. This improves consistency in the questions and grading and increases the overall fairness of our interview process.
A New Systems Design Question
One area in which we knew we were lacking strong hiring signals was in our systems design assessments. In addition to an interview focused on large-scale system design, we created a “Choose Your Own” Systems Design Question. This question is a discussion between the candidate and interviewer about a system that the candidate has worked on in the past in any capacity. The recruiter prepares the candidate for this portion of the interview before coming onsite.
By speaking about something the candidate already knows in their first interview, they build some confidence moving into the rest of the interviews. It also allows for the candidate to share their subject matter expertise. While we are still very early on with this format, we have seen a notably greater confidence in the hiring team’s assessment of a candidate’s design skills since we now we have two systems design interviews - one focused on the candidate’s experience and one related to large-scale systems.
Continuing to Iterate
Ultimately, we made these updates to our engineering interview process to create a candidate experience that is less intimidating and fairer. We also felt that the changes with the new interview structure and reduced question repository will lead to more confidence in our hiring assessments.
We are still in initial stages. The engineering team is going through ongoing training with the question repository. Once we have a significant enough data set from candidate’s going through the hiring process, we will look at the data to determine if the goals that we set are being accomplished. From there, we will use the data and feedback from both our team and candidates to continue to iterate.
The engineer hiring process is one that continues to be iterated across technology companies. We encourage more companies to share their stories so we can all learn from each other.
Illustration by Wenting Li
