We created this set of questions to help corporate managers select Agile-experienced consultants and candidate employees for project work. Assembling a team of qualified Agile people is one thing, but the fact that some Agile practices and principles mean different things to different people makes it even harder to succeed in staffing your initiatives.
A consultant or potential employee who has great Agile experience in small product companies may be exactly the wrong type of person to help you inside a large corporation. An Agile evangelist from a small product company may actually be horrified by some of the compromises that are absolutely necessary in your large company environment. We developed this questionnaire to make those distinctions.
When we're talking about the "large company environment" we mean a Fortune 1000 company that has an internal IT shop with lots of development projects going on, most of which are non-Agile. Large projects and projects-within-programs are very common in this environment.
A big reason we're providing this information is because we've seen too many situations where an "Agile expert" is brought into a big company and falls flat on their face because they have an expectation that the large company will be able to switch over all at once to the Agile mindset. While it's true that Agile can be very compelling, we've seen these attempts to be futile time after time. It makes much more sense to move to an iterative/incremental project lifecycle iteratively and incrementally.
Depending on the role you are interviewing for, you may want to leave certain questions out or perhaps add some of your own. Certainly there will be questions you want to ask about things like the person's outlook towards businesspeople and technical people working closely together, which is important no matter whether the project is waterfall or Agile.
- How long were the iterations (or sprints) on the projects you worked on?
Agile project methodology moves at a fast pace and you should already have a good idea of the length of the iterations for the pending project. Answers of between 1-week to 3-weeks are ideal. If your candidate has worked on Agile projects which had long iterations (4 weeks or longer), or wildly variable-length iterations, it will make sense to determine if this person is comfortable with the iterations as defined for your project. A steady set of fixed-length iterations that are fairly short is best. The theory that big companies need longer iterations is not based in fact. We've done many big company projects in the 1 to 2 week iteration range.
2. Are you a Certified Scrum Master (CSM)? If so, how do you view the way Scrum projects need to be organized?
Often we use certifications as a golden way to qualify candidates. But be somewhat careful with people who have gone through the Certified Scrum Master (CSM) training. CSM training can sometimes have an anti-project management bent to it, depending on the class instructor. Quiz your candidate to find out if they feel that the idea of project management is "antiquated" or "out-of-date." Ask them if they feel that "every leader of a team should also be a coder." The candidate may use terms like "self-organizing team" which should signal you to ask more questions. This is not to say that CSM training or even the concept of a self-organizing team is bad or unworkable, but we are hoping to help you, the corporate manager, to identify people who will be able to work within the limitations of your group. If you want to move towards an environment without project managers, that's fine. But if you cannot do that in the short term, be sure to find out if your candidates fit what you require.
3. Did you use automated test tools on your projects? Explain how that worked.
Agile project team members should have experience (or at the very least, the desire) to use automated testing tools for regression and performance testing of each iteration. At the end of each iteration you want to have something that your customer/client can "see." Automated testing allows quick identification and isolation of development defects as well as the ability to test development work completed in previous iterations. Expect the candidate to talk about automated regression testing, continuous integration and even performance testing techniques and tools. Also expect them to discuss the need for manual testing as well as automated regression testing, due to the ever-changing nature of the code base in Agile.
4. Have you done continuous integration on a project before? Describe.
Here you want to get a pretty detailed explanation of how the candidate has used continuous integration on previous projects. Continuous integration is a set of automated build, integration and test steps that executes against the code base in a configuration management repository. For instance, if you were using Java and CVS, the CVS repository would have a set of triggers that automatically built, integrated and unit tested the code often, perhaps each night, perhaps a few times a week, or even every time someone checked in new code. Each of these is a valid continuous integration story.
5. Did your iterations overlap? For instance, were the testers still testing Iteration 6 while Iteration 5 was being designed/developed?
There are many views of how iterative/incremental projects should run under Agile. Overlapping iterations is certainly one strategy. Pay attention to the candidate if they say "iterations should never overlap." This may tell you that the candidate is used to having what is called "multidisciplinary teams." This type of team consists of a set of generic team members who all have the skills and enthusiasm for requirements, design, coding and testing and who each participate in those activities almost equally. If your company has defined roles (business analyst, test analyst, etc.) and career paths then this candidate may have a tough time fitting into your structure. They will want BAs to code and testers to do design. Is that okay? It is your decision. Again, nothing wrong with multidisciplinary teams, but your organization must be able to handle the change if you decide to go that way.
6. Have you used story cards or use cases? Explain how that worked for the team.
Often, Agile projects are associated with the use of story cards. However, our goal is to simply understand if our potential team member is comfortable implementing a project (designing, developing, testing, etc) with business information that has been documented in some way. The requirements (story cards or use cases or a combination of both) should have enough information to provide guidelines for developing and testing and also allow the development team to come up with a creative and effective solution. By asking this question, you want to understand if your potential team member is comfortable working with requirements in a structured development environment versus brief summaries augmented with face-to-face communication, from which the developers build code. Again, this is a matter of matching the Agile candidate's experience with your organization's needs.
7. How did you manage traceability of the requirements to testing?
The point here is to make sure testing goes all the way back to requirements for validation. Not only is it important to test that the functionality a developer has created during an iteration actually functions, it is also important to determine if it functions the way the business wanted it to function. Does it meet the requirements defined in the story card / use case? Your team members should understand the importance of this concept and if they understand and accept traceability, you should be able to count on this person to help you meet project goals.
8. How comfortable are you with ever-changing requirements?
Many development methodologies specify that requirements are locked down at the beginning of a project. Although that is not the case in Agile, it does not mean that requirements are loosey-goosey! The advantage of Agile and its short iterations is that it is easy to quickly recognize that work is not progressing as desired - that what the customer ‘asked for' is not what they ‘wanted' - and the requirements must be changed. Team members should be able to handle such changes on an Agile project. The requirements picture shouldn't be so tied to code, a story card or any other component of work that prevents providing a solution that meets the customer's needs. The general idea is that requirements can change a lot at the beginning of the release, and very little at the end.
9. Did people play multiple roles on your development team?
Here we get back to multidisciplinary teams. What you are really trying to understand from this question is how well an individual would fit into your organization and your style of Agile development. If your organization is one in which individuals select a specialized role (such as Java Developer) as part of their career path, your interview candidate may have difficulty on your team if she is more comfortable working in Agile projects where she has had the ability to play multiple roles (Scrum for example). Conversely, if your candidate's primary experience was developed on projects where roles were clearly defined and individuals did not work in multiple capacities, then he may be uncomfortable in an organization where team members can play any role on a project to achieve its end goal. You must choose the candidate that fits your organization well and is flexible enough to suit the needs of your Agile project implementation. Two particular roles that are more easily interchanged are business analysts and testers, other roles are harder to be "multifunctional" ."
10. What project management tools were used on your project?
This question is more for Agile project managers. Do you have corporate PM tool standards? If so, this question becomes quite important. Agile has a new breed of PM tools including Rally Software, Version One and XPlanner. These tool bear no resemblance to the waterfall PM tools like MS-Project or Clarity. If your candidate is more comfortable using one of these Agile PM tools, try to identify if they will be able to fit their Agile project plans (and issues, milestones, change requests, etc.) into your company's structure.
11. Can you explain the purpose of a burndown chart?
This is more of a question for candidate project managers, although all Agile practitioners should know the answer. A burndown chart is a graph that shows the progress of the team in terms of work "burned through." The work is usually put in terms of a set of "story points" which represent functionality. Once a piece of functionality is coded and tested and reviewed by the user, it is considered to be "burned" and the graph will reflect this. The graph should show a steady movement down until it is clear that the team will have completed the story point backlog by the release date. There is also a variation called a "burnup chart" which is similar to the burndown chart. Again, you want to see how the candidate will link their burndown/burnup chart into your overall project management structure.
12. What does project velocity mean?
This is a project manager question, but most Agile practitioners should know the answer. Project velocity is the rate at which a team is "burning" through story points, so a possible velocity might be "30 story points per iteration." That means that so far, the team has been able to identify, code and test 30 units of functionality (story points) in an average iteration and can expect to do about that much in future iterations, giving a fairly good view towards what can be accomplished by a release date.
Use these questions when interviewing people for specific roles on your Agile projects, or especially for those you are considering to bring in as Agile coaches or mentors. You want to have a very clear idea that the coach's ideas will mesh well with the structures that you have in place in your large organization. Unfortunately, there are Agile coaches who feel that the external environment doesn't play a factor in their mentorship of your teams. As you can tell, we disagree. Agile can work in a large company, but it has to be tailored and even compromised to fit the realities that you face when setting up and executing projects.
Hopefully, these twelve questions can be a good start for your qualification process in bringing in new Agile consultants or candidate employees. Our hope is that you can build a highly-qualified team that will fit in well with your corporate development process, culture and PM standards.
References
Our work is based on the research paper by Dr. Hong Li.
Here is some good reading on Agile in the enterprise:
Augustine, S. - Managing Agile Projects (Prentice Hall PTR, 2005), 264p
Highsmith, J. - Agile Project Management (Addison-Wesley, 2004), 312p
Leffingwell, D. - Scaling Software Agility (Addison-Wesley, 2007), 384p
Possible related articles and blogs of interest:
LitheSpeed Blog lithespeed.blogspot.com
The Agile Executive trailridgeconsulting.com/blog
Leading Answers leadinganswers.typepad.com
Scaling Software Agility scalingsoftwareagility.wordpress.com
About the Authors
Anita Shankar is a consultant with Perficient, Inc (htttp://www.perficient.com), a NASDAQ-listed IT consulting firm. She is currently enjoying an assignment consulting for an international consumer goods company and working on her next article. Anita is based in Columbus, Ohio and can be contacted at [email protected].
