In this interview, Melissa Benua, senior backend engineer for PlayFab, explains the new way of life that continuous integration brings. She imparts practical advice for creating builds and running automation on the fly without spending hundreds of hours or thousands of dollars.
Jennifer Bonine: All right, we are back with our next interview, and we are really excited to have Melissa with us. Melissa, thanks for joining me today.
Melissa Benua: Sure, happy to be here.
Jennifer Bonine: Glad to have you, and for folks out there that don't know, I met Melissa about a year ago when we were at STARWEST and we were doing the Women Who Test, the first annual version of Women Who Test. I know Alison [Wade] talked about it if you were watching the interviews yesterday. Why don't you tell us, Melissa, a little bit about your background? You have just kind of an interesting background in what you've done and how you got into this and how you actually got here, you know, at these conferences. I think it'd be great for people to hear.
Melissa Benua: Sure, yeah. I came from, originally, Boeing, but I was a long time at Microsoft, on Bing in particular. I was on Bing before Bing was Bing. I started as an SDET on the live search team. I was a part of Microsoft as they started doing transitions from SDETs to combined engineering. We were on the team that demoed it years before they rolled it out. Also on the team, since I was the tester that rolled in with the devs, I still took on a lot of the test duties, but in a different fashion, so doing a lot of continuous integration stuff, code quality, test frameworks, all that stuff as a combined engineer.
Since then, about eighteen months ago, I left to go, as all Microsoft developers do, it seems like, go in a startup and try my luck there. I'm doing a lot of the same stuff that I was doing at Bing, but now I'm the sole responsible person because my startup is still quite small. I started a game startup called PlayFab, so doing a lot of code quality. When I showed up to my startup, I think they had seventy tests total and maybe forty of them passed.
Since then, you know, I've enrolled ... We're a very small engineering team, but I've enrolled my engineering team into doing what we believe is a really high end, really cool continuous integration framework. We test with every code review, we run our full suite of automation tests, thousands of tests. By the time a human has looked at it, we already know that it's good. You get past your quality bar or you don't, so when you land into your master branch that has already passed quality bar, so you have very minimal checks before anything goes live.
Jennifer Bonine: Wow.
Melissa Benua: We deploy it pretty safely, pretty fast. Very fast.
Jennifer Bonine: Yeah, wow. You guys moved into now when we talk about continuous integration, there's also continuous delivery and then continuous deployment. Are you guys all the way into ... Do you stop the code at all along the way or are you going right to production with it?
Melissa Benua: Right now we are stopping it just because we have a lot of live games, and more for the comfort of our game developers who are on our platform as opposed to us, because they are very ... The game world is a very slow development world as opposed to the online services world, which is very rapid, so they get very nervous about it. For their sake, we deploy ... We could deploy daily. We would deploy as often as we need to, but not every changed master every deploy, even though we could.
Jennifer Bonine: Perfect. You could do that. Important to note, right, because a lot of people talk about how advanced can people get and a lot of companies won't get to that point where they actually do it for various reasons, but if you wanted to, you guys have evolved to the point where you could.
Melissa Benua: Yeah, it wouldn't be ... There's some days where we do because we've got a bunch of changes have to go, so some days we do deploy four, five, eight times, perfectly safely. Honestly, the game developers never know but when they see the list of ... We publish our release notes, they get a little ... What do you mean you published? How could you possibly? To them, it's a sign of danger. When they do eight patches, it's a bad day.
Jennifer Bonine: Yeah, it's not good. No, so that's so neat that you guys have gotten to that point. I know we talked a little bit yesterday about some of the way that you guys are getting ready because with any startup, the next generation is okay, how do I get the next round of funding? How do I ramp up? I think it's very interesting about ... In case people out there are looking at how do I bring in interns, how do I select them, how do I ramp that up?
You, I think, put in a great process to be able to say, how do I know who the right person is to bring in? I know a question people always ask and when we were talking to Jeff Payne yesterday he said in his leadership summits people always say, "How do I get the right people? How do I select the right people in my hiring or just even in my temporary staff? How do I do that?" Would you mind sharing a little bit about how you're doing that?
Melissa Benua: Sure. Yeah, so something we've started for a lot of our engineering hires is the concept of a take home test. It used to be first that you would come through and do a loop with us and then if we liked you we would give you the take home test and if you did okay, then that was fine. Now we've inverted it, so we're hiring what we call a pack of interns this year, so the only way to reasonably screen a pack of interns is to have some relatively non-subjective way of verifying who can do the basics?
We put together a coding test first, like a little project. We estimate it would take maybe ten hours at most. A relatively interesting coding problem, and then also integrated with a lot of our SDKs and a lot of our backend service features. Using that, it became kind of a screening process. We stuck it up on GitHub and everybody said, thank you. Everybody who passed a basic resume review, thank you for applying. Please go do this test.
This test, nice thing about it, is it's an obvious pass/fail. We have a set of sample inputs we gave them and we know what the output should be, so in our system it ends up being a leaderboard, so it's a very obvious thing. It either worked or it didn't. We have another additional set of inputs that we use to verify more advanced cases, so if you pass all of these stages, you're pretty much good to know. We'll do a phone screen with you to make sure you're a good culture fit, but that's very hard to fail that test.
Jennifer Bonine: Yeah. Then with those applicants, obviously besides pass or fail, you guys had the opportunity to look at their code and just like art, it's all different. It's subjective. I'm sure you see all different ... Some people get there very seamlessly and beautifully and then some people it's a little bit more tricky on how they get there, just their own styles.
Melissa Benua: Right. These interns that we're bringing in are going to be writing real production code. We can't afford not to. We're too small. We want them to not just be able to write the correct output code, but this code has to be somewhat maintainable. Obviously we don't expect true professional code from an intern, but the hallmarks are there, somebody who knows how to write more than one class, everything not in the main method. The ones we actually included as a seed, we said here's your ... We gave them a fusion file and a .csproj file where their code should go, and then we also just attached an empty unit test project as a gentle prompting of, are you going to write some unit tests?
Jennifer Bonine: Are you going to do that, please?
Melissa Benua: The developers who've written unit tests, almost everyone passed and the ones who don't generally don't. Their answer's not correct. It'll be correct for the basic case, but not for the more advanced cases.
Jennifer Bonine: Interesting.
Melissa Benua: Very interesting.
Jennifer Bonine: Wow. For people out there to note, again, great way to do that, great way to screen through, and a very objective way to evaluate people so that you're looking ... I equate it to The Voice where you just get the code versus evaluating how the person looks or other things about them.
Melissa Benua: Exactly. I don't even want to know their names even. If somebody could strip the names off of that, I would happily process as Candidate A, B, C to try and take as much bias as we can out of the system.
Jennifer Bonine: Exactly. That's awesome. Great way to think about it. I want to touch on, too, with you a little bit about what you're talking about at Women Who Test for those that didn't attend last year and maybe can't attend this year. Again, I would tell people out there if you're local to the area, if you can join us tomorrow, you're more than welcome to come down and participate in Women Who Test. It's a free event. No cost. Obviously it starts in the morning, tomorrow at 8:30, I believe.
Melissa Benua: I think so.
Jennifer Bonine: It goes all day, so you can come for part of it or all of it, but just come down and register if you're local. What is the topic, Melissa, that you'll be talking about tomorrow?
Melissa Benua: I'm speaking about how to be ... I think the title is Lone Female Voice of Reason.
Jennifer Bonine: I love that title, by the way. When I saw that, I'm like, oh my gosh, how many of us can relate to that, right, being the lone female voice of reason in a room sometimes?
Melissa Benua: Exactly. You know, I'm very often the only woman either in my group or in my product group or expand the room, it still keeps going, so it's a lot about how to function not necessarily as entry level or as the high level, but you're an engineer day to day doing your job, how to be heard and how to survive and float and thrive in an environment where you're surrounded by men. That's the reality for a great deal of us is you're one woman in a room with a lot of men who are bigger and sometimes older, but just so many.
Jennifer Bonine: Yeah, so many of them. Any tips for people out there who are in that position? I know you give a lot of very practical examples of how do you do that, be that voice of reason, get your point across, get people to listen and hear you when you have something to say.
Melissa Benua: My biggest feedback is always have data. Use your data. Before you go into any meeting and have to persuade somebody verbally, see if you can have all your ducks in a row. Say, rather than I think that or I feel that, say, "This data shows from the run that I did last week that 33 percent of our users are leaving this page after eight seconds, or have a very short dwell time. I think this metric feeds into this other metric which feeds into this other metric." People are much more likely to take you seriously if you have some data backing that they can relate to because your communication styles may be different, but everybody ... We're generally all engineers and we generally all understand data and metrics and what the actual impact is.
Jennifer Bonine: Absolutely. I know right after you leave us here for this interview you're headed off to do a session.
Melissa Benua: Yes.
Jennifer Bonine: What session are you doing next?
Melissa Benua: My technical session here is called Continuous Integration: A New Way of Life. It's about the pipeline that I was talking about earlier that we built, making sure that your process works for you and so that it's quick and that it's safe because ... We use a couple of really cool tools, one of them is called Fabricator, which I know a lot of people don't know about, and one of them is Jenkins, which I'm sure almost everybody knows about.
Using these two things plus monitoring on the end and process in the beginning, you can ... As I think I said before, you can roll your build and your test and most of your verification stages into code review, and then with the right monitoring on the end and the right deployment process in the beginning, you can deploy anytime very safely.
It goes a little more to that than how to set that up, the pitfalls, some of the cool tech things to do, but the focus that I have is always on it should be cheap, fast, and free, or as close to free as we can. It's easy to set up since we don't have time, especially to start we don't have six months to set up and a team to set up integration pipeline. We have me with my four hours a day and whoever else I can convince to give me an hour an convince to give me an hour. It has to work. It has to be stable and it has to be fast.
Jennifer Bonine: Yeah. Any insights for you for folks out there who maybe are in a large Fortune 500, Fortune 100 company. You obviously came from Microsoft, very large organization, and did, as you say, a lot of people who go into a startup. What was your experience or transition going from what I call one of the behemoths to one of the small organizations where it's you and if you don't have the ideas, the knowledge, the way to do it, the suggestions, you don't look around and see hundreds of other people doing what you're doing.
Melissa Benua: Right. I guess my best piece of advice about that is to get used to figuring it out on your own and to be okay with failing, to be okay with trying something and it's wrong and say, "Okay, great." You pick yourself up and you move on to the next thing. To be in an organization that is okay with failing. To not be afraid of the responsibility. I spent my first probably month or two terrified of making a change because not only was I not ... They didn't have the CI process that I was familiar with in Bing, but any major change I could do could actually sink the entire company.
Jennifer Bonine: Right.
Melissa Benua: You know, if I take the site down, it's not just whoopsies, it's ...
Jennifer Bonine: This is huge.
Melissa Benua: This is huge, exactly. My whole team is very ... My whole company, I should say, because my team is my company, are very forgiving of making mistakes. Make a mistake, it's okay to make a mistake once, and then we learn from it, we put a process in place so it doesn't happen again, and we move on. We do the best we can just to ... Don't get stopped, don't get stuck. You've got to just keep going.
Jennifer Bonine: Yeah, and understand that's part of the process, right?
Melissa Benua: Right.
Jennifer Bonine: You know, the whole concept with agile and everything is fail fast so that you don't wait six months to make the mistake. Let's figure out what those mistakes are, adapt and adjust quickly and then move on.
Melissa Benua: Exactly. That was actually how I enrolled my team, but I would make a mistake and then fix it and put a test in place and slowly they would see that my mistakes became less and less and that my changes got safer and safer and the rest of the team's like, "Hey, that's really cool." Everybody started enrolling into the process and doing great code together.
Jennifer Bonine: Yeah, and that's a great way to do it. Any advice on obviously when you selected your startup making sure every startup and every organization has a culture and understanding, like you said, the culture you guys have accepts that it's okay to make a mistake, but you learn from it and move on. Did you go through any process to make sure it was a good fit for you, personally, because that's a big risk going from a large company to a smaller one.
Melissa Benua: It is, and especially for me, as I walked into my startup and I said, "How many women do you have? Who are you guys?" Gaming is kind of a notorious field for being a woman in and sometimes you have to be a little bit careful.
Jennifer Bonine: Right.
Melissa Benua: I actually came through with a personal recommendation from an ex-teammate of mine who said, "Hey, these guys are really good. You should go talk to them." I went in and I take my first look and remember you're interviewing them as much as they're interviewing you.
Jennifer Bonine: Absolutely.
Melissa Benua: They need you, right? They need you just as much as you need a job. I made sure that I asked a lot of questions about, what do you guys do together? What's your development process? How do you function day to day? Just trust your gut feeling that you get. If everybody is open with you or if they seem to be hiding stuff. When I came in, they had a big culture of openness, so if I asked about the financial, how long are you funded?
They answered. They were straightforward. They said, funded for X months, X years, whatever. Just make sure that you have a good gut check. Personal recommendations I can't emphasize enough. Knowing somebody who knows somebody who can say, "Yes, these guys not just are technically confident, but I've worked with him. He's a good guy. She's a good girl. They work well together."
Jennifer Bonine: Yeah, they're good people to work with.
Melissa Benua: Exactly.
Jennifer Bonine: Thanks, Melissa. The time goes so fast I can't believe that we're ...
Melissa Benua: Oh gosh.
Jennifer Bonine: I know, right? It just flies and we're done. How can people contact you if they want to know more, have questions? Maybe they're facing a similar challenge of I'm at a big company but I really want to do more and be in a startup. What's the best way to get a hold of you?
Melissa Benua: The fastest way is my Twitter handle is @QueenofCode.
Jennifer Bonine: You will never forget that, yeah.
Melissa Benua: You can also look up I'm on Slideshare, and on LinkedIn with Melissa Benua. I'm the only one, so it's very distinctive.
Jennifer Bonine: Perfect. Easy to find. Melissa, thanks so much.
Melissa Benua: Thank you. I'll see you tomorrow.
Jennifer Bonine: Yes. Tune in for our next interview.
Senior backend engineer for PlayFab Melissa Benua has worked in nearly every software development role—engineer, test, DevOps, and program management—in her career. Melissa has created and run high-availability, high-quality services at Boeing and Microsoft on Bing, Cortana, and Xbox One. She discovered her love of massively-scaled systems while working on the Bing backend, honing the art of keeping highly-available complex systems up while undergoing massive code churn. Whether working with classic or cutting-edge testing models, Melissa isn’t afraid to mix traditional approaches with bold new ideas to make her products better, faster, and more reliable. She is passionate about about sharing best practices with colleagues and the tech world.
