Testing, Heavy Metal, and Unrealistic Automation Expectations: An Interview with Paul Grizzaffi

Josiah Renaudin: Welcome back to another TechWell interview. Today I am joined by Paul Grizzaffi, a principal automation architect at Magenic and keynote speaker at this year's STAREAST. He'll be speaking on parallels between testing and heavy metal music. Paul, thanks so much for joining us today.

Paul Grizzaffi: Hey, thanks for having me.

Josiah Renaudin: Absolutely. And before we really dig into content of your keynote, could you tell us just a bit about your experience in the industry?

Paul Grizzaffi: Sure, I was recruited out of grad school by a company called Bell Northern Research, or BNR, which was the research arm for Northern Telecom, and then fast forward about ten years there, Northern Telecom became Nortel. A bunch of the listeners here probably have at least heard of, if not actually worked for Nortel.

Nortel, I was there for sixteen years. I started out as, sort of stumbled into being an automation developer. I wanted to go and do C++, and I wanted to do parser theory, which is what my grad school specialty was. I found a place to do it there at Nortel, and it actually happened to be on what we now would call a test automation team. I loved it, and sort of grew up as an employee and as an automator through that world. And eventually, when I left due to bankruptcy, I was the automation architect, and then eventually the tools architect there.

I did a stint with a stock trading company, then I spent almost three years at GameStop as their automation manager and architect, then on to a company called MedAssets. And now I work for a company called Magenic. We're a consulting company where we do general consulting, sort of end-to-end, turnkey stuff. But we also will do QA only engagements, and automation only engagements as well.

Josiah Renaudin: And, of course, since I work at TechWell, I think all of the keynotes and sessions sound great, but yours stood out, as I said, because testing and heavy metal music is not something I've really heard compared before. What made you think about the parallels between those?

Paul Grizzaffi: When I did my first talk, four, five ... It's 2013, so what's that? That's five years ago. I kind of got the bug for doing talks and stuff. I said, "Man, I've got to figure out a way to do one on metal. I've got to find a metal convention or something like that to go and do a presentation at." And then as time went on, I started figuring out that I'd been a fan of rock music, and rock and roll, and heavy music since early childhood. Some cousins came to live with us for a while, and they brought their Kiss records and their Beatles records and stuff, and I was really drawn to the Kiss records, partially for the imagery, because I was like seven years old at the time, but also the music, and the multi-part harmonies, and the fact that it was heavier but also sort of radio-ish brought me into that whole thing, so I kind of got indoctrinated there.

But then fast-forward to the talks and all. I really wanted a way to do this. What I figured out was I have songs running through my head all the time—lyrics, movies, all kinds of things running through my head all the time, in addition to all the tech stuff that I'm doing, so it's kind of jumbled up there. But what I will do is when I run into a situation, it kind of reminds me of a lyric, and I'll kind of play that lyric in my head, and I started writing those down.

So, this talk has been in slow formation over the last probably four years. Taking notes here, taking notes there. It was going to be an article for StickyMinds. It was going to be a blog post. Then I had the opportunity to do the keynote here and I said, "Well, let me float that one out there too. There's no way they're going to pick this one, but I have to at least propose it because this is sort of ... " In my head, it was always keynote material. Lo and behold, it was picked, and then I had to write it.

Josiah Renaudin: The hardest part, of course. When you pitch something thinking, "They'll never choose it," it's probably a positive and negative surprise. Because when it actually gets picked, you're like, "Oh, now I've got to put this whole thing together."

Paul Grizzaffi: It really was. Because in one way, it was very easy to write, because what I did was I said, "Okay, I've got these thoughts that I want to get out, and then I've got these lyrics that always kind of jibe to that." But then there were other thoughts that I had lyrics for and went, "Oh, those are kind of offensive," or, "The lyric's not offensive, but the song or the album cover might be." So, I had to go and be a little more creative for some of those, simply because I didn't want to offend the audience. If you or if anyone listening is familiar with or isn't familiar with heavy metal, sometimes it can be a bit extreme, in its politics, in its lyrics, in its vocabulary, and I didn't want to alienate a lot of the people that would be attending.

Josiah Renaudin: So this is a little bit of the PG-rated keynote versus maybe what you initially had in your head.

Paul Grizzaffi: More of a PG-13, because I'm a bit on the edgy side, but I'm not expecting anybody to run out with their eyes covered or their ears covered going, "No!"

Josiah Renaudin: Well, at least we got it up to PG-13.

Paul Grizzaffi: Barely.

Josiah Renaudin: In the abstract you provided, you talk about how testers and those involved in automation are often faced with less than stellar situations. What are some of the most common unrealistic expectations that you found different management groups at different companies put on testers in the modern day?

Paul Grizzaffi: So you really hit on the core, is that it's unreal expectations. But if I had to pick one that's sort of thematically through all of the jobs I've had and the engagements I've been on, it's that automation is something you just do. I get questions like, "How long is it going to take you to automate five hundred test cases?" Well, depending on how cavalier I want to be, the response back is, "I don't know. How long is it going to take the developers to write five hundred classes?" Because it's kind of the same thing; you can't know because you don't know until you get into it to figure out where the icky bits are going to be, where the challenges are going to be.

Once you get in, you might figure out, okay, on average it's probably going to take us about this long, plus or minus 25 percent. But really at the beginning of an engagement, 25 percent's probably the best bit you're going to do, because there's always going to be some ugly bits where either it's not feasible to automate it or you could automate it, but the return that you're going to get, the value you're going to get out of there, is not going to be what you want it to be.

And then also what's really ... There's a lack of understanding that you need to treat this as a software engagement. You need to have experienced people working on it. Or, if you're going to start with inexperienced people, that's fine too, just be prepared for them to make mistakes, and allow them to grow and evolve on that, but you have to be very patient. It's going to take a lot longer to do it that way.

And it takes planning. You have to plan this in. There's an opportunity cost where if you're automating with people that you already have, then there's some other stuff they're not doing. Or, if you're going to spend some money to bring in consultants, or contractors, or additional employees, then you're not spending that money on something else, and those things aren't always realized when the initial conversations around automation happen. It's really, "Hey, we're going to get some automation done, and we're going to automate because we do the Agile or we do the DevOps, so we do the automation."

Josiah Renaudin: Building on that idea… Having talked to different people during these interviews, I feel like one of the main things is that management doesn't fully understand what automation is, the time it takes, the skills it requires. So how difficult can it be for testers and those who have a grasp of automation to convince these decision makers that not every test case can and should be automated? Is there a belief, and this is something I've heard, that maybe automation is automatic, when in reality, that's not even close to the case?

Paul Grizzaffi: Yes, there frequently is that it's automatic. Now there's some interesting things coming on the horizon with AI and such that I don't completely grasp, and I don't completely know where it's going to fit in this continuum of automation and tools to help testers do their job. But when I look at these things and say, "What are the bogus expectations?" For the most part, we're not at the point anymore where the average leader believes that 100 percent automation is possible. But they often believe that more is possible or more is probably than really actually is.

But if we start talking about opportunity costs, like we talked about or I mentioned earlier, and they understand opportunity costs, and you've got someone else that can talk to them about that, they start to understand these trade-offs that they may have, that perhaps all the crank turning that they're doing; perhaps that could be automated. Perhaps a machine can do that for them, but still you need some human interaction there. I've talked about this with other peoples. In my career time, and probably in my kid's career time, and they're only four, so in their career time, I don't expect that you're going to see this removal of human beings from the testing process, wholesale.

I believe some companies are doing it. I believe some companies are saying that, "I'm going to do ... Every check-in, it's going to get pushed out. No human looks at it." And that's fine. That's fine, but that's a risk that company's willing to take. A risk that a company's willing to take to say, if I have a website and I'm doing it with that, I'm taking the risk that nobody fat-fingered anything and all of the text and the background are pink, so not only is it unreadable, it's a little hard on the eyes.

You're taking that kind of a risk, or risk with things at that level. That may be good business for you. It may be good business for that organization, but it might not be good risk for an organization that's say working on ... I don't know, a guidance system for a missile, or perhaps for a heart monitor, or a pace maker, something along those lines.

You have to look at this in terms of business and in technology. I think a lot of times people miss the business part. They look just at the technology part and this notion, this illusion of now we're going to save money.

Josiah Renaudin: And you did mention before how you realized you kind of always had different songs playing in your head during testing. To kind of bring it back to the music side of this, have you found a lot of testers are also musicians? Do you know a lot of people in software who either play an instrument or have bands? Is it more about the musical nature of creating music, or more about the aspect of understanding and respecting music?

Paul Grizzaffi: Some of both. And since you expanded it out into software, I find in general the people that I interact with in the software industry, there is a high percentage of them that are either very much into music, or play music, or a combination of both. I can't say that it is proportionately higher than any other industry or any other sort of discipline. But yes, I do run into people who definitely do have bands and understand and appreciate some of the interaction and the analogous nature between music and creation of software where it's part mechanical and it's part art.

As far as bands, I've been fortunate. I've got a lot of friends that are in bands and band related jobs. My best friend actually used to tour with some of the bands back in the '80s, some of the bigger metal bands, so we've gotten hooked up for concert tickets, and backstage, and stuff like that. It's been very cool because you're a little teenager growing up in little oil town Louisiana and you go, "God, I just would like to see this band once," much less go and sit and have drinks with them, or have dinner with them, or whatever.

Josiah Renaudin: Do you think heavy metal in particular holds answers to these difficult testing or software questions? Or let's just say, for example, can ska help you? Can country music? Which I'd prefer if you actually said it couldn't. If we can leave country music out of this, that would be fine. But can different genres speak to different people?

Paul Grizzaffi: Yes, absolutely. To me, I'm a metalhead. Metal speaks to me. I do listen to some other things, especially now that I have the kids. I try to give them a broad palette of things to listen to, but we do occasionally have our metal mornings with dad on the way to work or the way to school. But yeah, absolutely, whatever inspires you. Some people, it's going to be different types of music. Some people, it's going to be different types of art, different types of movies. I know some people that are doing a podcast on ... It's called Screen Testing, and it's on movies. They talk mostly movies, and sometimes they weave a little testing in there as well. So it's really whatever inspires you, can help you make these parallels and kind of make it a little more personal to you, bring it to where it's a little more at home, and something you're even more comfortable, or even more excited about doing.

Josiah Renaudin: You did say before, "This is PG-13," so some of the songs that maybe you wanted to include, you couldn't fully include. But what are some of the songs you're going to reference in your keynote that you can kind of tease?

Paul Grizzaffi: Well, I won't tease the songs, but will tease that you're going to hear ... So when I chose the songs, I chose from bands that people would recognize and some bands that will be new to most of the audience, simply to give some people something new in addition to the lyrical content and the analogous parts of testing, and maybe some new parts of testing and automation they haven't thought of. But, you're going to hear lyrics from Ozzy Osbourne and Anthrax, which are fairly mainstream, popular. Ozzy's very mainstream in the normal pop culture now. Anthrax is more so, or sort of in that Metallica, Megadeth range, where metalheads know them and a few, probably rock people have heard the name as well. But I'm also going to reference some bands called Iced Earth and King's X, which I'm big fans of both of these bands for very different reasons, and we'll talk a little about their lyrics and what they mean to me, and what they mean to testing and automation as well.

Josiah Renaudin: All right, perfect. Again, Paul, I appreciate your time. Just last thing here. More than anything, what central message do you want to leave with your keynote audience? What's the thing you want them to take away and maybe tell their team when they get back?

Paul Grizzaffi: So in addition to attempting to recruit a big new legion of metalheads, what I'm really looking for is I want people who are engaging in automation to be responsible. I want them to think about and embrace the fact that there are two major facets when you're dealing with automation.

One is technology, and we usually get that right about half, 75 percent of the time. And then there's the business, which we get right about 25 percent of the time. Because people just aren't thinking about this as a business endeavor, they think about it as a process endeavor. They think about it as a technology endeavor. "Oh, we're Scrum. We have to have automation. We're DevOps. All the testing has to be automated." It's automation without forethought. It's automation without planning. It's automation without understanding the potential business consequences of what you're doing.

So, if you take this business face and this technology face, and you let the two inform each other, you can make a much more cogent, and a much healthier decision around what to automate, when to automate, and how to automate. Because if you do anything else, you run the risk of being irresponsible. Being irresponsible with automation is one of the things that I really see a lot of out there. When people have problems, it's usually because they weren't fully informed at the beginning.

Josiah Renaudin: All right, perfect. Thanks so much, Paul, I really again appreciate the time. Your topic does stand out. Thank you for expanding a little bit here. I'm excited to hear the full thing at STAREAST this year.

Paul Grizzaffi: Absolutely. My pleasure. Anybody that's going to give me a chance to talk about metal and automation, I'm going to take it.

Paul Grizzaffi is a principal automation architect at Magenic. His career focus has been on the creation and deployment of automated test strategies, frameworks, tools, and platforms. Paul has created automation platforms and tool frameworks based on proprietary, open source, and vendor-supplied tool chains in diverse product environments of telecom, stock trading, e-commerce, and healthcare. A Certified ScrumMaster, he is on the Industry Advisory Board of the Advanced Research Center for Software Testing and Quality Assurance at UT Dallas. Contact Paul via LinkedIn or Twitter and read his blog.