A Talk About Test Automation: An Interview with Mukesh Sharma

Cameron Philipp-Edmonds: All right. Today we have Mukesh Sharma. He's going to be speaking to us about test automation and his role at QA InfoTech and about his passions and insights as a tester. To start things off, can you tell us a little about yourself?

Mukesh Sharma: Yes. My name is Mukesh Sharma and I am the founder and chief executive officer of QA InfoTech. We are an independent software testing and quality assurance company. We've got around 20 years of experience in the software development world and I have had an opportunity to work in both development and testing across software companies before I moved back to India to start QA InfoTech around ten years back, more than ten years now.

Cameron Philipp-Edmonds: OK. The bio on your company website describes you as avid reader, but I also know that you're in the process of finishing up a book on crowdsource testing. What led you to go from reader to writer?

Mukesh Sharma: Yes. Test evangelism has been a special area of focus for me. I've been speaking at a lot of conferences. And, I think [a lot], for publications including TechWell, StickyMinds.

Cameron Philipp-Edmonds: And Better Software Magazine.

Mukesh Sharma: I have also been encouraging a lot of my employees to partake in such efforts and to this effect, we have an even which is internal to our organization, it's internal technical symposium that we organize every year with all of this work over the last two to three years, I have been selectively picking topics where I can share my experience in a broader forum to help create a more lasting knowledge base. The first book I wrote was on software testing at large called Are You Smart Enough to Be a Tester? The reason of what inspired me to write that book—so back when I graduated from my school, I did my Computer Science in '96, I was not aware of a career in software testing. All I knew was you could be a programmer to write C programs, C++, Java, or RDBMS database.

I wanted to be able to make people understand, some people who are going through the curriculum in their college or school, they deserve a career in software testing. When I formed QA InfoTech back in 2003, I had a major challenge wherein it was so difficult to find people who are willing to do software testing. So I thought of writing a book that would encourage people, those who are reading the book, go and speak in conferences take sessions and then I thought if I could write something like, “Are You Smart Enough to Be a Software Tester?” And have a clear career path specified in the book as a chapter. We have since published that book. The current one on crowdsource testing, which I am co-authoring with one of my co-workers, and I am happy to tell you that Taylor & Francis will be publishing this book which is scheduled some time, I believe, in September 2014.

Cameron Philipp-Edmonds: Fantastic and we're going to sit down in July and talk about that book as well, so thank you for that.

Mukesh Sharma: Yes.

Cameron Philipp-Edmonds: All right, and you also have some writings. If you want to check out Mukesh's writings you can check them out on TechWell.com, Stickyminds.com or the new and next issue of Better Software magazine. Now I'd like to jump into some questions about test automation. So really, when should a team look to get into test automation as either a supplement or replacement for manual testing?

Mukesh Sharma: Firstly I don't believe or see manual testing being fully replaced. That human touch is important. We are creating these softwares unlike the pharma industry wherein it takes a lot of time to research before they come up with one medicine with like 20 years of research. Here it is like quick products that are being created, that the thought of development cycle is very short. Over the years it has been shortened too much. You can't, the product itself is a piece of code, you can't have code testing the code and then completely rely on it. There is usability factor, there is accessibility factor, so I don't believe or see manual testing being fully replaced.

In fact, with the advancements in technologies such as wearable computing, etc. that are at now buzz or hot buzz coming into the mix, I believe some of these will take us back to the roots of software testing and bring in more of manual testing. That said, automation testing is a great supplement to the manual tester first, because they help you in bringing reputability and test coverage in specific areas. Now, when should a team look at test automation? Well, when you have a product with working functionality, the functionality is working fine and you know that there are multiple releases that are going to happen, there will be multiple upgrades or releases, you would want to consider automating the base functionality, to make sure that the future base do not end up breaking the working or the existing functionality.

So that is one scenario, the second scenario could be you are building a product in agile methodology. Sprint one: you created one feature. Sprint two: you added a few more. By the time you come to sprint four or sprint five, you do not want to come to a situation wherein the features in sprint one are broken due to whatever new thing is added. So it is important that you automate while you develop. That is how our test driven development is happening with a lot of continuous integration so, you can write automated test and have it integrated with continuous integration so when developers are checking in their code the test would be fired automatically.

Cameron Philipp-Edmonds: And then with test automation a lot of people tend to argue that they go to test automation because it kind of takes out the user error element and so a lot of people argue that test automation is infallible but when you end up creating a testing script from scratch, you know that kind of leaves just as much room for user error as manual testing would. So does test automation tend to give a false sense of security to some testers?

Mukesh Sharma: Yes, I would agree with you on that. There are times when it does give false sense of security to test engineers. It's true because sometimes teams look at test automation just as a feel good element of their testing achievements. We all know that that nothing is fool proof—even if it has been adequately tested for. There are false alarms that one needs to know. There are situations when you've created the script, you've written the script but it'll end up not working because of change in UI. I think what we can do is bring in more reliability into the test automation effort for us to buy into the research. Like I said the false positive so it is important to add reliability to it.

And testing teams have long understood the importance of testing the test script which is like, unless you do that, you might be raising false alarms. And then that is going to ensure both its consistency and reliability in the results. However, we have most often stopped with the initial round of tests that were performed on the automated test. What is now becoming more important and very common is to incorporate metrics comparing manual versus automated test when looking at the test effort metrics. These are metrics that will show what kind of bugs each of these test groups are reporting. What percentage of these are valid or invalid? For example let's say you've got thousand test cases that you automated but the bugs are, the majority of the bugs are, reported by the manual testers automated tests are always, “oh everything is passed”.

Cameron Philipp-Edmonds: Right.

Mukesh Sharma: These kind of analysis will give us a sense of how reliable the test automation effort whether the test team as a whole has the right test automation strategy or if it needs improvement and will also determine the amount of maintenance needed.

Cameron Philipp-Edmonds: Right. Speaking upon that with the human tests you can see the errors more than the computer would. Whether or not it's against automated testing is that it doesn't replicate the human experience. So with that being said, is there any room for creativity and emotion in automated testing?

Mukesh Sharma: Good question. Well automated tests are also indeed, they are like software they create. It's basically a piece of code that you are writing to test a piece of code. They do what we instruct them to do which is really pick candidates that will give us an increased return on investment on the automation cost we are incurring while today frameworks and tools are available in the tester's toolkit, to empower them to automate a very powerful and customizable, especially with the advent of open source technologies like Selenium and on. These still cannot completely replace the human touch.

For example, there are number of tools that greatly support an accessibility test effort but nothing can replace a visually challenged user evaluating a product. We can provide you real user feedback. We have a team of accessibility tester genius who are visually impaired. They work from our office and these folks have reported issues in some of the products that we test that even the product teams, they are people with six by six vision and also not being able to report. Because these guys are able to provide you with that human touch, the creativity and emotion so you cannot replace that, automation testing cannot provide that.

Cameron Philipp-Edmonds: Well that's very interesting. We talked about some of the drawbacks where it gives a false sense of security and you also got that human touch there, are there other drawback to automated testing that maybe someone new to testing or someone who is just learning about automated testing may not be aware of?

Mukesh Sharma: Well, understanding the value of test automation rather than just seeing it at face value is very important especially the newer ones with not much experience of doing test automation, they miss it. For example in my teams, it is still seen as a number game. In many teams, I am not sure if I got that word correct, so in many teams automation is still seen as a number game. People say “oh we are trying to have 100 percent test automated so my regression test is going to be completely automated and will achieve this” rather than seeing how that number will help them both in the current release and into the future.

There are some common risks with automation testing which I’ve mentioned in one of the earlier question also. Number one, have you picked out the right tool? Because if you end up choosing a tool that works with your current product but with the few enhancements or new things being added, your current tool would not support that so how does that, was that considered? Choosing the right tool is very important. Second one is, be aware of the false alarms. Just because you've automated, it is very, very much possible. You see, your automation scripts that you are writing are essentially to find defects in the product, right? To ensure that there is no functionality that was working earlier ends up breaking because of a newer release.

But what also needs to be clearly kept in mind is that it relates to your earlier question, human touch, human being can see an error, what if your application encounters an error that you have never thought of? How are exceptions handled? So exception and handling is also very important. Was exception handling in automation using the right messaging is very important? And then obviously false alarms. So, it results on an ongoing business, it results from the test automation are not analyzed it can lead to increasing test failures. It will not only impact the test effort, but also pull down the test team’s reputation. In this last thought, while we anyone can from reada planned test automation—effort is immense. It can be a double-edged sword, if not well-thought of. The initial planning will seem a little time-consuming but at the end of day it is an effort well-spent to build that robust test automation strategy.

Cameron Philipp-Edmonds: So you talked about testers coming across things that are unexpected, and having the skills to build a well-thought out, well-planned test automation so should test automation practices only be done by testers or can developers and programmers or otherwise complete the tasks of test automation just as well?

Mukesh Sharma: Developers can do and create if they were to think as a tester. But, I think the basic difference here with being a developer and a tester is that developers are aiming to build something and their test are going to essentially focus on ensuring that what they are building is working, OK? A developer could be writing a simple code and then that gets integrated with another developer's code. But the test, so this developer is only focused in testing its piece of code. As a test engineer, when we start testing we think about how is it going to be used by the end users?

We look at the over-all picture and we think of scenarios, we think of different types of customers who are going to be using the product. While a developer can be an excellent tester but in order for that to happen, the person would have to move completely into the test engineer role, so today, this whole thing evolving. I know that companies are hiring developers for testing, right? Their primary role is to go testing they are not writing code to build a product, but they are writing code to test the product that is being built.

Cameron Philipp-Edmonds: Right, OK. You've talked about tools too and, you know, kind of writing codes to test a product, and there is often people who will tout a prevailing theory that there is one tool to rule them all or strategy a process that is simply superior to the others. Is there an approach to test automation that you have found works really well for you in your team?

Mukesh Sharma: Very good question. There is no silver bullet. This is an area of ongoing research in an attempt to arrive at that one solution that maximizes scope of coverage, OK? In our experience so far, the test automation frameworks that we have built on top of existing open source tools have worked really well. We don't say no to share tools specifically if a client is already heavily invested in it and wants us to adopt it. We understand the reasons why they picked that tool and work with them on it but I can tell you there have been a number of instances. Very recently, one of our clients had invested heavily in a particular tool. They bought multiple licenses, but there were challenges with the tool. So my question to them was, you're using an automation tool for which you've paid significant amount, we are finding defects in the tool itself. We are working with their tech support team to get the tool and defects fixed. What are we trying to achieve here? Are we testing their product or are we testing your product?

So it is important, so again, there is no one tool to rule them all, it depends on what product you have and in the longer run, how you are going to use it? One more thought that I had was that with the way things are evolving in software engineering today, the role the tester used to have has evolved to a level that now test code is written prior to writing the code. So you are writing test cases, you are writing automated tests for your functionality that is being returned by your peer. You basically do peer programming. OK?

Cameron Philipp-Edmonds: Right.

Mukesh Sharma: So the role of testing has evolved to a level that it's happening in parallel with the development and with continuous integration which I mentioned earlier. Your developer has returned a test development of working with a pair resource now when they're checking the code, their specific functionality, that gets tested right away the moment they are checking the code. The chances of test engineers getting a build that is dead on arrival becomes very low.

Cameron Philipp-Edmonds: You spoke a little bit about working with clients and all the great work you've done. And you founded QA Infotech with a vision to provide unbiased quality assurance testings and you're growing your company into a juggernaut can you speak a little bit on that vision and what led you to founding QA infoTech?

Mukesh Sharma: Absolutely. This is definitely something that I love to speak about. When I started my career back in '97, I started as a developer. I was in software development and those were days where independent testing or independent QA companies were not there at all. I am not aware of any company that was just providing independent QA. What made me think about providing or creating a company that could provide just independent QA, is my personal experience. I was seeing a lot of issues in production that were attributable to the same person handling development as well as testing.

Nothing against the team that were working on it, they were all very smart, but like I said to your earlier question, the developer's mindset is to build something. They are only going to test whether 2+2 is coming as 4 or not but what are the different scenarios that can be tested here, 'what if' situations. Those are the things that were getting missed. I can say that they were in a way biased doing their understanding of system internals. Then when I moved to QA and I was able to convince my management about the need for independent testing in our group and we were able to see measurrable results. I thought that, and there were still some situations, I was working in United States at a company wherein they had outsourced the product development to offshore vendor. Can you imagine the offshore vendor to come back and tell you that I have prepared, or I have built your product and it is buggy. That will never happen.

Nobody would come and say I have created the product based on your requirements but say its not going to work when you'll have this situation. And if they were to say that, your obvious response to them would be, go back and fix it. That is when I'll release your payment. So there is that conflict of interest also. And that I was seeing so I said why not provide the peace of mind to companies that are getting the products developed or the code developed, have an independent team do the QA and their bread and butter is simply QA. They aren't creating the product then saying, “oh it works well in all situations”. So that is what worked very well in fact, I can tell you we do not have a sales team and we've grown from just a couple of members team back in 2003 to more than 750 people team that we have today.

Cameron Philipp-Edmonds: That's amazing. That's fantastic. You've had a lot of success in your career, what advice can you give to someone who is just starting their career or maybe has just graduated or is just transitioning into a career in software engineering and testing?

Mukesh Sharma: Well, I would just say be passionate about what you do. The opportunities are increasing by the day than in the past and specializations scope is also significant. While your course focus is on your day-to-day work or project execution, don't miss on the larger happenings in the industry. There's a lot in terms of discussions conferences, community projects, and crowdsource testing that is happening. If you're starting your career in software testing don't think that you, earlier during my days people used to say, “Oh, if you do not know programming, you can become a tester”, that is not the case. If you know programming I can guarantee that you can do much better as a software test engineer or software engineer doing testing because you understand the code, you can tell people "hey, here's what you can change". So there is, I think software engineering, specifically in testing, has more potential than anything in software engineering.

Cameron Philipp-Edmonds: OK. And I want to kind of put you on the spot here with one final question, knowing what you know now, what advice would you go back and give to a younger Mukesh?

Mukesh Sharma: Very interesting question. What I am going to tell a younger Mukesh is that, hey remember, relationships are very important. Everything that you are doing has an impact. Times are changing, technology is changing our lives significantly. While these can be a boon, they can also soon become overwhelming, especially for you because as an individual you are wanting to run and lead an organization and you already have so much on your plate. So why do you rely on technology to a large extent? Don't miss on people relationships, enjoy your time. These relationships are indispensable and whatever role discipline or role you are in, even more when you are the CEO of an organization because, people relations and their trust they have had in me and my organization have helped us grow this far. The other advice that I would give to a younger Mukesh would be you need to learn to delegate, and you need to identify the right people to delegate. That is the key.

Cameron Philipp-Edmonds: All right. Well thank you so much once again this was the always insightful, and very successful Mukesh Sharma, the CEO and found of QA InfoTech. Thank you so much for taking time out of your day to speak with us.

Mukesh Sharma: Thank you very much it was a pleasure speaking with you. I really appreciate your time and I look forward to speaking with you again.

Cameron Philipp-Edmonds: All right. And if you want to check out Mukesh's new book, it's coming out in September. We'll be talking about it in early July with him again and you can check out his writings on TechWell.com and StickyMinds.com. Thanks again Mukesh.

Mukesh Sharma: Thank you very much. Have a wonderful day.

Cameron Philipp-Edmonds: You too.

As founder and CEO of QA InfoTech Worldwide, Mukesh Sharma is responsible for the company's vision and leads the organization's worldwide operations, marketing, sales, and development efforts. Founding QA InfoTech with a vision to provide unbiased testing solutions, Mukesh has grown the organization to five Centers of Excellence with more than 650 employees. He began his technology career with DCM Data Systems, and then worked at IBM Corporation, Quark Inc., Gale Group, and Adobe Systems in software engineering and testing disciplines. Mukesh’s passion for excellence, eye for detail, and customer commitment have enabled QA InfoTech to stand apart in the exceedingly competitive software testing industry.