e-Talk Radio: Paulk, Mark, 28 November 2000

TEXT TRANSCRIPT: 28 November 2000
Copyright 2001 Quality Plus Technologies and Carol Dekkers. All rights reserved.

Welcome to Quality Plus! E-talk with Carol Dekkers. This program will focus on the latest in the field of technology. All comments, views and opinions are those of the host, guests, and callers. Now let's join Carol Dekkers.

Carol: Welcome to Quality Plus! E-talk this week. I'm Carol Dekkers. And I run a company called Quality Plus Technologies, which is a management consulting firm where we specialize in using measurement to help companies to build better software. I've had a number of guests over the last several weeks. And we've talked about everything from technology to do with the Florida election, to do with the Palm Beach County. We've talked a little bit last week with Ed Yourdon about extreme programming and the capability maturity model. And every week it seems to build. We seem to get better and better guests.

I feel very, very privileged this week to be able to introduce you to one of the true greats in the software industry. I'd like to introduce you now to Mark Paulk, who is a senior member of the technical staff at the Software Engineering Institute at Carnegie-Mellon University. He was the project leader during the development of the software capability maturity model Version 1.1. And he's also actively involved with software engineering standards, including ISO 15504, which is a software lifecycle process; no, I just messed that up. It's the framework, software process framework, the ISO 12207 standard, which is the software lifecycle processes standard, and the ISO 15288. Prior to joining the Software Engineering Institute, Mark was a senior systems analyst for Systems Incorporation at the Ballistic Missile Defense Advanced Research Center in Huntsville, Alabama. He has degrees in mathematics and computer science from the University of Alabama in Huntsville and Vanderbilt University. Mark is a senior member of the IEEE and a senior member of the American Society for Quality. He's an ASQ certified software quality engineer and an SEI lead assessor. And one of the best things that I like about Mark is that he's, underneath all of this pomp and ceremony and fame, he's a real down-to-earth person. So I'd like to welcome you to the show, Mark.

Mark: Okay, thank you very much for those kind words. It makes me sound a whole lot better than I am in all truth, but I'll take that with a certain degree of salt, I guess.

Carol: That's a high maturity person that it takes to say that.

Mark: Thank you very much.

Carol: To get started, tonight our topic is high maturity processes in software. And to get started, Mark, we've had different people talk over the last few weeks about the CMM and that acronym. And if you could just spend a couple of minutes talking to our listening audience who may be new to the show and just tell them a little bit about what is the CMM. What are we talking about, about high maturity processes?

Mark: Okay. The CMM, the capability maturity model for software was actually inspired by some of the work that Phil Crosby did with his maturity grid and his work with zero defect products and the total quality management movement several years ago. Watts Humphrey is the person who initially articulated the software process maturity framework back in 1987. And what he observed was that there were some common patterns, if you will, in terms of how software organizations and software projects got into trouble. And so he applied some of the ideas that Crosby had expressed in his book, Quality, a five-level model. And you kind of go through a sequence of questions in terms of what the different issues are at each of the levels. And kind of the first question that Watts asked was: Is project management a good thing? And when he was talking to executives and senior managers, as you might expect, the answer would be: Well, yes, management is a good thing. Secondly, is organizational learning, a standardization, a good thing? And as you might expect, the answer would typically be yes. Is measurement and management by fact a good thing? Well of course, obviously. Is continual improvement a good thing? Of course, you know, this is trivially obvious, easy to answer kind of stuff.

Carol: Right.

Mark: And then of course the ending question is: Well, why aren't you doing any of those things? And so the five-level model basically focuses on project management issues; on creating standards and organizational learning through common processes and measures and training and then using the measurement to drive the decision-making processes; and, lastly, to continue the journey of continual process improvement and quality management. And that's the five-level framework which we refined quite a bit over the years since Watts initially expressed that five-level framework. Carol: And it's interesting. Because ever since it was first, I guess, now it's the first published, most of the companies that went through an assessment in the first place were very low maturity companies.

Mark: Yes that's certainly true. Back in 1989 and 1990 when we were doing the first reports on the maturity of the organizations that had been assessed using the model, we were at that time running about 90 percent of the organizations as Level 1 organizations. Which meant that they had very fundamental problems in terms of having a common understanding with a customer about what was to be built, in terms of planning, in terms of tracking what was going on; in some cases, actually losing the source code, not knowing what the releases were. They were out in the field in terms of problem reports that might come in. You could ask yourself: Well, what version is that against? And nobody would be able to reproduce that version. So there were some pretty fundamental issues that has changed over the years. It is certainly a biased sample, because we talk primarily with organizations that are doing assessments. And so that's not a random sample of the software community at large.

Carol: Right.

Mark: But we've gone from roughly 90 percent being at Level 1 in 1990, to less than half of the organizations being assessed today being at Level 1 and a pretty fair percentage at levels 2 and 3 and a sprinkling of a little bit less than five percent at Levels 4 and 5, the top levels of the model. Carol: And conceptually, when you talk about high maturity, what does that really mean to the Software Engineering Institute?

Mark: Well, what that really means is that you have a process, a way of building software that you have standardized as appropriate for various environments that you may be building products for, where you can measure what's going on in your process, use some statistical analysis, quantitative management techniques so that when unusual things happen, you can identify that something odd happened and take corrective or preventative action as appropriate, that you can make decisions based upon a consistent use of measurement data. And that's a real culture shift for most software organizations.

Back in 1991 there was a guide, ISO 9000, Part 3, that was written for interpreting ISO 9001, which is the international standard for quality management systems. ISO 9001 is used in a number of different environments, manufacturing, services, et cetera. And it has a section, a clause, on statistical techniques. The guide for interpreting that in the software arena had the section entitled "Metrics."

Carol: Right. Mark: So instead of statistical techniques, we'll just talk about measuring something. And then in the text it talked about the fact that in the software field, at least at that time, we don't really know how to measure quality. And so if you measure something, if you measure anything, you will in essence satisfy this particular clause of ISO 9001. And in fact that was a, probably a reasonable way of capturing the state of the practice in measurement back in the 1880s, 1980s, excuse me, time frame. And so we have as a community, grown in our use of measurement thanks to folks such as yourself, Dick. Norm Fenton and a whole bunch of other measurement gurus around the world who have given us better and better insights about how to do measurement effectively. But it's been a slow process. And it's been growing over the years.

Carol: And the sampling that you took in 1990 with 90 percent Level 1, have some of those people dropped off? Has your sample size increased? Today's sample, when you say that we've got more than 50 percent that have moved beyond kind of utter chaos, is that still the same samples you had back then, or have we had people added on and people dropped off? What are the demographics? Have you seen any changes in the sample size?

Mark: Well that certainly has been one of the fascinating things about how the process movement in the software world has grown over the past decade or so. In 1990 when we were talking about process improvement, we were fundamentally talking to government contractors, to DoD contractors. The SEI is a federally funded research and development center. We're sponsored out of the U.S. Department of Defense. Our sponsor is the Under Secretary of Defense for Acquisition Technology and Logistics, so we have a primary focus on the military and the military contracting communities. And so the initial adopters of our ideas were in essence the DoD contracting community. And back at that time, I don't recall the exact numbers, but there were probably about a hundred organizations represented in the first set of data points that we looked at back in 1990 that were 90 percent Level 1. Today when we look at it, we're talking at an order of magnitude, more organizations. Some time ago, about three or four years ago, the percentage of DoD and DoD contracting organizations dropped below 50 percent of the organizations reporting data to the SEI.

Carol: Right.

Mark: The majority are now banks, insurance agencies, aerospace companies, telecommunications, companies that are building software intensive systems. And those are the ones who are picking up the ideas that we captured in the CMM and has moved far beyond the DoD contracting community.

Back in 1990 time frame, when we had a workshop for folks interested in software process improvement, we got at our first one about 30 to 40 folks come into that workshop. There were about 50 or 60 at the next one. There were about 80 at the third one. At the fourth one, there were 411 people…

Carol: Wow. Mark: …attending that. And that was the point we decided it wasn't a workshop anymore. It was a conference. And that conference now has over 2,000 people attending it on an annual basis. And there are conferences in Europe, in India and Australia. And I just heard of one that's being started up on the Pacific rim in Hong Kong, which are annual conferences for folks who are interested in software process improvement issues. So the pickup has been fairly dramatic.

We still get only a small percentage of the software community as a whole, I would have to admit. But I think that's true of any software engineering technology that you can think of that, you know, where you're still struggling to become an engineering discipline.

Carol: Right. Are you kind of amazed? Did you have any idea when you were starting off on this project, how big it would become?

Mark: Absolutely none whatsoever. I don't know if I would have been willing to work on it. I would have been too intimidated to work on it if I had realized how much of an impact we were going to have on the software community back in the 1987 time frame when I joined the SEI. So it has, it's definitely been an interesting ride over the last 13 years or so as we have changed the software world in many significant ways.

Carol: And we'll be back in a few moments after a few words from our sponsors with more of Mark Paulk and talking about the capability maturity model in software. Welcome back to Quality Plus! E-talk. This week I'm talking to my guest, Mark Paulk, who is a senior member of the technical staff of the Software Engineering Institute at Carnegie-Mellon University. Mark has been involved in the Capability Maturity Model for software for quite a number of years. And we've been talking about conceptually what does high maturity mean to the Software Engineering Institute. And I have a question for you, Mark. How does a company go from being chaotic kind of a Level 1 organization to a Level 2? Isn't there a huge investment in time and money, and how would a company even get started from this type (inaudible)

Mark: Well in a very real sense, you're right. It is an enormous investment, certainly in time and effort, perhaps to some degree in money. But the thing that we ask folks who were considering whether or not to do process improvement is whether or not they are satisfied with the status quo. If you like the way that things are working in your software organization today, then the likelihood of change is pretty dim. So the very first thing that we want to probe into is whether or not there is sufficient dissatisfaction with the way things currently are so that you're willing to actually change your behavior. And if you're, you know, pretty satisfied with the way that things are, nothing is going to change.

If you are dissatisfied and you think that you need some better performance, for example, there have been some studies done by the Standish Group that said that the typical software project is over a hundred percent over budget, it is significantly late, it is only going to deliver about 42 percent of the functionality originally promised. And some managers, some customers find that, what we would say problematic. That's why the term software crisis is bandied about sometimes.

Carol: Okay. Mark: So if you're, you know, dissatisfied with that status quo, then what can you do to make things better? And what we observed in looking at problem projects was that the problems typically weren't technical problems in the sense of, you know, we tried to try some leading edge technology, and it just was not successful. They were real fundamental planning and management kinds of issues. And when you look at the CMM, there's a set of six things that are the fundamental attributes of the Level 2 process capability.

The first one is requirements management. Do you have a common understanding with your customer of what the software product is going to be? Project planning, do you have a realistic plan? Are you making realistic commitments in terms of how long it's going to take you, how much it's going to cost, what the functionality is that you're going to deliver?

Third, can you track what your progress is in actually delivering that functionality on that schedule to that cost? Four, do you have an ability to manage your subcontractors? A lot of organizations that do subcontracting will be critically dependent upon the capability that's being supplied by their subcontractor, so you have to manage that effectively. Otherwise, that risk can prevent you from being successful on your project.

Carol: Okay.

Mark: Then fifth, the idea of software quality assurance. Are you doing what you said you were going to do? So do you have the process assurance and product assurance mechanisms in place that will objectively verify that we're doing what we said we were going to do so that you don't delude yourself in essence. And last, configuration management. The idea that we know what our product is, they we're controlling changes to it in a, you know, a reasonably systematic fashion. And if I may make a side comment here, we sometimes talk about the bottom-dwelling, mud-sucking Level 1 organizations of the world, somewhat facetiously. And one of the things that I use as a sign post, if you will, of the real bottom Level 1 organizations are the ones that have lost the source code, don't know where their product is anymore. And I was amused at one of the banks that was auditing their code in preparation for Y2K, reported that they had found 97 percent of their source code. They were only missing three million source lines of code that were either going to have to be rewritten or re-engineered or somehow recaptured in order to prepare for the Y2K issue.

So, you know, these are all very fundamental basic management kinds of capability. And if you have major problems in any of those areas, there are going to be some pretty visible and pretty significant consequences in terms of being able to meet your commitments.

Carol: Now the source code that went missing, I find that quite interesting, because some of our listeners may say: Well, you know, what can happen to source codes? And was it as simple as people leaving and not essentially having it in their libraries themselves their.  What types of things can happen? Because I don't think people go into work purposely to do a bad job. I think that people go in and do the best job that they know how to do, generally. But maybe they just don't know some of these fundamental things.

Mark: Absolutely. And in fact, one of the things that we say is that people do the best work that they can in the system that they have to work within. And when I was getting out of school with a bachelor's degree and a Master's Degree in Computer Science and Mathematics, I had a degree in Computer Science, and I had never heard the phrase configuration management and that... Carol: And we will be back to hear what you thought it meant as soon as we hear a few messages from our sponsors... Welcome back. If you're just joining us, you're in for a treat. We're talking to Mark Paulk, who's a senior member of the technical staff at the Software Engineering Institute at Carnegie-Mellon University, this week. And we've been talking about high maturity processes. What does it take to make software using a standardized process? And just before we went into the break, Mark, you mentioned that when you were in university, you had never heard the phrase "configuration management." And we cut you off right before we went into break. So I'd like you to finish that thought for us.

Mark: Well, there are a lot of things that I never heard of when I was getting my degree in computer science. I never heard of configuration management or quality assurance. Those were all terms that just never crossed my horizon.

Once you get out into industry and start building real software products, all of a sudden some of the things start popping up that you never think about as a student. And so, for example, configuration management, I kind of accidentally fell into configuration management, because in the days when I was writing software, we didn't have our own private workstations. We worked on super minicomputers and things like that, the vax in particular. And I was editing a file, and I exited from it and compiled it and linked it and ran the program. And none of the things that I just put in were showing up in the way that the program was executing. And so I went and looked at the source code again, and none of the changes that I had spent several hours putting in were in the source file. And it turned out that someone else, one of my colleagues, was editing the same file at the same time as I was. And so when we exited, it turned out he exited a couple of minutes after I did and his version of the software, in essence, overwrote my version of the software. So I had several hours of work that was lost because, you know, we hadn't coordinated our use of that file. That was kind of my introduction to the need for change control and version control. And I started learning about things like configuration management. And we, from a technical perspective, kind of reinvented a lot of that before I ever realized that there was actually a field called software configuration management. And it is possible for some fairly sizable companies to actually lose their source code.

In Newsweek a few weeks ago, there was a brief little article about Pixar when they were getting ready to do the Toy Story II DVD release. They were getting ready to put the movie on the CD, and they realized that they couldn't find about 17 percent of the digital animation files for the movie.

Carol: Wow.

Mark: This is not good news. And after they scurried around looking on various people's workstations and things like that, they found most of the missing animation. But there was about one percent of the movie, according to the article, that they had to reproduce from scratch. And so if you actually lose the master files or something, that is millions and millions of dollars in profit expected to come from it, that's a very nerve-racking experience. I would guess that they have put some new configuration management procedures in that company so that they have the base lines and they have the mark out of the way, and you don't lose the golden egg, if you will. And that kind of thing happens all too often in organizations that don't have a good configuration management system in place. Carol: And it's hard enough to write software in the first place that you don't want to have to go and rewrite it. In fact, based on the fact that it's gone missing or isn't the whole thing about the CMM applying a discipline, forcing yourself to, I guess, to become organized?

Mark: Absolutely. And that's the exact point that we make when we are training people on the CMM, that it's basically just common sense stuff. And whenever we're talking to folks about the Capability Maturity Model, one of the things that we wind up saying is, this is just common sense. If there's anything that you are doing that you think is not a good common sense thing to do as the result of doing process improvement, then it would probably be a good idea to stop doing that and think about what you should do in order to do reasonable things, you know. And so if schedules and cost and budgets are important things in your environment, then you need to think about requirements, management and planning, and tracking and those fundamental management disciplines. And it's not that we don't know the right things to do, it's that we're so busy scurrying around fighting fires that we don't take the time, if you will, to do the right things that we know how to do.

Carol: And it's sometimes that when you get into those crisis modes, if you don't have a plan of attack while it's not an attack in the first place, it can go awry very quickly. And having hid, well, we all know how plans sometimes go awry and even more so in software. Once the company has, you know, the requirements and the planning and the basic tracking and the ability to manage the subcontractor, software quality assurance, fundamental configuration management in place, in that Level 2, where do they go from there? What does that mean to be Level 3 or Level 4 or Level 5? Mark: Well Level 3 is where we really start getting to the point where we can start talking about being a high maturity organization. Less than 20 percent of the organizations are at Level 3. Less than five percent are at Levels 4 and 5. So you're talking about the top quartile when you get into Levels 3, 4, and 5.

When we talk about Level 3, what we are fundamentally talking about is putting a, if you will, a learning system in place, an organizational learning system. So that when we learn something, we have the mechanisms in place that other projects can learn from what we have done without having to go through the same trial and error kind of experience, that we don't lose the knowledge that we have acquired sometimes quite painfully. And so it deals with you know defining common processes that are used on multiple projects in consistent ways within the training in place so that people have the kind of tools that they need to do their job in terms of both training and software tools and infrastructure. It's putting the common measures in place so that when we measure things, we know what the comparability is so that we can compare apples to apples and oranges to oranges and know what we're really talking about. And that is fundamentally what Level 3 is all about.

When we get up to Levels 4 and 5, we're really talking about getting a degree of process consistency and measurement in place so that we can do true management by fact. We can look at our data and get a good statistical, if you will, insight into what's going on, applying some of the ideas of statistical process control and other rigorous measurement techniques. And this is stuff that has been done in the manufacturing environment and other industries for decades, you know, going all the way back to the 1920s. And what we gradually learned over the last -- mostly over the last ten years, is how we can get a degree of process consistency that balances the creative nature of the problem-solving work, the design intensive kinds of work that we do when we're building software with the rigor and discipline that allows us to use measurement in some fairly disciplined, systematic ways. And it's really a paradigm shift. It's really a culture shift that we're talking about. And when I'm talking to folks in Level 1 organizations, I basically don't talk to them a whole lot about Levels 4 and 5. I tell them: You've got a major culture shift to go through in getting to Levels 2 and 3. And I can tell you what the advantages are, the benefits are, and hopefully make a plausible case for it that you'll understand. But when I start talking about Levels 4 and 5, we're talking about two culture shifts away, if you will.

Carol: Right.

Mark: And you're just never going to really understand that if you're in a Level 1 environment fighting fires all the time. When we start talking about some of these other ideas, it's a culture shift.

Carol: And we'll be back with more of Mark Paulk and the capability maturity model after these messages...And welcome back to Quality Plus! E-talk. I'm Carol Dekkers. And we're going into one of our final segments with our guest this evening, who is Mark Paulk, a senior member of the technical staff at the Software Engineering Institute.

Now, Mark, we've been talking about the Capability Maturity Model and why processes and standardizations can lead to better software. And I can just hear in the back of my mind a lot of people saying: Well you're out of touch with the real world. When we're building an F16 and all the software with it, we're only doing it once where artists say, you know, the capability maturity model must be for those companies like Microsoft that are just stamping out the same thing over and over again. We're artists. How can we possibly standardize? What's your answer to them?

Mark: Everybody always says "We're different". It's almost a universal comment that people make. And you will sometimes, you'll hear people saying that: You know, I'm different from everybody else in the world. You know, when Michael Fagan talks about the optimal rate for inspecting software code, I'm different. I can inspect mine in order utter of magnitude quicker than Fagan recommends. And then you start going through a process improvement, things such as the personal software process training or something like that. And what you find out is, when I start measuring my actual performance, I'm just like everybody else. In other words, the good engineering techniques, the good management techniques that work for F16 or Microsoft or anybody else are the same kinds of things that have been done over and over again. And when I refuse to take advantage of those ideas, I'm hurting myself.

There are differences. That's certainly true. But good engineering and good management practices are good engineering and management practices. And a lot of the things that you've seen, extreme programming and lightweight methodologies and things of that nature, are in many ways repackaging of ideas that have been presented as good engineering and management practices many times before over the years. Carol: Right. And I've heard that an excuse for poor management is not measuring, that if you can avoid measurement, then you can hide behind poor management practices.

Mark: There probably is some truth to that, yes.

Carol: Now one of the things that we had talked about is the fact that there are some things that the Software Engineering Institute has learned or that you've learned about high maturity organizations that aren't really captured in the current version of software capability maturity model. Would you just expand on that a little bit?

Mark: Well there certainly are, in particular at the higher levels. When we published Version 1.1 in 1993, there were a number of things about Levels 4 and 5 that we had more of a theoretical understanding of than a practical understanding of. We had a very limited set of organizations that we were using as our models in terms of the best practices, if you will, for Levels 4 and 5. And in the years since then, we've seen a growing number of organizations. There are about 85 Levels 4 and 5 organizations that we know about now. And we found out things that work. And one of the things that we see at Level 4, which is not captured in Version 1.1, is that it's not just process knowledge that you want to capture in a systematic way, it's also product knowledge. And so we see the Level 4 organizations doing systematic reviews. We see them doing domain specific software architectures. We see them establishing product lines and product families. We see them putting mechanisms in place that help them to capture product knowledge just like process knowledge. And that gives them a distinct competitive advantage in terms of understanding the work they're doing and in terms of understanding how to analyze and control their processes more effectively. We also see a number of things in terms of how you implement the Level 4 and 5 practices efficiently and correctly and in particular controlling your processes at more of a process element level, keeping a balance at the systems level with the day-to-day process management issues, which is a balance between quantitative process management and software quality management more at the systems level. Those are all things that we have learned how to articulate better now. And hopefully as time goes by, we'll get that message out to folks through various training classes that we offer and through the CMM integration work that's going on now.

Carol: And you're doing a lot of work. There's so much work going on. And I can hear companies kind of saying: Well why would I bother? If it costs me money to go through an assessment, if it costs me money to put in these requirements and these plans, what's the benefit is the bottom line. What do I get out of this? Does it save money to do things right?

Mark: Well that's always the classic question. Does quality cost? And certainly in the TQM field and in the software field, the organizations that have stayed the course and have collected the data and published it have reported that, usually, there's a savings or a return on investment of somewhere between four to one and eight to one. So for every dollar that you spend in process improvement, you save $4 to $8. When you invest in better preventive and appraisal processes, you wind up saving in terms of reworking it. It actually shortens your overall development time. As Jerry Weinberg is famous for saying: You can satisfy any requirements so long as the software doesn't have to work. Carol: Right. And we will be back to close off with Mark Paulk after these short messages. And welcome back. We've been talking to Mark Paulk, who is a senior member of the technical staff at the Software Engineering Institute. And one thing I'd just like to finish off with, Mark, if you don't mind, is that Ed Yourdon last week said only about 15 percent of American companies have gone through any sort of assessment to find out whether or not they're even doing anything right. Does the Software Engineering Institute have any plans to address that 85 percent of organizations that aren't aware yet?

Mark: Well, one of our challenges is just keeping up with the workload that we already have. And in fact several years ago, we had to put together transition partners, establish a, if you will, a consulting community to provide services to the organizations who wanted to take advantage of the work that we were doing, because we're actually a fairly small organization. And there's no way that we could supply the services that the community was requesting at that time.

I find it plausible that 15 percent is a reasonably correct percentage. What I find amazing is just how quickly and how thoroughly the software community has picked up on the ideas. So in one sense, we still have a lot more to do, and we're continuing to expand what we're doing. In another, it is amazing how well known and how widespread the CMM work is. So is the glass half full or half empty is the question to ask, I guess. Carol: And I think it's remarkable that in a short 10 years to have pervaded 15 percent of American companies. You know, 15 percent of American companies probably don't even track their site, that type of thing. I'd like to thank you, Mark, for spending this last hour with us and educating our listening audience and just sharing your expertise with us. Would you like to give out the Web site of the Software Engineering Institute to anyone that's interested?

Mark: Yes. If anyone is interested in finding out more about our work, www.sei.cmu.edu, which is the worldwide Web site for the Software Engineering Institute at Carnegie-Mellon University. On the educational subnet for those of you who are interested in the lithography there, you can find out about our work there. Look for CMM on the search engine and you'll find out where all of our stuff is.

Carol: And this has been a wonderful educational hour for me. I really enjoyed talking to you. And I'd like to thank you again, Mark, for being part of our show.

Mark: Always a pleasure.

Carol: And next week I hope that you'll join us as we talk to somebody that Mark was talking or alluded to, the author of Extreme Programming, Kent Beck, who will be joining us to talk about the lightweight methodologies, the up and coming extreme programming. Does it conjure up ideas in your mind of bungee jumpers with laptops? Who knows? We'll be talking to Kent Beck, and I'd like you to join us. If you've got any feedback that you'd like to give us about future shows or future guests, please send them to Dekkers, D-e-k-k-e-r-s at qualityplustech.com. And, Mark, would you like to say a final sendoff, five seconds. What would you like to leave our audience with?

Mark: Well the bottom line is to always examine what you're doing and try to do it better. As long as you do that, you should work out okay.

Carol: And that's a great sendoff and final words from our guest. And I'd like to say goodnight to everyone that's listening. And for more Quality Plus! E-talk next week, please join us with Kent Beck. And we're out, have a great week.

Copyright 2001 Quality Plus Technologies and Carol Dekkers. All rights reserved.