"It is inevitable that sooner or later, someone higher up the food chain is going to ask you for documentation metrics." In this article, Donald Le Vie distinguishes between "mechanical metrics" that don't really measure your productivity, and "quality metrics," which are more difficult to use, but much more accurate. Le Vie provides excellent examples of these, and analyzes the value of various types of metrics.
This article reprinted with permission from INTERCOM, the magazine of the Society for Technical Communication (STC).
Regardless of the industry in which you work, it is inevitable that sooner or later, someone higher up on the food chain is going to ask you for documentation metrics. Metrics can be defined as a system for measuring production or product standards, so naturally somebody, somewhere will want to know how "productive" the technical writers and editors can be. Or they will want to measure the quality of your work. Much of the time, the types of metrics used seem to have little correlation with the value technical communicators actually provide.
Because these measurements can be critical to your success in an organization, there is a danger in relying on standards that offer little in real value. Rather than continuing to go along with the game, technical communication professionals should exert some influence on the types of measurements applied to documentation projects.
This article focuses on metrics you should avoid—hopefully, this will give you the ammunition you need to defend yourself against managers and bean counters who want to impose arbitrary standards on your work. This article also details metrics you should embrace so that you can get moving in the right direction.
Metrics: An Overview
The theory of metrics is that some properties can be measured and that some relationship exists between what we can measure and what we want to know. At times, it can admittedly be difficult to relate what we can measure to some quality attribute(s). You can easily count apostrophes per page, for instance, but that doesn't tell you anything about the quality of the writing.
I've been involved with defining, collecting, and interpreting metrics for more than twenty years. I've seen metrics adapted for all sorts of different reasons. When I worked as a research geological oceanographer for the National Oceanic and Atmospheric Administration in 1970s, for example, choosing the proper analytic standards for research was critical to obtaining continued funding. The metrics used in the field and in the labs were great for studying sediment dispersal patterns on the continental shelf and the engineering stability of seafloor muds. But the U.S. Department of Commerce was ultimately interested in only one thing: What kind of bang were the taxpayers getting for the buck? Were the results of our efforts meaningful to the people paying the bills? We had to translate the results of our field and lab evaluations into a language taxpayers could understand: dollars, as expressed in risk assessment costs, environmental assessment costs, and contingency costs associated with the economic development of offshore areas. These became our metrics, and they measured our successes.
Starting Right
It is important to define standards for both product and process right from the start. The type of measurement to be used and the questions to be answered with respect to quality should be determined at the outset of the project. The fundamental question of "What do we really want to measure?" should be addressed in the documentation project plan, before any writing is done. From the very beginning, raise the question, "How does this metric contribute to a qualitative assessment of our information products?" If you start by asking, "What’s easy to measure?" you'll probably end up with a meaningless metric. You might as well ask, "How many angels are dancing on the head of this pin?"
Another way to put it is that you should strive to use quality metric rather than mechanical metrics. A quality metric is a metric intended to be a predictor of document quality (though the relationship between these metrics and quality has to be examined carefully). When we assign a definition of quality to some documentation metrics, we can set goals. These goals are expressed as a level of quality that must be met for a project. Metrics can also be used to measure the quality of the finished document. (JoAnn Hackos describes this method in chapter eight of Managing Your Documentation Projects, John Wiley & Sons, 1994.)
A mechanical metric is one based on something that can be measured mechanically. The number of words on a page, the number of pages in a book, the number of commas in a document—all these are mechanical metrics. The more sophisticated mechanical metrics use computer programs to do various counts and analyses of a text (I discuss some of these below). But the extra sophistication doesn't change the fact that mechanical metrics don't tell us what we need to know. Often, they cause more harm than good.
If a metric isn't a predictor of quality, then it's very likely a mechanical measure, with little or no application in this context, as the table in the sidebar illustrates.
Metrics of Questionable Value
Mechanical metrics are based on concrete measurements, which actually makes them very trendy. The computer industry thrives on statistics and forecasts, and this finds its way into other areas of study.
The problem with mechanical metrics is that to arrive at a viable metrics "matrix," we'd have to have an n-dimensional chart or graph that accounts for all the factors and variables that exert some influence on the writer’s productivity. Although we can benefit from measuring our processes by assigning valuation in a comparative schema, we have to be careful about approaching user documents in this way. We aren't in the best position to make that determination: The customer is.
Following are some common mechanical metrics and the problems I see with them.
Page Count
The page-count metric is used to estimate the scope of a project. It's an internal measure—it's useful for internal planning purposes. By comparing completed projects of similar scope, we can determine the level of effort that most likely will be required to complete the project at hand. We can determine how to allocate resources and schedule production and then use that information to plan for larger projects.
However, the number of pages has no intrinsic value. You cannot draw a correlation between a 500-page document or a 50-page document and their inherent communication or knowledge-transfer quality.
Readability
The three major readability metrics used in communication today are the Gunning Fog Index score (grade level units), the Flesch-Kincaid formula values (grade level units), and the Flesch Reading score (with 100 representing the greatest ease of readability and 0 representing the least). All three are generated computationally—a computer program examines the copy and assigns a score.
The problem with readability metrics in technical documentation is validity, as the technical backgrounds of a target audience will determine the baseline of our measurement. For example, I helped write a software architectural specification overview for one of our product lines, and I edited the entire document. The Flesch Reading score for the specification was 17, indicating a difficult-toread document; however, the document was praised by the executive management team and the board of directors as well written and easy to understand.
In this example, the executive management team and the board of directors constituted the audience. It was their assessment that mattered, not the arbitrary numbers on the reading scale.
Pages per Day
This metric is also sometimes called the "documentation rate." Management may like it because it's straightforward, but what is its real intrinsic value?
The number of pages that Writer X produced last month depends on many factors, including the writer's workload for that month. What does that have to do with the current project? You'd have to track many variables to see if the writer was truly being as productive, on a page-by-page basis, as he or she was last month. (Tracking these variables would be as much work as writing the manuals!) But more to the point, measuring documentation rates can lead people to believe that "more pages = better documentation." We know that’s not true; isn't it time we dropped metrics that support this idea?
Software Dependent Metrics
If your publications team reports to engineering or development, management may have a tendency to relate documentation standards to software development (X lines of code per day = Z pages of documentation per day). But remember that you are writing for the user audience, not the developers. Why should the users care about the ratio of X to Z? They don't. They'll judge your product on how easy it is to use, and nothing more.
Metrics That Add Value
Again, a quality metric is a metric intended to be a measure of document quality. You can't really make this measurement in the abstract—you need to consult with the user. Whether you do it before release of the product (through controlled usability testing) or after release (through customer feedback), you need to hear from the people who are using the product. They're the final arbiters of quality.
Below are some quality metrics I believe have value.
Web-based Feedback
Develop a Web-based documentation feedback utility or survey that offers some reward (a drawing for a palm-held computer, for example) in exchange for the user's suggestions on a technical information product. When designing Webbased feedback forms, be sure to ask for information that will help you make accurate measurements. Don't limit the feedback to "1-to-5" responses, where a 1 represents "unacceptable” and a 5 represents "exceeds expectations." Allow users to comment on the information they need (or don’t need) in future revisions of your product. Then publish those results throughout executive levels on a monthly basis. Provide cumulative and current month/previous month evaluations of the responses you receive. It's a great way to elevate the visibility of the technical publications professionals.
Usability Testing/Prerelease Feedback
Conduct some small-scale usability testing of beta versions or final drafts of user documentation with customers and use that feedback to create a polished information product that will effectively transfer knowledge to customers. A clearly defined rating scale in a documentation feedback survey of this sort has more value to the organization than measures such as "number of pages/day" that are irrelevant to the customer.
Customer Support Improvements
Team up with customer support to design a response form (or add fields to existing forms) to determine how many calls are related to documentation, and give that information to the documentation team so it can quickly update online publications. Track this information over time to show the decrease in the number of documentation-related calls so you can assign a dollar figure (a quantitative measure) to any metric. Incorporate nondocumentation customer issues and resolutions into subsequent information products. Broadcast your results throughout management just as you did for the Web-based feedback.
Fallout from Bad Metrics
Dangerous Misperceptions
Mechanical metrics contribute to the idea that "anyone can write documentation." And they can, if all that matters is conforming to some arbitrary metric that has no relationship to quality.
Downsizing Ammunition
Managers often seek mechanical metrics to simplify their accounting. However, when it comes time to trim "overhead services" (as documentation and publication groups are often called), these mechanical metrics make technical communicators easy targets for the budget axe. This can and does happen in many organizations, as I've experienced.
The Bottom Line
We can produce user documentation that is accurate, complete, organized, and clear and that contains sufficient entry points (table of contents, index) and visuals, but still does not meet the needs of the target audience. This leaves us with two firm conclusions:
- It is the feedback (and input) from the target audience that should matter the most when it comes to implementing any documentation metrics
- Documentation metrics must be tied to the bottom line and broadcast to management if they are to have any meaning outside the documentation department.
I have never heard a customer complain about a misspelled word or a few misplaced modifiers. However, I have had customers pitch fits about ambiguous, incomplete, or incorrect content. Our documentation metrics should reflect the realities we experience in our exchanges with the customer.
Imagine having a technical publications operation that works in close cooperation with customer support to track and correct the deficiencies reported by customers and internal users. Our information products would ultimately make customer support unnecessary because of the quality of the product and information design...What halcyon days those would be.
