PC: I think it's less about how we define the term and more about how we define our approach. In broad terms, evaluation is about systematically monitoring and collecting data about the work you're doing. That's important, but what's even more important is how you go about doing that. Our approach is to not come in to an organization and say, "This is good" or "This is bad." Instead, we take a look at the whole program, we interview the stakeholders — clients, funders, the board, staff — and we try to determine how success in that particular context should be defined. Then we try to develop ways to measure the program's results that are not burdensome to the organization. Evaluation, in our view, shouldn't be something that organizations have to cut or curtail other activities to do. It's something that should be built into their operations, and it should be informative; stakeholders should get something out of it that they can use, not just a grade indicating that they passed or failed. Effective evaluation should tell them what they're doing well, what they're not doing so well, and, most importantly, what they can do to improve.

We firmly believe that the folks who are doing the work know best. And so we work to help those folks help themselves by defining what success in their particular context means, as well as what, realistically, they can do to achieve it. If you're running a job-training program for a hard-to-place population, say, men recently released from prison, and you have one hundred men who have just entered your program, to say that you're going to place all of them in good jobs in the first year is unrealistic. Instead, you start by defining what real success would look like, and from there you design a continuum, or series of steps, that leads you to your goal — a certain percentage of the men will get placed in jobs, a certain percentage will get job training but might not actually get jobs, and a certain percentage will drop out of the program. It's all about putting success in a realistic context.

PND: Innonet describes the work it does as logic-model oriented. What does that mean?

PC: Basically, a logic model is a map. It's a way to get nonprofits to think systematically about their programs, to think about their resources, and to think about the way those resources and the tangible outputs of those programs — the number of people trained, the number of people served, the number of meals served — align.

Creating a map helps a nonprofit do several things. For starters, it helps it plan its programs in a logical fashion. It also helps it think through what it's trying to measure. Is it the number of people who walked through the door, or is it how their lives were changed as a result of walking through the door? Both are important. Finally, it helps a nonprofit identify what is realistic. If I'm the executive director of a nonprofit and we have finite resources and a set of programs designed to achieve certain outcomes, is it realistic to think we can accomplish those outcomes with the resources available to us? Should we scale back our ambitions, or, given the available resources, should we be trying to do more? That's the crux of it.

PND: How do you help organizations define their outputs and outcomes?

PC: That's really the core of our practice, and we do it in two ways. First, we'll work with them to build their capacity for evaluation; and second, we can come in and do the evaluation for them. Even when we do the latter, however, we firmly believe that capacity building has to be part of the equation; we understand that an organization may not be able to conduct a comprehensive evaluation on its own the next time such an evaluation is required, but at a minimum we want it to be able to do some of the work on its own.

So, we'll start by sitting down with the executive director and the program staff and looking at what it is they want to evaluate — maybe it's the entire organization, or maybe it's just a particular program. Then we help them build a logic model. We ask them what they're trying to accomplish with the program, and about the resources they're able to dedicate to the program, and about the activities they've designed to achieve their objectives, until we have a really nice picture of the program and its desired outcomes. Then we take that information and start to design the evaluation: What do you want to know? And how can we measure that? Again, because we're stakeholder focused, we do that with the full participation of program staff, of funders, and of clients.

Now, one of the tricky things about this kind of participatory approach is how one actually gets the information needed to conduct a successful evaluation. In some communities, surveying will work; in others, it won't. In some communities, we'll rely on key informant interviews; in others, it might be focus groups. Each community is unique, and the trick is to tailor your data-gathering efforts to the community in question in a way that feels non-intrusive.

PND: Do you offer guidelines to help an organization determine which programs it should evaluate?

PC: That's a good question. People always ask us whether they should evaluate their little program or their huge, multimillion-dollar program. And our answer is always the same: You should evaluate everything, but you should apply different levels of evaluation to different things. If you've been awarded a multimillion-dollar grant to do a huge program serving thousands of people, you should spend the time and money to do a serious evaluation of that program. You want to make sure that that money is spent well; you want to make sure that the thousands of clients served by the program are receiving the right services, and that the services are having the intended results; and you want to make sure that you evaluate the implementation of the program — how it was designed and rolled out — as well as the outcomes. If you do all that, you'll definitely come away with a set of best practices you can use the next time you receive a multimillion-dollar grant.

On the other hand, if you have a tiny little program, maybe one that's not even funded and only serves a few clients, you're still going to want to know whether or not it's working. Now, it probably doesn't make sense to invest significant amounts of time and money in doing a thorough evaluation of the program, but you might want to do something simpler. It could be as simple as asking clients in the program what they think about it. You're still doing evaluation, but you're doing it at a different level. It's all a matter of degree. Sometimes we find ourselves working with organizations that are willing to dedicate resources to showing that a small program is getting results because they want to scale the program. Ultimately, it's up to stakeholders to decide how much time and money they want to spend on evaluation. But the bottom line is, you should evaluate everything.

PND: We've been talking about evaluation as a process. Are there templates that organizations can apply to their evaluation efforts, or is every nonprofit and every program sui generis, making a templated approach beside the point?

PC: More the former, I think. In fact, we provide a set of free tools on our Web site that are designed for just that purpose. Using those tools, visitors to the site can develop a logic model on their own and then use that model to develop an evaluation plan. It's all templated, because the basic tenets of evaluation tend to be the same regardless of the program or the organization: You have a set of intended results and you're trying to determine whether you achieved those results. It's really that simple.

Now, there are different ways of doing that, and that's where the participatory approach comes in. If you're evaluating a program for kids and need to interview a bunch of nine-year-olds, you're going to use an approach different from what you'd use if you were going to interview parolees recently released from prison. In both cases, you might also want to do surveys and conduct focus groups, or maybe not. The point is, the participatory approach allows the organization doing the evaluation to determine the best way to get the information it needs. So, it can be templated, but it's a templated approach subject to modifications.

PND: Is it possible to do the kind of evaluation you just described without the help of an objective third-party facilitator?

PC: Well, you know the old saying: There's no such thing as objectivity; you just learn to refine your subjectivity. But it's a good question. We learn more about evaluation all the time, and one of things I've come to realize is that there are two types of evaluation. One kind happens at the field or subfield level and is designed to learn more about the best way to serve a particular population. Typically, that kind of evaluation requires a lot of design and serious resources, and it's not what we're talking about.

We're really talking about a different kind of evaluation, evaluation that measures impact and results at the program level. It's evaluation that asks, Are you doing what you set out to do? Are you serving your clients as effectively and efficiently as possible? Are you having the kind of impact you hoped to? We've been big believers — actually, I think it's one of the most important lessons we've learned over the years — that many nonprofits are able to do this kind of evaluation themselves. It all has to do with the intensity of the evaluation and the infrastructure and systems the nonprofit has in place. If a nonprofit is in the habit of collecting information on its clients, if it has good case files and keeps good records, well then, obviously, it's in a good position to do a substantial amount of evaluation work on its own. That's our vision, at any rate. We want nonprofits and their boards to view evaluation as a tool that they can use themselves to improve their programs and processes. And what we're figuring out, as I said, is that some nonprofits are able to and some are not. Again, it really depends on the quality of the infrastructure and systems a nonprofit has in place.

PND: In the context of evaluation, is there such a thing as too much data? Is there a rule of thumb that nonprofits should apply to the data collection aspect of the evaluation process?

PC: Absolutely. We strongly believe that organizations should not collect data that they're never going to use. You really have to think through what you want to learn and limit your data collection to those areas. We've worked with clients who'll have their clients fill out a fifteen-page intake form. And when we ask them, "Why are you collecting all this information?" they'll say, "Well, it's the one time we get to really sit down with clients and get information from them." That's great, but what in the world are they going to do with all that information? So our rule of thumb is, use common sense. Collect the data you need and don't bother with stuff you're never going to use. And you'll know what that is if you develop a logic model and an evaluation plan based on that model.

PND: Have new information technologies — e-mail, the Internet, et cetera — changed the evaluation process?

PC: I think so. When I was coming up in the nonprofit world — I'm talking about the late 1980s, early '90s — we didn't have any of that. Nowadays, everyone, or almost everyone, has access to e-mail. Which means if you wanted to do a client satisfaction survey or a survey of past clients, you really could do it without having to go door-to-door. That's a significant change.

The other thing technology has done is to make evaluation resources much more widely available through Web sites like ours, through university Web sites, through various listservs, and so on. More importantly, technology has also been a great enabler of peer learning in the field. There are online forums and sites like ours where nonprofit practitioners can interact with people who have the same questions and concerns, where they can learn from other organizations about what works and what doesn't — and not just in terms of evaluation. That's probably been the greatest contribution of technology to the field.

PND: Have new technologies made nonprofits more efficient?

PC: I think so. In general, I think technology has contributed to improvements in efficiency in a lot of ways, across all sectors. Let me give you an example. We do evaluation training, a sort of Evaluation 101, where we bring people together in a room and spend a couple of days teaching them about evaluation. In the old days we had to do that locally, or we would fly people in to a central location from all over the country; now we can do it online and save everyone a lot of money. Sure, online training is different from in-person training, but it also allows you to reach many more people for the same amount of money.

PND: Has technology made the sector more effective?

PC: Technology is a great thing, but it's not a silver bullet, and it hasn't changed the way many nonprofits relate to their communities and client bases. Technology is not the answer when you are working with hard-to-reach populations and hard-to-reach communities, which many nonprofits do. You still have to do the legwork, you still have to go into those neighborhoods and do your community building from the ground up.

PND: Does Innonet evaluate its own efforts? And if so, do you use an outside evaluator?

PC: We haven't used an outside evaluator. We do run focus groups with clients we've worked with to evaluate what's working and what isn't. And we also get regular feedback from registered users of our Web site, which we incorporate into our online tools and use to refine our approach. For example, the biggest change we've made over the past couple years has to do with the recognition that nonprofits can't do it all on their own. So, while we used to focus on building the capacity of nonprofits to do evaluation, these days we're more focused on building their evaluation preparedness and getting them to a point where, when it's finally time to engage an outside consultant, they're in a position to really get something out of it.

PND: Evaluation often is associated with for-profits and the for-profit sector. Did the dot-com bust and subsequent corporate scandals at Enron, WorldCom, et cetera change the way Innonet approaches its work?

PC: That's an interesting question. As you know, during the dot-com boom of the late '90s there was a movement afoot to encourage nonprofits to operate more like for-profits — to become more efficient, more hands on, more competitive. And I think the implications of that were, a) nonprofits and the nonprofit sector were completely inefficient; and b) the private sector was totally efficient. Well, it's pretty clear that neither of those views was accurate. The example I always give goes back to my caseworker days when I used to have to juggle three hundred or four hundred clients. Obviously, I did the best I could in serving those clients, but the reason I had such a high client load was because we didn't have the resources to hire more caseworkers. Needless to say, in a situation like that, you become pretty efficient pretty quickly.

Now, where all this really comes to a head is around the issue of evaluation, which, as you suggest, means something different in the private sector and in the nonprofit sector. Measuring success in the private sector often — though not always — has to do with market share, net margins, and profit, all of which can be measured on a quarterly basis.

In the nonprofit sector, in contrast, success is about changing people's lives for the better, and that's much harder to measure, especially on a short-term basis. Take something like the civil rights movement — at what point, using objective measures, could one say it achieved success? You can't, right? That's why it's not uncommon to see people who are steeped in the ways of the private sector become frustrated when they try to apply their favorite private sector evaluation models to nonprofit activity. In the example of the job training program for parolees I mentioned earlier, we all know there's no way all one hundred of those parolees are going to get placed in a job in the first year they're out of prison. Does that mean the program is a failure? In the private sector the answer probably would be yes, but in our sector the answer isn't as clear cut.

Now, where we can learn from the private sector, and where Innonet has learned from the private sector, is in how to run an operation more efficiently, how to find and implement a billing system that enhances organizational efficiency, a personnel management systems that enhances organizational efficiency, an employee benefit systems that enhances organizational efficiency. Those are the lessons we need to learn from the private sector. But when it comes to measuring success, that's where the two sectors diverge. And that's not necessarily a bad thing.

PND: In a world of instant gratification, is there anything the nonprofit sector can or should do to inoculate itself from the what-have-you-done-for-me-lately syndrome?

PC: I don't know. People want results right away. You see it in education, for example. Parents, administrators, the media — most of all, the media — expect kids that scored fifty on a particular test to go through a one-semester program and score a hundred the next time they take the test. That's not going to happen, and we're kidding ourselves if we think it is. It comes down to over-promising, and I think we do that because we want to change the world and we're forced, by circumstance, to try to do it for $17,000, or whatever. But that's just not going to happen.

PND: What's next for Innonet?

PC: Well, we want to continue what we're doing in terms of our core practice, and that includes further development of our online evaluation tools. We're spending quite a bit of time and money in that area, and we recently re-introduced several free tools in a more user-friendly form. We're also developing a number of fairly robust subscription-based tools, including a data-analysis tool that will allow nonprofits to input data, have it analyzed, and generate reports, as well as a survey builder tool. Our nonprofit clients are always telling us, "We want to survey our clients, but we don't know what questions to ask." So we're building a searchable database of questions tested by other nonprofit organizations that people can use to create their own surveys.

On the sector level, we remain focused on getting nonprofit boards and executives to view evaluation not as a punitive thing but as something that can help them and their clients. We realize that, for too long, evaluation was used as a kind of club on nonprofit executives, and we want to help people see it in a different light, as something that can tell them what they are doing well, where they could improve, and whether or not they're having an impact. After all, isn't that why people get into nonprofit work in the first place, to have an impact? But if you don't know whether you're having an impact, what's the point? That's what it's all about for us, showing nonprofits how to use evaluation to improve their practice and demonstrate their success.

PND: Well, thanks for speaking to us, Patrick.

PC: Thank you.

Mitch Nauffts, PND's editorial director, spoke with Patrick Corvington earlier this year. For more information on the Newsmakers series, contact Mitch at mfn@fdncenter.org.