Monitoring, Evaluation and Learning Systems

Greetings! I’m Pat Mueller, President of Evergreen Evaluation & Consulting, Inc. of Jericho, Vermont. EEC’s business focuses on evaluation of federally funded Special Education programs. You’ll hear more about the rebranding of this niche firm later in the week.

The purpose of this week’s blogs is to share some thoughts, resources and musings on the marketing and branding aspects of promoting your evaluation business. Do you know your brand? If your brand was a person, what kind of person would it be? How would you market your brand for the digital age?

For some of us, marketing comes naturally and is an enjoyable part of owning and advancing our businesses. For others, it can be an after thought or “the last thing on the to-do list” that never quite gets done, or if it does, the results may not be particularly satisfying…how many times has the web site been redesigned?! Do I really use business cards? We hope this week’s blogs stimulate your thinking about creative and new ways to promote your consultancy.

This week’s posts will feature content on authentic branding, rebranding, positioning yourself and your business for success, determining the size of your consultancy, and the implications of niche businesses on branding.

Rad Resources:  In preparation for the week, we suggest that you preview the web sites for this week’s bloggers as they illustrate the range of marketing and branding approaches employed by the featured consultants. One of our bloggers does not have a digital presence, as you will read later in the week.

www.evergreenevaluation.net

www.ritafierro.com

www.acetinc.com

www.barringtonresearchgrp.com

To read more about building your consultancy, marketing and branding, check out these resources:

• Entrepreneur Magazine
  
• Gail Barrington’s recent book:  Consulting Start-Up and Management: A Guide for Evaluators and Applied Researchers

The American Evaluation Association is celebrating Promoting Your Consultancy Week with information on marketing and branding. The contributions all this week to aea365 come from our colleagues who own evaluation businesses. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No related posts.

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor.

Hot Tip – Collaboration with BetterEvaluation: Today, we’re excited to announce a partnership with BetterEvaluation. BetterEvaluation is an international collaboration to improve evaluation practice and theory by sharing information about options (methods or tools) and approaches.

Rad Resource – BetterEvaluation Rainbow Framework: BetterEvaluation has produced a framework that organizes over 200 evaluation options into 7 clusters of evaluation tasks that can help you to plan and manage an evaluation. On their website at http://betterevaluation.org/ you’ll find the framework itself as well as extensive explanation and examples in support of the framework tasks. This resource brings together contributions from evaluators working on the ground in various contexts around the world.

Hot Tip – Series of eight free short-form webinar trainings: BetterEvaluation and the American Evaluation Association are teaming up to bring to you a series of eight Coffee Break Webinars in May. This series is open to the public (please help us to spread the word), registration is free, and the speakers represent deep expertise applicable in both domestic and international contexts. The series of eight webinars walks you through the components of the Rainbow Framework and will include takeaways immediately applicable to your practice:

1. Overview of Rainbow Framework for Evaluation – Irene Guijt
2. Define What Is To Be Evaluated – Simon Hearn
3. Frame the Boundaries of the Evaluation – Patricia Rogers
4. Describe Activities, Results and Context – Irene Guijt
5. Understand Causes of Outcomes and Impacts – Jane Davidson
6. Synthesise Data from One or More Evaluations – Patricia Rogers
7. Report and Support Use of Findings – Simon Hearn
8. Manage an Evaluation – Kerry Bruce

Pre-registration is required and you can register for as many as you would like here: http://comm.eval.org/coffee_break_webinars/CoffeeBreak/BetterEvalSeries

Rad Resource: In 2012, AEA co-hosted a coffee break webinar series with Catholic Relief Services, USAID, and the American Red Cross. This collaboration resulted in a series of four public coffee break offerings, and anyone – AEA member or not – may view the recordings from all four online here. Included in this set are:

• Monitoring and Evaluation Planning for Projects/Programs
• Evaluation Jitters Part I and Part II
• Simple Measurement of Indicators and Learning for Evidence-Based Reporting

We hope to work further with this wonderful team in the future, and currently Scott Chaplowe who helped to spearhead this series is offering an AEA eStudy.

Rad Resource: AEA hosts a weekly Coffee Break Webinar series for members only. AEA members are welcome to attend any of the Thursday afternoon offerings live throughout the year (see list of upcoming offerings here), or to access recordings of over 100 coffee break webinars via the webinars archive (see the public list of what’s in the archive here). If you aren’t currently a member, we encourage you to join!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Corinne Ranney-Philbrick on Free Webinars for Professional Development
2. Susan Kistler on Free Coffee Break Webinar Series for M&E
3. Stephanie Evergreen on the Monitoring and Evaluation Coffee Break Webinar Series

My name is Soledad Muñiz and I’m the Head of PV M&E at InsightShare. Susan has kindly introduced our work and methodology in this post some weeks ago. And as she anticipated, I’m here today to share some of our experiences using Participatory Video for Monitoring & Evaluation (PV M&E).

Lessons Learned – how we’re using participatory video:

In the last 5 years, we have been developing Participatory Video for Monitoring & Evaluation with a broad range of partners. We’ve seen how Participatory Video allows for organisations to gather qualitative data that often escapes traditional monitoring and evaluation tools. It can monitor the project lifecycle over time and space through interviews, on-site visual monitoring and most significant change stories (MSC).

Hot Tips – integrating participatory video in evaluation:

It is not always easy to gauge and communicate what significance a programme or activity had in the lives of those who intended to help. Those best positioned to explore and convey these messages are those invidividuals – the main actors of development themselves – they can speak first-hand about impacts and outcomes. They can select relevant individuals to interview in their communities and monitor relevant key events as they happen. All actors can reflect back on changes in the community through screenings, where stakeholders are brought together to reflect and discuss.

This video offers a summary of the PV & MSC process in one of our latest initiatives.

A Short Documentary on PVMSC from InsightShare on Vimeo.

Lessons Learned – when and when not to use video:

Our methods help stakeholders tell their stories and communicate their perspectives in an accessible, compelling and versatile format through a participatory and authentic process. Following stringent informed consent procedures, these stories can then be used to communicate lessons or new ideas across to new groups, other organisations or decision makers. It is an overt process, so it’s important to make sure there is an in depth informed consent process through which participants fully understand the implications of sharing their voices in video and can decide on content, the shape of the final product as well as who can watch the video.

Hot Tip – taming the technology:

Experiential learning is at the core of PV M&E. Our motto is “Mistakes are great” and the process is guided by InsightShare’s values & core charter. This encourages participants to feel safe and own the learning space, lose fear of equipment, work at their own rhythm, have fun and enjoy the learning journey. The suite of tools employed include; PV games, editing games, Participatory Learning in Action exercises, visualisation techniques, Theatre of the Oppressed games, role-play and various art exercises.

Rad Resources:

Participatory Video for M&E: in our website you can find plenty of resources about PV M&E, including videos, photostories, case studies and articles.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Video in #Eval Week: Cindy Banyai on Putting it in Their Hands – Using participatory video to foster evaluation ownership
2. Susan Kistler on Free Guides to Participatory Video and the Most Significant Change Technique
3. Video in #Eval Week: Paul Barese on the Value Added of Video

This weekend, CEP president Phil Buchanan, advisory board member Fay Twersky, and board member Anne Warhover will discuss the topic of overall foundation performance assessment at the Council on Foundations Annual Conference in Chicago. As a primer for this session—being held Sunday, April 7 from 4:30 to 6:00 pm—here’s a data point from our archives on why foundations conduct overall assessments.

In our 2011 report State of Foundation Performance Assessment almost half of foundation CEOs (48 percent) reported that they were combining information across functions into a foundation-wide performance assessment.

The most frequent reason cited for doing so was to learn and to improve the foundation’s future performance. Other frequently cited reasons were demonstrating accountability for the foundation’s use of resources and understanding the external impact that can be attributed to the foundation’s work.

CEOs working at foundations that reported combining information into a foundation-wide performance assessment differed from those who did not on a few dimensions:

• They tend to collect more types of information to understand effectiveness
• They find the operational data they collect more useful in understanding how effective they are
• They report having a better understanding of the progress their foundation is making to realize its strategies

* * * * * * * *

If you are attending the COF conference this week, be sure to swing by to visit CEP at booth #500 for a demo of our new online reporting tool.

To read about foundation CEOs’ attitudes toward assessment and what foundations are doing to understand their performance, see the report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs, published by the Center for Effective Philanthropy.

Ellie Buteau is Vice President of Research at the Center for Effective Philanthropy. You can find her on Twitter @EButeau_CEP.

Hi there. I’m Corey Newhouse, the Founder and Principal of Public Profit, based in Oakland, California. We help public service organizations measure and manage what matters.

Lessons Learned – how we’re using video: Public Profit uses video in our evaluation of the Partnerships for Learning (PFL) initiative of the National Equity Project.

We use video as an elicitation technique in our teacher interviews. That’s a fancy way to say that we take video of PFL teachers in the fall, and then show the footage again to them during a follow-up interview in the spring. When teachers can see themselves in the classroom, they are able to be very specific about the ways in which their practice has changed while receiving coaching from PFL, and helps outsiders (including us!) better understand the changes they describe.

We work with a professional videographer to get high quality footage, and usually edit the video in-house. We have also taken our own video using inexpensive handheld cameras. If you chose to shoot your own footage, invest in a lapel microphone, as it makes a big difference in the quality of the footage.

Hot Tips – integrating video in evaluation: Enhancing subjects’ ability to recall their performance improves the quality of our interview data, particularly when we’re interested in such a complex phenomena as teaching.

Plus, we have video of teachers’ practice to share, improving the explanatory value of our reports.  Speaking of, here’s a clip of teachers’ practice, along with their reflections on their work with students. If you can’t view the video below in your browser or email, you can view it directly at: http://www.youtube.com/watch?v=Huw9IYY8hO4.

Lessons Learned – when and when not to use video: Take time to negotiate with your client – and their clients, if needed – about when videotaping will take place, and how it will be used. This will help to put your subjects at ease and assure that the shoot day runs more smoothly. A one-page description of your video project is a really useful conversation starter.

Hot Tip – taming the technology: It pays to plan when taking video! Collaborate with your team (including a videographer, if you have one) on a moment-to-moment calendar for your shoot day, including what kind of footage will be taken and when. While shooting, make notes to yourself about when particularly important moments were taped so that you can find the footage easily during editing.

Rad Resources: Video in Qualitative Research by Heath, Hindmarsh, and Luff, is an exceptionally helpful guide to those of us new to using video in evaluation.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Video in #Eval Week: Paul Barese on the Value Added of Video
2. Corey Newhouse on Using Video in Evaluation
3. Video in #Eval Week: Kas Aruskevich on Telling the Story Through Video

Greetings. I’m Paul Barese, Owner-Director of Quimera, a small business based in Washington, DC. Quimera isn’t a traditional media house, rather, teams are interdisciplinary and our approaches are informed by social science and qualitative research, change management, film studies, and social enterprise.

Quimera helps clients identify innovative ways to use video. In some projects video-based approaches are integrated into the data-collection and analysis process as part of the evaluation toolkit (ie: participatory and ethnographic methods– video as process). In other projects, video documents data collection and/or discussion of findings for reporting and dissemination–video as product). In some cases it’s both. We’ve co-developed projects as part of strategic design, evaluation, knowledge management/ training, reporting/ dissemination/ communications/public relations.

Quimera focuses on the added-value of video, from conceptualization, co-design, production, through editing and dissemination. At times Quimera is a member of the research team, using video creation as part of the data collection process, and raw footage as qualitative data contributing to analysis. Quimera’s work is collaborative, ethnographic where appropriate (applied) and it’s common that stakeholders, beneficiaries and research/ evaluation members are involved in content creation. Quimera leads an editing process that includes varying degrees of review and collaboration with the client and little if any participation from beneficiaries.

Here’s an example of a video report that we made (if this does not display in your email or browser, click through directly to: http://www.youtube.com/watch?v=XxgTw2Ya5DQ)

Lessons Learned: When working with video it can be important for clients to think about organizational dynamics, politics, and the relationships between functional areas.

• Look at maximizing (and co-funding) investments in production and editing by coordinating with other departments. Folks doing knowledge management, training, and communications are often interested in video and could be included in the planning process prior to field production so crews capture that extra bit of content which makes the raw footage useful to colleagues. Multi-purposing video can be important for financial and collaborative purposes.

• Discussions with colleagues in other departments, ie: communications, can help defuse what might become tense or frustrating relationships between uses and dissemination of video created by “evaluation” but that carry interest, relevance or transparency concerns to communications.

• Integration of video into evaluation can impact functional roles within an existing organization, required skill sets, work processes, equipment and IT infrastructure. It’s useful to look at bigger picture issues and longer-term ramifications.

• Participatory approaches to content creation (putting a camera into the hands of a beneficiary) are valuable on many levels… engagement, dynamics, transparency, accountability, comfort, openness, sensitivity, data-quality.

Lessons Learned – Taming Technology: Don’t overlook audio equipment (external microphones). Audiences are forgiving when it comes to rough visuals but bad audio is the surest way to lose viewer attention and almost guarantee the click off.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Corey Newhouse on Using Video in Evaluation
2. Video in #Eval Week: Cindy Banyai on Putting it in Their Hands – Using participatory video to foster evaluation ownership
3. Video in #Eval Week: Corey Newhouse on Using Video as an Elicitation Technique

My name is William Rickards; I am currently senior research associate in the Office of Program Accreditation and Evaluation at USC Rossier School of Education. My career has largely been focused in higher education, although I have worked in program evaluation in delinquency prevention, youth services, and in a range of educational and social services.

Lessons Learned-How I’m using video: Over the last few years I have been particularly interested in the use of video-recording for interviews; in my case, this has usually meant interview studies with students and graduates. Use is primarily as data collection, although I use select segments for reporting to faculty; I do the taping on my own, often with portable equipment.

Two examples:

• In evaluating the use of an undergraduate learning e-portfolio, I interviewed graduates regarding their use of the portfolio to monitor and assess their own development
• In an evaluation for a graduate teacher education program for teachers in international schools, I interviewed the teachers on their paths into international school work to understand how to best meet their needs

Hot Tips—Considerations when using video in evaluation include:

• The video as a particularly rich artifact presents potential challenges in terms of analysis: How will the transcript be handled? How much depth will be included in the text?
• At another level, the video record offers a unique opportunity—and often a stark one—from which to study and hone one’s own skills as an interviewer.
• Additionally, the video artifact can provide material that can be used in reporting, depending on clearances, in presentations, websites, or project videos.

Hot Tips—Taming the technology

• The biggest consideration with the technology (particularly in field settings) will be the microphone. External mics—that plug into the camera—are usually best, even if they must often be purchased separately.
• Data storage and transfer need to be studied in relation to individual situations, equipment, and comfort levels.
• Power will always be a consideration—as in battery life and access to a power supply.

Rad Resources

The ethics of informed consent and participation are always a concern, but video complicates this because of participant identity recorded visually. For example, it is standard practice to de-identify data that are being stored for analysis, but this is difficult with video records. These factors need to be considered in the consent and video release forms.

• The American Anthropology Association’s statement on Ethnography and Institutional Review Boards may be helpful in this regard.
• The AEA Guiding Principles are also helpful.
• There is considerable information available on YouTube and Vimeo under ethnographic interviewing that can be useful for evaluation applications (e.g., http://vimeo.com/1269848 ).

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Video in #Eval Week: Corey Newhouse on Using Video as an Elicitation Technique
2. Video in #Eval Week: Cindy Banyai on Putting it in Their Hands – Using participatory video to foster evaluation ownership
3. Corey Newhouse on Using Video in Evaluation

Impact investing has been hailed as potentially transformational by nonprofit sector, corporate, and foundation leaders. Elizabeth Littlefield, President and CEO of the Overseas Private Investment Corporation, has called it “the game-changer we need in the quest to end poverty” that will allow us to “solve many of the world’s social problems while making attractive financial returns.”

The difference impact investing will make has been compared to venture capital’s impact on the private sector, something that Ronald Cohen and William A. Sahlman argue “will bring much needed change to the social sector.”

I’ll admit that, though I am excited about the promise of more capital deployed to do good, I find myself looking for some hard data to back up these assertions.

At this point, I simply have too many questions. What is impact investing, exactly—and where is the data on its performance? And, given that it sounds almost too good to be true, should we be worried that maybe it is?

Perhaps most relevant for us at CEP, what is the role of foundations?

In many of the accounts heralding impact investing as the potential solution to the world’s ills, foundations, with their more than $300 billion in assets, are featured prominently. They are portrayed as leaders, or likely leaders, in this new realm. A story on American Public Media’s Marketplace reported:

Private foundations have always been legally allowed to invest their considerable endowments in social causes. It’s just that most of them chose not to, preferring to separate the Wall Street side of the organization that handles bank accounts from the Skid Row side that handles charitable giving. That’s done in the form of grants, usually around five percent of assets each year because that’s what the IRS requires. The Wall Street side earned money, the grants side gave it away. That’s changing. Foundations increasingly see for-profit investments as a tool to do their social good, sometimes as a better tool than grants.

But it is unclear to what extent the claim in the Marketplace story is true; that is, whether there is a broader trend among many foundations to practice impact investing, or simply media interest in a few important, but isolated examples?

Clearly, there are some major foundations that have taken significant and important steps into impact investing, such as the W.K. Kellogg Foundation (see CEO Sterling Speirn’s excellent post on Kellogg’s experience for the CEP Blog) and The Greater Cincinnati Foundation. But, beyond a handful of much discussed examples, it is unclear how much of the talk about impact investing is just that—talk—and how much is reflected in actual practice. (Disclosures: Kellogg is a grant funder and client of CEP’s; Greater Cincinnati Foundation is a client and its CEO, Kathy Merchant, is our board chair.)

On the other hand, I have heard many foundation CEOs and board members eschew the idea that they’d ever seek anything other than maximum returns for their foundations’ endowments.

A little hard data would be nice, at this point, to bring the prevalence of this practice into focus.

CEP has been gathering some data on this, most recently in a broader survey of CEOs we will share results of at our conference next month and in a report we’ll publish later in the year. What we know at this point seems to indicate that a majority of our respondents (foundation CEOs with grantmaking budget of $5 million or more; 45 percent response rate—total of 211 responses) are either doing “impact investing” or considering it. Thirty-five percent say they are doing it; 23 percent are considering it. And 42 percent say they aren’t doing it and aren’t considering it.

So perhaps my skepticism is misplaced and the Marketplace story’s idea that the practice is becoming more widespread is correct. But, frankly, we don’t entirely know what to make of that data—because definitions of “impact investing” vary so widely and because this was a broad survey that covered a range of equally important topics covering a range of topics.

To be sure, the responses to our survey question seem to indicate that something is happening—meaning there is much more still to understand. That is why we at CEP are planning a major study on foundations and impact investing. In it, we’ll ask much more granular questions, about definitions, about whether impact investments are coming out of the grants budget or endowment investments, and about amounts deployed. We’d also like to get a sense of their performance, both in impact terms and with respect to financial terms. In short, are impact investments living up to the hype?

That’s also the question that will be at hand during a session at our national conference May 21-22, which will bring together leading thinkers and practitioners, as well as a skeptic, for a robust discussion. Kathy Merchant and Sterling Speirn will be a part of the session. Also speaking will be Nonprofit Finance Fund CEO Antony Bugg-Levine, and co-author of the book Impact Investing: Transforming How We Make Money While Making a Difference, and Social Finance CEO Tracy Palandjian.

Just how big is the impact investing trend and what difference will it make?

To us, the answers remain unclear. But we’re working toward clarity, and we hope you’ll help us get there.

Phil Buchanan is President of CEP. He is also a regular columnist for The Chronicle of Philanthropy. Follow him on Twitter at @philCEP and join the conversation on Pursuing Results at #CEP13.

Hi! I’m Cindy Banyai, Executive Director of the Refocus Institute – an international collaboration focusing on training and participatory practices in evaluation. I want to share my experience using participatory video to fully bring evaluation participants into the process.

Lessons Learned – how we’re using video: Video is one tool for analysis and expression in what I describe as participatory action evaluation (using action research – cycles of discussion and activity, involving participants in the entire process of evaluation). Evaluation participants come together to decide what aspects of a target program/community they are going to evaluate, design questions and create a video to answer those questions. The group decides what to film, then films it. It can be descriptive or dramatic and the entire process is in the hands of the evaluation group. After filming, the group analyzes the video according to their pre-discussed evaluation framework, giving the images context. The group then agrees on the meaning of the video (collectively coding it), prepares for presentation (editing is optional, since the film is a discussion stimulus not a product of evaluation), and exhibits the video – to the community, program funders, or other potential evaluation group participants. Group members use the video as a tool for discussion during the exhibition, providing analysis and interpretation of the images to a new audience. The evaluator serves as 1) facilitator, guiding the group through technical aspects of evaluation, and 2) observer, detailing the process, content and group interactions, adding another layer to the evaluation (and material for a formal evaluation report if desired).

Hot Tip – It’s not your video Mr. Evaluator! The video can be a product of the evaluation, if the group decides they want others to see it. It is important to remember the video is the intellectual property of those who made it – not the evaluator or the donor/program that commissioned it. If you want to use the video after the evaluation, the group must agree to allow you to use it (a formal agreement may be necessary).

If the video below does not show in your browser or email reader, go to http://www.youtube.com/watch?v=2rOPn9PPC-Q to view it on YouTube.

Hot Tip – taming the technology: iPad/iPhone and iMovie – These are handy, simple to use and easy to teach to others. The price of the devices is coming down, especially for older versions, and to download a simplified version of iMovie (for editing) on them is only $4.99.

Rad Resources: YouTube – This is a great way to share and now even edit videos. You can also share a group’s evaluation findings with a broader community, if that’s what they want to do. It provides even more interesting potential for doing entirely virtual participatory videos too!

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Video in #Eval Week: Corey Newhouse on Using Video as an Elicitation Technique
2. Corey Newhouse on Using Video in Evaluation
3. Video in #Eval Week: Kas Aruskevich on Telling the Story Through Video

Greetings from Alaska. I’m kas aruskevich, principal of Evaluation Research Associates (ERA), I work in rural Alaska with a great team of evaluators, associates, and local intermediaries. In the unique Alaskan context in which we work, telling the story through video helps us to show the context of people, place, and situations. Video clips, compiled into a video report, can be used as evidence of accomplishment as well as to educate an audience (often the funder) holistically about a project. Shorter impact videos can also motivate participants, giving the evaluation an effect beyond reporting.

Most of us have used written interview quotes in our evaluation reports. As example, below is a quote from an interview with a Gaalee’ya STEM project student:

Uvana atiga Nanuuraq (my name is Nanuuraq) I’m from a place called Noatak, my name is Brett James Kirk, 18 years old, incoming freshman at the University here in Fairbanks. So far what I know about STEM seems great. I really agree with how they’re incorporating the indigenous ways with the western ways here because we have a chance to talk about the similarities and differences between the two. And I’m looking forward to all the other meetings throughout the school year.

Compare the 40 second video clip below of the text quoted above. If the video does not show in your browser or email reader, go to https://vimeo.com/62366707 to view it on Vimeo

Gaalee’ya-AEA from kas aruskevich on Vimeo.

Lessons Learned – Generally:

1. Good audio is EXTREMLY important.
2. Shooting footage is easy, editing the video is challenging.
3. Editing is time consuming. One minute of finished video may take 8 or more hours of editing – and that’s after clips are selected and cut to approximate size.
4. Take good pictures. It easy to put motion to a photograph and use it as background to an audio quote taken from an interview.

A good evaluative video starts with data collection in the form of video and photos that gives evidence of accomplishment and provides visual description.

Lessons Learned – Taming the technology:
For the majority of video reports I work with a local videographer who has also mentored me in both camera use (Cannon 7D) and audio (Zoom H4n 4-Track Recorder). After three years of video production, I primarily stick to photographs and video editing (Final Cut Pro 7). I’ve produced video reports 20 minutes in length and less, however now I prefer to produce supplemental impact videos that are 3 minutes and less. Remember it’s technology, and with technology comes glitches.

Rad Resources to explore:

• Find a videographer near you, see if you can watch a shoot and then the editing process
• View impact videos (my alma matter): http://www.uhfoundation.org/impact-video
• Get good equipment: http://www.bhphotovideo.com

But most important, know how to conduct an appropriate evaluation, be reciprocal, gather good evidence, and report out. The rest is technology.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Video in #Eval Week: Cindy Banyai on Putting it in Their Hands – Using participatory video to foster evaluation ownership
2. Video in #Eval Week: Paul Barese on the Value Added of Video
3. Video in #Eval Week: Corey Newhouse on Using Video as an Elicitation Technique

I am Susan Kistler, AEA’s Executive Director and aea365’s regular Saturday contributor. Today I wanted to share one of the raddest resources of all!

Rad Resource – AEA Summer Evaluation Institute: Registration is now open for the AEA Summer Evaluation Training Institute, to be held June 3-5 in Atlanta, Georgia. Let’s take a look at just a few of the great presentations and presenters.

Rad Resource – Using a Sytems Orientation in Evaluation: Earlier in March, you heard from Beverly Parsons and her colleagues in the ECLIPS Community of Practice here on aea365 about Systems Thinking and Evaluation. Beverly will be giving a keynote at the Institute as well as this three-hour workshop.

Rad Resource – Strategies for Evaluation Plans and Planning: Sheila Robinson, aea365’s lead curator, will help attendees to improve this fundamental evaluation process, including sharing strategies for conducting literature reviews, developing broad evaluation questions, leveraging the Program Evaluation Standards and AEA Guiding Principles, and describing  the evaluand.

Rad Resource – Assessment and Action Planning for Sutainability: Did you know that Annaliese Calhoun penned the most read aea365 post ever when she wrote about Measuring Sustainability Capacity and Planning for Long Term Success back in December of 2012? She’ll be returning to the summer institute with this extremely well received workshop for 2013.

Rad Resource – Basics of Program Design: A Theory-driven Approach: John Gargani and Stewart Donaldson are bringing their outstanding workshop from the annual conference to the Institute in Atlanta. This is a unique opportunity to learn more about program design via an evaluation lens.

Rad Resource – How to Tell a Compelling Performance Story About the Stuff That Really Matters: Jane Davidson of Genuine Evaluation fame is coming all the way from New Zealand to share her expertise around truly practical and actionable evaluation,

Rad Resource – Workshop on Practical Methods for Improving Evaluation Communication: Stephanie Evergreen will be  offering this full day course on the Sunday, June 2, as a pre-institute workshop. There’s been lots of talk on aea365 about about Stephanie’s work and this will be the first time she’s offered this 6-hour session in Atlanta.

Hot Tip: Oh yea, I’ll be there too! Join me for 25 Low-cost/No-cost Tools for Evaluators or Popping the Question: Developing Quality Survey Items.

I only had space enough her to highlight nine of the 25 workshops at the Institute. Explore the full agenda to check them all out and register early to ensure that there is space available in your favorites. Hope to see you there!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Susan Kistler on Taking a Professional Development Break This Summer
2. Susan Kistler on Evaluating the Zombie Apocalypse
3. Susan Kistler on Remembering the Ides of March and Thinking About Your Message

Hello! I am Caryn Mohr, a Research Scientist at Wilder Research in St. Paul. I’m one of more than 40 researchers in the office who conduct primary research. My own work focuses on education programs addressing opportunity and achievement gaps. Our office also manages Minnesota Compass, a nationally recognized community indicators project. Opportunities to collaborate with Compass staff have shown me the power of connecting primary research to community indicators data.

Those of us who conduct primary research gather new data first-hand. We use a variety of methods to evaluate the impact of individual programs and test research hypotheses. My own work ranges from case studies to long-term, quasi-experimental studies of education programs. We administer surveys, conduct interviews, convene focus groups, and employ a variety of methods that give us deep and direct knowledge of study participants’ experiences. We work closely with individual programs and organizations to help them understand their impact.

My colleagues at Minnesota Compass help us see the big picture. Compass provides a common framework for measuring and tracking state and local progress on a range of topics, including education, the economy, health, housing, and other important social issues. In each area, an advisory committee of stakeholders identified key indicators which are monitored over time to understand the health and progress of our community.

Lessons learned:

• See the big picture. Considering results of individual program evaluations in the context of community indicators provides a broader perspective and meaningful context to stakeholders. For example, results of a recent STEM (science, technology, engineering, and math) education program evaluation can be considered in the context of indicators of progress along the STEM cradle-to-career continuum. This context can help program staff consider their goals in relation to benchmarks and gaps pertinent to the continuum. Connections to indicators afford exploration of questions such as: How do program goals relate to research-based benchmarks and community needs? Are resources being targeted effectively?
• Identify themes. Considering study results in the context of community indicators can help researchers identify meaningful themes across individual program evaluations. In education, our community indicators show stark achievement gaps. It is important to consider what our first-hand knowledge of individual programs tells us about addressing these gaps, and how this relates to research literature and community needs. Moving the needle on indicators requires understanding how programs work on the ground. Likewise, effectively targeting program resources necessitates an understanding of community needs.

Hot tip:

Explore connections to community indicators to provide meaningful context to individual program evaluations.

Rad Resources:

• The Community Indicators Consortium provides links to community indicators projects and resources.
• Minnesota Compass Project Manager Craig Helmstetter offers tips and lessons for indicators work in this AEA365 post.
  (Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. MN EA Week: Craig Helmstetter on ‘Evaluating’ your community
2. Sona Madison on More Free Photo Sites
3. STEM Week: Alyssa Na’im on Using Culturally and Contextually Responsive Practices in STEM Education Evaluation

I’m Paul Bakker, the founder and lead consultant of Social Impact Squared. I help agencies with a social purpose understand, measure, communicate, and improve their outcomes. One of the services I provide is data analysis, so I deal with statistical significance quite a lot.

Hot Tip: Abandon the 95% rule. In statistics classes, they teach you to reject the null hypothesis (of no difference) if there is less than a 5% chance that the hypothesis is true. It makes sense to provide students working on text book examples with a rule of thumb, but people don’t use such a rule when making real-life decisions. Before you analyze your data, discuss with your clients and the relevant decision makers the level of confidence they need to make a decision. Maybe they want to be 95% confident, or maybe being 80% confident is good enough for them to act.

Hot Tip: Consider the need to adjust for increases in error due to multiple tests. Often, you need to run multiple tests to analyze your data. The chance that at least one test will conclude that there is a difference when there isn’t is equal to:

For instance, if you ran 20 tests at the 95% significance level, then the chance that at least one of those tests provides you with a wrong answer is:

The typical advice is to increase the significance level of each test. However, consider the following possible scenario. Out of those 20 tests, 6 tests are significant at the 95% level, but only one is significant at the 99% level. What is more important to your client? Acting on one incorrect difference or not acting on four real differences?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Steve Fleming on Explaining Statistical Significance
2. ROE Week: Anne Heberger Marino on a field experiment to increase survey response rates
3. S Mooshegian on Making the Most Out of Kirkpatrick’s Lvl 1 Measurement in the Classroom

“We started off where many of your respondents started, which was having an intention around evaluation, wanting to know more. It’s not our core business. Our core business is not being evaluators—our core business is teaching kids how to play and get along with each other on the playground—so it’s not surprising that we weren’t good at it.”

– Elizabeth Cushing

On January 30, 2013, the Center for Effective Philanthropy hosted a panel discussion titled Dispelling the Myth of Nonprofit Complacency, sponsored by the James Irvine Foundation. Panelists included CEP Research Manager Andrea Brock, REDF President Carla Javits, and Playworks CEO & Chief Operating Officer Elizabeth Cushing. The event was moderated by Kevin Rafter, manager of research & evaluation at the James Irvine Foundation.

The discussion was based on findings from CEP’s report, Room for Improvement: Foundations’ Support of Nonprofit Performance Assessment, which was released in September 2012. One of the major myths this report was able to examine is that nonprofits don’t care about performance assessment. Nonprofit leaders surveyed for the report stated that they do in fact care about assessment, but cited a general lack of support—both nonmonetary and monetary—from their funders.

Elizabeth Cushing shared how Playworks came to grips with what they needed in terms of assessment and what they wanted to measure. She explained that assessment can often be extrinsically motivated and explained how Playworks was able to make this an integral part of their work with help from their funders.

Carla Javits discussed how REDF approaches performance assessment from both the funder and grantee perspective. To combat what may seem an initial resistance among grantees, for example, Carla said funders need to give grantees both the time and space to think it through, as well as offer the resources for them to do so.

Panelists spoke candidly on the apparent disconnect between what funders think and what nonprofits are saying, alluding to a greater need for more effective communication between funders and grantees.

We hope you’ll listen to the full discussion here and take the time to think about how you might improve some elements of your own funder-grantee relationships.

   Dispelling the Myth of Nonprofit Complacency panel discussion

Addy Ashiru is Senior Coordinator of Communications, Programming, and Development at the Center for Effective Philanthropy.

Hello, we are Linda Cabral and Laura Sefton from the Center for Health Policy and Research at UMass Medical School. We often collect qualitative data from interviews and focus groups. One challenge we frequently face is how to quickly and efficiently transcribe audio data. We have experimented using voice recognition software (VRS), and we’d like to share our approach.

You will need headphones, a microphone (stand-alone or attached to a headset), and a computer with audio playback and VRS installed on it. We use Dragon Naturally Speaking Premium Version 11.5 voice recognition software, however other VRS is available. Use of audio playback software will allow you to control the playback speed, so you can slow it down, pause, fast forward, and rewind as needed.

Open the audio file in the playback software and open a new document in the VRS. While listening to the audio via the headphones, repeat what you hear into the microphone. During this step, you can format the document to indicate who is speaking and to add punctuation. Because VRS works best when trained to understand a single voice, a designated team member should repeat all spoken content, regardless of how many voices are in the audio file.

This process will generate a document in the VRS that can be saved to your computer as a Word file. As a final review, read through the Word file while listening to the audio file and make needed corrections. This could be done by another member of the project team as a double check of the document’s accuracy.

Hot Tips:

• Spend time training the VRS to recognize your voice. A few practice sessions with the software may be needed where you can read dummy data into the software in order for it to learn your voice. This will improve the transcription quality, minimizing the time spent editing.
• Train the VRS to recognize project-specific acronyms or terminology prior to starting transcription.

Lessons Learned:

• Often, financial resources for evaluation projects are limited. In an effort to keep the transcription process in-house, our administrative staff transcribed the audio files. By using the VRS and someone from our project team familiar with the data as the designated recorder, we have found savings in time and efficiencies.
• No transcription yet has captured 100% content accurately the first time. Therefore, build in time to listen to the recording and to make manual edits.

Rad Resources:

These resources may be helpful as you explore whether VRS is right for you.

• “The Voice Transcription Technique: Use of Voice Recognition Software to Transcribe Digital Interview Data in Qualitative Research” by Jennifer Matheson

• VRS products Review by consumersearch: “In reviews, it’s generally Dragon vs. Dragon”

• (Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Sonny Hines on F4 and F5 Transcription Software
2. Robert Brunger on Practical Tips for Focus Groups
3. LaMarcus Bolton on Managing Local Computer Files

Hi, I am Kerry Bruce, Director of Results and Measurement at Pact.  I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs. We have begun integrating mobile technology into our programs by using mobile phones for baseline and endline data collection.

Hot tips:

• Mobile technology has advanced significantly since the last time you likely considered using it and now is the time to invest in learning about this technology.  Many of the early bugs have been worked out and the commercially available platforms make collection of data via mobile phone or tablet quite easy.
• New platforms are easy to use, there are many to choose from and most include built in dashboards that help you to review and visualize your data.
• A careful assessment of network coverage, power and power back-up should be done before you decide on a type of phone and platform.  While you don’t necessarily need a signal to use mobile phones to collect data (you can collect data offline) you will need a phone with long battery life! Many phones are now GPS enabled—and you should consider these if you would like to collect GPS waypoints and conduct geospatial analyses.
• Understand the skills and competencies of your data collectors.  Will they be people who are familiar with mobile phones or will they need significant training and mentoring?  What type of phone will be easiest for them to use?
• If you are using mobile phones for data collection of a baselines survey, for example, will you have a follow on use for the phones? You’ll want to consider what type of phone will be most useful for future activities so that you can yield a higher return on investment of your initial purchase.

Lessons Learned:  

• A careful assessment of your data collection needs, logistical issues, and possible future projects is necessary before you start utilizing mobile technology.
• Because not everyone sees the benefits of mobile technology, a basic overview of the advantages of this innovation is helpful to get your co-workers on board.

Rad Resources:

• Online mobile technology training for a variety of uses is available for a fee from TechChange.
• There is a free online mobile data collection selection assistant at NOMAD.

*Thank you to Mobenzi Researcher and DataWinners (DataWinners free data collection App for Android devices built using Open Data Kit tools) for the use of their images in this post.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Cameron Norman on Information Technology and Evaluation
2. LAWG Week: Fatima Frank on Using Technology in the Field of Data Collection
3. Melissa Cater on Visual Evaluation Methods

This is Kim Snyder, Associate at ICF International, Rene Lavinghouze, Evaluation Team Lead for CDC’s Office on Smoking and Health, and Patricia Rieker, Adjunct Professor of Sociology at Boston University and Associate Professor of Psychiatry at Harvard Medical School. We have been investigating public health program infrastructure as an ignored component of the left hand side of logic models.

Evaluators often are asked to focus on outcomes or the right hand side of the logic model. How often is life better for people because of a successful public health program (e.g. fewer heart attacks, less exposure to second hand smoke)? While we value the importance of this type of evaluation, we were concerned that the inputs or foundation of our activities are not fully understood. If we don’t start out with the foundation that enables organizational capacity, how are we supposed to really know what affects the outcomes on the right side of logic models?

Lesson Learned:

• The left hand side of the logic model is something that is rarely defined or explained in public health programs. Take a look at the Office on Smoking and Health’s logic model for eliminating nonsmokers’ exposure to secondhand smoke. Under Inputs, what is meant by “State health department and partners”? If it is interpreted and replicated differently can we expect the same outcomes?

So we decided it was important to define and study what functioning public health program infrastructure (or the foundation of public health outcomes) looks like. Previous work, a literature review across public health programs (see Rad Resource) and data from 19 tobacco control programs were used to further our understanding of functioning program infrastructure.

Building on previous work (that is currently in press with the Journal of Public Health Management & Practice) we define infrastructure as a key component and the foundation or platform that supports capacity, implementation, and sustainability of program initiatives; a definable entity, a cyclical process and part of a larger system that requires constant vigilance to be effectively maintained. Using a grounded theory approach we developed the Component Model of Infrastructure or CMI for short.

Rad Resource: Infrastructure: More Than Platforms For Moving Vehicles available in the American’ Evaluation Association (AEA) Public eLibrary.

Sneak Peek:

We are still refining the CMI and hope to share a final version this year. We define five core components of public health program infrastructure:

• Networked Partnerships,
• Multi-Level Leadership,
• Engaged Data,
• Managed Resources, and
• Responsive Plans/Planning.

We see the CMI as a practical model of public health program infrastructure that could provide the framework that grant planners, evaluators, and program implementers need to measure success, to link infrastructure to capacity, and to increase the likelihood that health achievements will be sustainable.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Susan Sloan on CDC’s Framework for Program Evaluation in Public Health
2. SIOP Week: Dale S. Rose on Organization Development: A Program Worth Evaluating (Logically)
3. Sally Honeycutt on Developing Logic Models

Hello all! We are Kristi Fuller and Glenn Landers, staff at the Georgia Health Policy Center. The Center is housed within Georgia State University’s nationally ranked Andrew Young School of Policy Studies and provides evidence-based research, program development, and policy guidance.

We gave a roundtable presentation at the American Evaluation Association’s (AEA) 2012 conference focused on ensuring utility in evaluation practice, in which we used the Centers for Medicare & Medicaid’s Money Follows the Person (MFP) demonstration program as an example.  Our current evaluation of the MFP program for the state of Georgia has the potential to last ten years.

Hot Tip: When conducting an evaluation over a long time frame, it is conceivable to get into a pattern and produce reports in which stakeholders begin to lose interest. However, keeping the Joint Committee on Standards for Educational Evaluation (JCSEE) Program Evaluation Standards regarding utility in focus can help evaluators avoid this trap.

Lessons learned:

1. Utility standard 2 emphasizes the importance of devoting adequate attention to all relevant stakeholders. For MFP, regular evaluation steering committee meetings bring diverse perspectives of those interested in results, as well as those impacted by the program. Through this interaction, we gain important information used to plan the evaluation so that it provides benefits to a broad range of stakeholders including program participants, familial advocates, attorneys providing legal assistance, programmatic staff, and nursing facility advocates.
2. Utility standard 5 discusses the importance of providing information relevant to needs that are both known and evolving. Recognizing that as programs develop and grow the needs of the invested parties also change is important for ensuring that what is being studied continues to be of relevance to stakeholders. In our experience with MFP, we’ve found that program personnel are interested in delving into data to understand their clients’ experiences, whereas the state’s Medicaid agency is particularly concerned about how services are being utilized.
3. Utility standard 6 describes utilizing various communication methods to create processes and products that are meaningful for challenging and reinterpreting understandings. Interpretation of data can be done in a myriad of ways, and AEA’s Data Visualization and Reporting TIG provides great ideas. One way that we’ve tried to manage this is through dropping the production of our full report from quarterly to semi-annually, allowing more time to develop dashboards and ad-hoc analyses.

Rad Resource:

Food for Thought:

• What are you doing that works well regarding how you engage stakeholders?
• How are you managing different points of view successfully?
• What do you think works well with your data presentation?
• What could you do either more of or differently?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Gita Upreti and Carl Liaupsin on Developing Efficient Measures with Educational Data
2. Alberta Mirambeau on Program Stakeholders and Evaluation Stakeholders
3. OL-ECB Week: Bonnie Richards on Setting the Stage: Evaluation Preparation and Stakeholder Buy-In

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. On occasion, I offer training on different aspects of evaluation. Following a recent workshop at the Minnesota Evaluation Studies Institute, I posted the following to AEA’s LinkedIn Group:

Do you know of songs with an evaluation bent? This might be filed under ‘weird question of the week.’ I am teaching workshops on evaluation, in particular on data visualization. I regularly use two songs as transitions- “Do you want the truth or something beautiful” from Paloma Faith (a great bluesy rumination) and “I Love Charts” from Sid the Sience Kid (a children’s song that shouts “Charts rule!”).

Ever-responsive, the LinkedIn group came back with a plethora of suggestions.

Rad Resources – Evaluation Related Songs: Suggestions from our colleagues included:

• Paul Duignan suggested a compilation of 20 songs related to No Child Left Behind
• Keren Brooks added What If No One’s Watching, Fire Door, and My IQ from Ani DiFranco
• Alan Gumbel went back to the 1973 musical Lost Horizons to consider The World Is a Circle and Question Me an Answer
• Seanna Hasson suggested What’s the Use from Jamie Lidell
• Colleen Duggan recommended Cognitive Bias from Brad Wray
• And both Colleen and Nayiri Dolabjian suggested Terry Smutylo’s The Impact Blues
• Allison Titcomb’s recommendation ranged widely from My Way by Paul Anka (popularized by Frank Sinatra and others), to STAT-60 from MC Lars, and the Indiana RISE Evaluation Song from K Downey
• Elisa Avila recommended a song I love and hadn’t thought of – Effect and Cause from the White Strips
• Cliff Carr added (I can’t get no) Satisfaction from the Rolling Stones
• Tongue in cheek, Harry Jones suggested ‘the ultimate negative assessmet,’ God May Forgive You, But I Won’t

Hot Tip – AEA on LinkedIn: AEA’s LinkedIn group is a great start to connect with colleagues far and wide for guidance on questions from the lighthearted to the fundamental. Members and nonmembers alike are welcome to join the conversation.

Get Involved – Suggest a Song: Any songs we’re still missing? Share your ideas via the comments. Thanks!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Sharon Mahoney on Evaluation Songs
2. Susan Kistler on Getting Engaged in Environmental Evaluation
3. Susan Kistler on AEA on LinkedIn and LinkedIn’s New Endorsement Feature

Hello! We are Tarek Azzam (Claremont Graduate University) and Matt Keene (Environmental Protection Agency). We are members of the External Review Panel for ECLIPS.

Whoa! Even though it’s oh-so-tempting to try, you don’t need the perfect string of words to define “systems-oriented evaluation.”

Lesson Learned: It’s in the roots of evaluation 

At its core a systems approach to evaluation encourages the evaluator to consider the physical, political, and structural issues that surround a program, and to examine how these factors help or hinder the success of a program. This examination and reflection process is incorporated in the work and writings of lead evaluation scholars such as Lee Cronbach, Robert Stake, and Jennifer Greene. The presence of systems thinking also can be seen in our standards (specifically standards F3, A4, and A7).

And it’s also something different, because it requires the evaluator to recognize that the program is part of systems that have their own dynamics. It forces the evaluation to examine issues that go beyond the process and outcomes of a single program.

Lesson Learned: How to become a systems-oriented evaluator

1)      Adopt some habits of systems thinkers…

2)     Know the domains of Social Ecology and use them to understand the leverage points of boundaries, relationships, and perspectives. Donella Meadows says that boundaries are problem dependent and messy. Don’t make the world linear for your mathematical or administrative convenience.

3)     Delve into the dynamics of systems to find the regions of organized, adaptive, and unorganized patterns.

4)     Find leverage points (places to intervene where small tweaks can lead to big changes). Here are some leverage points from Meadows you can use to influence relevant systems:]

5)     Let systems thinking do fuzzy things to your logic model. A fuzzy logic model takes into account the dynamic nature of the systems surrounding a program. It gives a visual image of the complexities that can affect processes and outcomes.

In the ECLIPS, all of our logic models used to look like this:

But after applying systems thinking, we made them into fuzzy logic models. See how different they look.

Try creating a fuzzy logic model to find and depict the system’s complexity, making your logic model more useful to people for a longer time.

That’s all for now

This concludes ECLIPS week on aea365! Don’t expect to learn about systems alone or in a short period of time. It may well be a journey into a new way of thinking about evaluation. Get involved in an existing community of practice about systems or form your own group. ECLIPS members are happy to share what they are doing.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. ECLIPS & Systems TIG Week: Beverly Parsons and Veronica Thomas Welcome You to ECLIPS Week
2. ECLIPS and Systems TIG Week: Karen Peterman and Marah Moore on How a Systems Orientation Influences an Evaluator’s Role
3. ECLIPS & Systems TIG Week: David Hata on Looking to Interconnections for Value Creation