American Evaluation Association 365 Blog

Hello! I’m Silvana Bialosiewicz, a doctoral student at Claremont Graduate University and Senior Research Associate at the Claremont Evaluation Center. Lately I’ve been giving a lot of thought to the relationship between evaluation and organizational strategic development, and how evaluators can support their clients by participating in strategic development efforts.

I’d like to share some of the valuable lessons I learned engaging in this process as part of a formative evaluation conducted on behalf of a local chapter of Habitat for Humanity. The factors that motivated me to build strategic planning into the evaluation were my action-oriented stakeholders and the Organizational Development expertise of my co-evaluator.

Word of Caution: As an evaluator, engaging in strategic development activities isn’t always appropriate. Let the context of the program, the goals of the evaluation, and your relationship with the organization be your guide when deciding if the situation lends itself to these activities.

Hot Tip: Start the strategic planning conversation early. In the evaluation planning phase, open up a dialogue with your client regarding their strategic planning processes, and the ways in which evaluative data could be used to create action plans, and direct energy and resources towards organizational goals,

Hot Tip: Build strategic planning into the evaluation contract. Knowing that these activities will take place from the start can help foster buy-in into the evaluation process and help the organization hold itself accountable to taking action based on evaluation findings.

Hot Tip: Develop personalized reflective self-assessment tools to support the process. Using the program’s theory of change and the evaluation findings, pose a series of questions to your stakeholders to stimulate reflection and discussion regarding where they are versus where they’d like to be.

Hot Tip: Be the voice of the data. If you find yourself sitting at the strategic planning table, speak on behalf of the data. As the evaluator you can support your stakeholders in making data-driven decisions by helping them continuously examine whether strategic plans are aligned with evaluation findings.

Hot Tip: Manage your expectations for organizational change. Understand that organizational development and change is a slow process and program stakeholders have competing priorities and busy schedules. Allow the key-decision makers to captain the ship and dictate the scope and timeline of the development activities.

Rad Resource: The Readiness for Organization Learning and Evaluation (ROLE) Instrument developed by Preskill and Torres is a great tool to help you and the organization determine if strategic planning is appropriate.

Rad Resource: Cassandra O’Neill recently hosted an AEA webinar on this topic entitled The Intersection of Evaluation and Strategy. This quick presentation provides an interesting perspective on the topic, as well as additional resources for interested parties.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

 

 

 

Related posts:

1. Frank Meintjies on How Evaluation and Strategic Planning Go Together
2. AZENet Week: Kori Kanayama on Using Business Model Assessment to Approach Organizational Sustainability
3. IC Week: Susan Wolfe on Small is Beautiful–the “Jack of All Trades”

Hello! I am Lara Hilton MPH, and I am a research analyst at RAND Corporation (Santa Monica, CA), Samueli Institute (Alexandria, VA), and a doctoral student at Claremont Graduate University (Claremont, CA). I have worked in evaluation contexts such as hospitals, clinics, military settings, and international development with a myriad of stakeholders. I have a few guidelines that I find helpful when developing new relationships. Akin to a first date, the first meeting with stakeholders is usually rife with anxiety, expectations, and assumptions. I hope these suggestions help you navigate the waters of this delicate period of relationship building:

Hot Tip: Be informed. Read ahead by accessing as much information as possible about the organization, people, program, mission, goals, and setting. The more you know about them, the more intelligent your inquiry will be. This suggestion cannot be overstated.

Hot Tip: Be alert. Have your antennae out to get a quick read on stakeholders. In a first meeting, many people reveal opinions, expectations, orientations, values, prejudices. Pay special attention to their level of evaluation expertise, interest in evaluation, and goals for what they want to get out of the evaluation. Conflicting opinions may begin to arise that are best negotiated early in the relationship.

Hot Tip: Be flexible. Match your language to theirs. If they are research or academically inclined then use evaluation jargon. If they do not have evaluation expertise then it is best not to overwhelm with meaningless terms. If they use special terms like acronyms then adopt them immediately. Get on the same page with language as soon as possible to enhance communication.

Hot Tip: Be of service. The most effective way to overcome the negative reputation of our profession is to be of service to the organization, transparently. State it up front. I have literally heard a sigh of relief from program directors and staff when they hear me say that I am here to serve them. Part of this process is to make sure that research questions include not only “does it work” but also “how” and “why” so that negative final results are accompanied by their context and feedback loops for improvement may be provided, if appropriate.

Rad Resource: The CDC’s first step in the program evaluation framework is to engage stakeholders. A practical guide for determining how and to what extent to involve stakeholders in evaluation can be found here.

Rad Resource: Donaldson’s 2001 article: Overcoming our negative reputation: Evaluation becomes known as a helping profession (American Journal of Evaluation, 22, 355-361.)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Alberta Mirambeau on Program Stakeholders and Evaluation Stakeholders

My name is John LaVelle, I am the Director of Operations and External Affairs for the School of Behavioral & Organizational Sciences and the School of Politics & Economics at Claremont Graduate University.  I’m also a PhD candidate specializing in Evaluation and Applied Research Methods, and a past curator of the AEA365 blog.  Today I’m going to share three Rad Resources about some organizations whose work overlaps with Evaluation and also maintain active Listservs as a way of sharing ideas and resources.

Rad Resource: The Society for Community Research and Action.  Several posts have been written about the Society for Community Research and Action, which is also known as Division 27 of the American Psychological Association. They are dedicated to advancing theory, research, and social action, and much of their work overlaps with evaluation.  SCRA maintains a very active listserv (similar to EvalTalk), and regularly share information about conferences and professional development, research and work opportunities, as well as professional challenges and questions.

Rad Resource: ARNOVA is the Association for Research on Non-Profit Organizations and Voluntary Action. Many evaluators work in a not-for-profit context or with organizations trying to increase volunteer and philanthropy activities, and ARNOVA is a good resource.  Similar to AEA’s TIG structure, ARNOVA has areas of specialization, such as community grassroots, social entrepreneurship, and even a section for individuals working at the intersection of practice<—->academics (they call this intersection “pracademics”).  ARNOVA’s listserv and webpages have much information about resources, conferences and meetings, calls for papers, etc.

Rad Resource: CBPR stands for Community Based Participatory Research (see Laura Myerchin Skarloff’s excellent post) and the University of Washington hosts a very useful listserv for sharing ideas and problem-solving.  They even offer a parallel listserv just for sharing work and research opportunities.  Membership information is available here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Kathy Bolland on EVALTALK
2. Margaret Riel on Learning Circles
3. Susan Kistler on Resources for Evaluating Communications

Greetings. We are Linda Cabral and Laura Sefton from the University of Massachusetts Medical School, Center for Health Policy and Research. We are part of a multi-disciplinary team evaluating the Massachusetts Patient Centered Medical Home Initiative (MA-PCMHI), a state-wide, multi-site demonstration project engaging 46 primary care practices in organizational transformation to adopt the PCMH primary care model.  To adopt a mixed methods approach, this evaluation utilizes 1) multiple surveys targeted at different stakeholders (e.g., staff, patients), 2) analysis of cost and utilization claims, 3) practice site visits, and 4) interviews with Medical Home Facilitators (MHFs).

We wanted to connect data from the TransforMED’s Medical Home Implementation Quotient (MHIQ) survey with our MHF interview data. We did this to better understand the practices’ MA-PCMHI experience. MHFs provide a range of technical assistance to aid their assigned practices in their transformation process, making them a great source of information about their practices’ transformation. In an effort to triangulate our evaluation findings, we presented the MHIQ results to the MHFs as part of a traditional semi-structured interview. Presenting site specific survey data to MHFs served the following purposes:

• It allowed for MHFs to share their reflections on why their practices scored the way they did on various domains;
• It prompted MHFs to point out major differences between their assigned sites;
• Focused the MHFs on providing practice-specific information; and  instead of generalities across all the sites to which they were assigned
• MHFs provided insight into some of the strengths and limitations of the survey instrument.

Lessons Learned

• Sharing survey data and having respondents reflect on it during the course of an interview, connecting data, proved to be a very helpful strategy. Specifically, we received more detailed responses from interviewees by asking “Why do you think Practice ABC scored a 5 on the care coordination module”? vs. “What can you tell me about how Practice ABC is implementing care coordination?” MHFs would make the case for or against why a practice scored the way they did on a particular domain.
• Involving the MHFs as “experts” on their assigned sites increased the MHFs’ investment in the evaluation process and their willingness to participate in future evaluation activities.

Hot Tip

• We held these MHF interviews prior to doing practice site visits. The practice-specific information that MHFs shared with us deepened our familiarity with the sites prior to conducting site visits.

Rad Resources

• Qualitative and Mixed Methods Provide Unique Contributions to Outcomes Research
• National Institutes of Health’s Best Practices for Mixed Methods Research in the Health Sciences

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Christine Johnson and Terri Anderson on the Quality Improvement – Evaluation Connection
2. Melissa Biel on the Uniform Data System (UDS) Mapper
3. MME Week: Terri Anderson on Using Best Practices for Mixed Methods Research in Evaluation

Greetings fellow evaluators! My name is Johanna Morariu and I’m a Director of Innovation Network, a nonprofit consulting firm that builds the evaluation capacity of nonprofit organizations and foundations. I’m also the co-chair of the Data Visualization & Reporting (DVR) Topical Interest Group (TIG) along with Amy Germuth, Stuart Henderson, and David Shellard and the DVR TIG is hosting all this week on aea365.

Hot Tip: Have you heard about treemaps? Treemaps are a relatively new data visualization technique—especially to evaluators. The technique was created in the 1990s by Dr. Ben Shneiderman for mapping computer hard drive usage.

Treemaps are useful for visualizing hierarchical data, or tree structure data. Here’s an example: there are 100 program participants. Of those participants, 55 are female and 45 are male. Using traditional dataviz techniques, the data looks like this:

Program participant data may contain additional categories of information, such as age. Keeping with traditional dataviz, the data might look like this:

At this point we have two levels of data. 1) Gender: is the program participant female or male? 2) Age: is the program participant between the ages of 14 – 17 or 18 – 21?

What if we want to add a third level of data to our visualization about attendance? Let’s try a treemap, which is designed for hierarchical data:

Area is used to proportionally illustrate differences in values, i.e., number of participants. The larger a rectangle, the more program participants it represents. Nested rectangles reflect the three levels of data—gender, age, and attendance—so that each level can be analyzed.

For example, the proportion of females to males can be ascertained by looking at the female and male rectangles (the outermost rectangles). Also, within male and female, the proportion of 14 – 17 year olds compared to 18 – 21 year olds can be estimated (first level of nested rectangles). And within those rectangles, the proportion of females and males in each age group who partially attended or completed the program is also represented. In this treemap, color is used to underscore largest vs. smallest values, like a heatmap.

Rad Resources: Wondering how to get started making your own treemaps?

• Learn more about treemaps from Wikipedia
• Check out NewsMap for a pop culture treemap example and some inspiration.
• For beginners, check out the user-friendly ManyEyes.
• For evaluators learning R, Nathan Yau on Flowing Data offers a step-by-step guide for getting started with treemaps in R

Lesson Learned: There are more dataviz options than the usual charts and graphs we’re used to! Varying visualization types is akin to paying attention to varied word choice to keep a reader’s interest. It’s not necessary, but it sure can help!

So, how can you use treemaps? And are there other treemap tools you’d recommend to the AEA community?

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. AEA is celebrating Data Visualization and Reporting Week. The contributions all this week to aea365 come from members of AEA’s Data Visualization and Reporting Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Related posts:

1. Susan Kistler on Data Visualization
2. DVR Week: Amy Germuth and Johanna Morariu on Kicking off the DVR TIG Week
3. DVR Week: Stephanie Evergreen on the Data Visualization and Reporting TIG

I am Susan Kistler, the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Rad Resource –  American Evaluation Association’s LinkedIn Group: AEA’s LinkedIn group is open to members and nonmembers alike from around the world and currently welcomes over 11,000 colleagues to the community exchange.

Lessons Learned – What is LinkedIn: At its most basic level, LinkedIn is a website dedicated to bringing together those with a common interest into groups for discussion and networking. Joining is free and there are thousands of “groups” within LinkedIn, AEA’s being only one. Once you have set up a free LinkedIn account, you may then join up to fifty groups. Here are just a few that may be of interest to evaluators:

• African Evaluation Association (AfrEA) – focusing on colleagues living and working in Africa, this new site hosted by AfrEA just recently came online
• American Association of Public Opinion Research (AAPOR) – great discussions in particular for those conducting surveys or in need of sampling guidance
• Environmental Evaluators Network (EEN) – our colleagues at EEN use this open group as their primary forum for exchange around environmental evaluation issues
• Small and Medium Enterprise (SME) Evaluation – hosted by the World Bank’s Independent Evaluation Group, here you can discuss work with SME’s and also contribute to the group’s study of SME evaluation

Lessons Learned – What types of things are discussed on LinkedIn: LinkedIn discussions are threaded, meaning that a discussion stays together and may be contributed to over the course of many days or even weeks or months. Here are a couple of recent discussions on AEA’s LinkedIn Group (feel free to join in and add to the exchange):

• Marcela asked “I am looking for a five point scale to measure improvement. I am currently using: None, Slight, Some, Much, Huge, but my client doesn’t like it. Any suggestions?” and 35 responses offered suggestions, raised further questions, and refined thinking on the topic
• Mary Alice asked “Does anyone have any experience assessing changes in non-academic skills (attitude, resiliency, motivation, consciousness, grit, etc) in high school age youth. Preferably, I am looking for an assessment that could be used in a pre/post format…” and 23 posts contributed to the exchange
• Katherine asked “Anyone got suggestions for [affordable] stat packages that I might consider as an alternative [to SPSS and SAS] for my own consulting work?” and 24 people shared their expertise

Consider contributing or raising a question!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Susan Kistler on Tableau and Users Groups
2. Susan Kistler on Getting Engaged in Environmental Evaluation
3. Susan Kistler on Must Read Books for Evaluators

Hello! I am Harlan Luxenberg, the Chief Operating Officer of Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis specializing in public health evaluation. Over the years we have worked on multiple projects where we needed to do one or more of the following:

• report to a diverse set of stakeholders about program outcomes
• report on the comparability of outcomes across multiple projects or summative outcome targets
• create report cards or dashboard-like reports for multi-site evaluations

In our search for a powerful software suite that would allow us to efficiently do these tasks while producing visually appealing reports for our clients, we found Crystal Reports. Crystal Reports is not well known in evaluation circles since it is primarily used in the financial industry. But don’t let that fool you! We have been using it over the past eight years to provide reports to educational institutions, health care providers, and individual stakeholders.

(Share Clip)

We have found Crystal Reports to be one of the most useful programs in our data visualization toolkit. While the latest version (2011) does cost nearly $500, you can download a free 30 day demo or buy an earlier version online (like 2008) for less than $400. We have even used Crystal Reports in conjunction with LimeSurvey, an open source and completely free online survey tool. To see more about why we love using LimeSurvey and our experience using it in our evaluations, visit our blog posts on it here.

(Share Clip)

Hot Tip: Create a report template to save time and reduce the potential for errors. To create similar looking reports for different grantees that you are evaluating, simply put the data into a worksheet or database (like Excel). Then connect Crystal to your dataset and you’re ready to create an attractive grantee-specific report that looks similar across grantees. Each report can be developed to only use data from an individual grantee.

Hot Tip: It’s secure! When using fancy dashboard software or even Excel, you often have to give your clients access to your raw data. With Crystal Reports, you can export reports into various formats (like PDF), or your clients can access reports online or through a viewer (both very easy to do). This is especially useful if you are comparing one organization’s data against others and do not want to provide raw aggregate data to everyone.

Resource: If you’d like to see Crystal Reports in action, you can view a template for a standard grantee-specific report that we created for processing Olweus Bullying Prevention Program data.

Lesson Learned: Learning a new software program can be hard! With Crystal Reports, there are extremely helpful online forums where other users will help answer your questions. My favorite is Tek-Tips.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Sally Bond on Peer Review to Improve Evaluation Consulting Practice and Reports
2. Susan Kistler on Free/Low-cost Software for Evaluation for Nonprofits
3. DVR Week: Rakesh Mohan and Margaret Campbell on Making Evaluation Reports Reader Friendly

My name is Chris Lysy, I am a Research Analyst at Westat, the creator of Eval Central, and cartoonist at freshspectrum.  You can also find me on twitter (@clysy).

At first cartooning was just a hobby, but over time I’ve started to discover the utility of a well designed cartoon.  The right cartoons can be potent tools for introducing complex concepts, engaging audiences, and sparking discussion.

On my blog I’ve been experimenting with the fine line between chart junk and good illustration design.  A topic briefly addressed by Edward Tufte on his website.  Here are some tips I came up with to help you avoid crossing the line.

Hot Tip: 95% Concept, 5% Illustration. If you have a great concept, the illustration doesn’t have to be that great.  I spend the vast majority of my time on the concept.  Check out the following cartoon from my attribution post.  Not the best illustration, but the point is made.

 

Hot Tip: Be Abrupt. Subtlety thrives in other forms of communication.  You have a few seconds to capture attention, just say what you want to say.  Here’s an example from my What is Evaluation? post.

 

Hot Tip: Costello your Abbott. Visuals should always complement the point, not simply repeat the point.  In the famous sketch, Costello didn’t completely parrot Abbott, and your visuals should not completely parrot your points.  Here is an example from my counterfactual series.  My cartoon complements David Henderson’s point.  They work together.

“Outliers do not make for compelling client testimonials. Use your metrics to identify what the average experience in your program looks like, and get testimonials from people who fit this profile.”

 

Hot Tip: Push the Limits. Cartoons give you a certain latitude to go over the top.  When used wisely, pushing the limits can spark great discussion.  Here is one of my early cartoons from a post on how to not really evaluate.

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

1. Nancy Aguirre on Likert Scales
2. Jan Losby and Anne Wetmore on Likert Scales—Odd or Even?
3. Brian Silvey on Grade Points and Evaluation

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc., a small firm in Minneapolis, MN specializing in public health evaluation. Last year as I was writing a report, I had the burning desire to wrap a paragraph of text around a smaller chart. This is usually easy to do in Word, except I was using automatic captions for my chart titles so that my figures and tables were automatically numbered. Therefore, my chart titles were outside of my chart and did not travel well with the chart itself.

With a little Google search, I discovered a little known Word function that allows me to create a frame around a chart, its title, and anything else I want grouped together. The framed components can then be moved around as one object.

Hot Tip: I first had to add the framing function to my Quick Access Toolbar in Word. To do this (in Word 2010), go to File > Options > Quick Access Toolbar menu. Make sure that All Commands is selected in the drop-down menu, and then find Insert Frame from the list of commands. Select Add to add this function to your Quick Access Toolbar and click Ok. Now a bright green circle should appear in your Quick Access Toolbar (located along the top left corner of your screen).

When you want two or more components framed together, such as a chart and its title, first highlight the components with your mouse and then click the green Format Frame circle. A frame will now appear around the selected components.

I recommend changing the color of the frame from black to white. To do this, right click on the frame and select Borders and Shading. Then select a white border. Move the frame to adjust its size and place it in the desired location.

Here is an example of a framed chart and caption next to a paragraph of text:

Lessons Learned:

1. The components that you frame still maintain their original functions. Automatic captions still automatically update. Charts can still be edited.
2. You can always remove a frame if you decide it’s not needed. Simply right-click on the frame, select Format Frame, and click Remove Frame.
3. Avoid having one line of text lingering above or below a frame. I always align my frame along the right side of text and then either align it with the first or last line of the paragraph.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Susan Kistler on a Fast and Free Option for Creating a Webpage from a Word Document
2. Susan Kistler on aea365 on your Cell, Bookmarking Keyword Searches, & Eval 2010 Prep
3. Susan Kistler on Working With Photos

I’m Clara Hagens. I work for Catholic Relief Services (CRS) as the Regional Technical Advisor for Monitoring, Evaluation, Accountability and Learning (MEAL) in Asia. I’d like to share with you a resource pack we have developed to support teams to develop and implement monitoring and evaluation (M&E) systems throughout different phases of an emergency response.

Rad Resource: CRS’ Monitoring, Evaluation, Accountability and Learning in Emergencies: A Resource Pack for Simple and Strong MEAL provides guidance on the key principles of MEAL in emergencies and helps staff to design use-oriented M&E systems that collect just enough information to inform high quality and highly-responsive emergency programming. The Resource Pack clarifies what is different about M&E in an emergency versus a non-emergency setting, namely that in an emergency the M&E system must remain dynamic, that gathering the perspective of the most vulnerable groups is valued over rigorous or heavy data collection methods, that teams are responsible for collecting information on changes context, and that results must be used immediately and often daily for the response to continue to meet the community’s needs.

The Resource Pack demonstrates how both information needs and the appropriate mix of methods and respondents to meet those information needs evolves during an emergency response. Please see the figure below for a graphic depicting this evolution.

(Reprinted by permission from Dominique Morel and Clara Hagens, Monitoring, Evaluation, Accountability and Learning in Emergencies: A Resource Pack for Simple and Strong MEAL (Baltimore, MD: Catholic Relief Services, 2012), [page 14].)

The Resource Pack also emphasizes the importance of monitoring community satisfaction with the response and developing feedback mechanisms to increase accountability.  Additional topics in the Resource Pack include sampling during data collection (both when and how to sample), conducting a daily debrief session, and selecting an appropriate mix of learning events at different points in the response.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Clara Hagens on Guidance on Monitoring and Evaluation
2. Poster Week: Shobha Mittal on Seeking a Higher Survey Response Rate
3. Irene Marconi on 10 Great Twitter Feeds for Evaluators

Hi. We are Kathy Muhr, Aniko Laszlo and Alexis Henry from UMass Medical School’s Center for Health Policy and Research. Through Work Without Limits, a Massachusetts network of employers and providers that aims to increase employment for people with disabilities, we evaluate and promote programs, policies, and practices related to recruitment, retention, accommodation, and career advancement of people with disabilities. One of our efforts has been the development of Regional Employment Collaboratives (RECs), which bring together cross-disability employment service providers to more effectively engage employers and identify ways of producing better employment outcomes.

We conducted a process evaluation of the RECs, asking the question “what does it take to build and sustain a collaborative of disability employment service providers?” using a concept mapping approach. Concept mapping is a participatory, multi-stage, mixed-method approach that, among its many uses, involves stakeholders in describing how programs are developed and implemented. To conduct the evaluation, we invited REC members to participate in “brainstorming” sessions during which they generated statements in response to the above question. Next, members sorted and categorized the statements into similar groups. We then entered the sorted statements into a concept mapping software program, which uses multidimensional scaling and hierarchical cluster analysis to generate a visual representation – a concept map – of how the statements were grouped by the participating members. As a final step, we reviewed our findings with the REC Project Directors, who helped us interpret the data, determine a final number of clusters, and generate meaningful labels that captured the concepts the clusters represented.

The concept map shows the relationship of clusters to each other – clusters closer together on the map represent concepts that are more closely related and those further apart represent concepts less closely related. This approach revealed concepts that were related to the necessary ingredients for building collaboratives in general, as well as concepts that were specific to building collaboratives focused on enhancing employment for people with disabilities. We feel that the concept mapping approach was very effective in getting the first-hand perspectives of the stakeholders involved in building the RECs, and provided us with some strategies for further development and replication of the REC model.

Lesson Learned: Mind the learning curve.Make sure you allow enough time to learn your concept mapping software, and to complete all stages in the concept mapping process.

Lesson Learned: Beware of concept mapping software overload. The Internet provides an extensive list of concept mapping software; some are free and others are not. It is important to select the software that best meets the needs of your project.

Rad Resources: Examples of concept maps and various concept mapping software products.

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Monica Hargraves on Supporting Evaluation Practice in Organizations
2. Mary Kane and Scott Rosas on Leveraging Concept Mapping
3. John Branch on Concepts

My name is Linda Meyer and I am an applied developmental psychology graduate student at Claremont Graduate University. I am going to provide you with a few basic tips to keep in mind as you are planning an educational evaluation with English Language Learners (ELLs) in PK-12.

Hot Tip: Find out how students were labeled as English Learners. Most students get labeled through a similar process: a home survey sent to parents upon enrollment, followed by a standardized test on English language development, and a score on that test that falls below what has been designated “proficient.” However, policies are often applied inconsistently and inappropriately. These tests may be inappropriate for young students and the school’s criteria may over- or under-identify English Learners. These students then receive inappropriate instruction that negatively impact their academic performance and development. Lawsuits against various states (see Arizona) demonstrate the widespread problems that stem from inappropriate identification and instruction. Before you evaluate the quality of instruction, make sure students are correctly classified and that the instruction matches students’ abilities.

Hot Tip: Consider English Learners’ ages and language exposure. Children at the elementary level are more likely to develop proficiency in a second language than older students because language acquisition ability trails off at about age 12. Students who fail to sufficiently develop English as a second language after a period of years are designated Long-term English Learners and are at higher risk of dropping out of school, among other things. Students who are not exposed often enough to their second language are less likely to become proficient in that language. Consider how often students hear and use English each day. Higher exposure at earlier ages increases the likelihood that they will become English proficient.

Hot Tip: Match your methods to the instruction type. Some instruction types emphasize learning English as quickly as possible, sometimes at the cost of grade level content instruction time. Others emphasize keeping up with the grade level content in whichever language is more proficient, with some moving students toward English-only instruction over time and others moving students toward proficiency in both languages for all content. Each has its strengths and weaknesses, so match your evaluation questions and methods to the instruction type to obtain the most relevant data.

Rad Resource: A comprehensive national study on English Learners’ academic achievement that investigated various instruction types.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. CREATE Week: Paula Egelson on Mini-Case Studies in Formative Assessment
2. Mary Talbut on Testing What You Teach
3. Susan Kistler on Going Global With aea365

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Last week, I saw the new Star Trek movie (yes, my inner geek shines through once again). Part of its premise was that a team had been asleep for years, only to awake and wreak havoc on the Star Trek crew. You would never do that, you would draw on the past to improve the future of course because evaluators are a good lot. That is why today I’m sharing information about the Wayback Machine.

Rad Resource – The Wayback Machine: The Wayback Machine allows you to look at the history of a website over time. You visit the Wayback webpage, enter the URL for the site in which you are interested, and you can see how the site looked historically, with most of the formatting and the links intact.

Clipped from: archive.org (share this clip)

 

Here’s a screenshot of the AEA website in February of this year:

And in August of 2004:

And in December of 1998:

If you are were exploring it on the Wayback Machine website, you’d be able to try out most of the links and read the content of each page.

Why is it useful? This is a great way to do background research on your evaluand, to understand issues of explore change over time as reflected in their web presence. If your evaluation includes an examination of your evaluand’s communications, the Wayback machine can also help you see how those communications played out online.

There are a few caveats – formatting is better for some sites than for others, and you can’t see a snapshot of every day in history for a site, only at times when the web crawler used by the Internet Archive visited the site.

The Wayback Machine takes all of 2 minutes to use and explore. And, its even a bit of fun, in particular if you are an historian at heart.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association.

DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>


















UnhideWhenUsed="false" QFormat="true" Name="Title"/>

UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
UnhideWhenUsed="false" Name="Table Grid"/>

UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
UnhideWhenUsed="false" Name="Light Shading"/>
UnhideWhenUsed="false" Name="Light List"/>
UnhideWhenUsed="false" Name="Light Grid"/>
UnhideWhenUsed="false" Name="Medium Shading 1"/>
UnhideWhenUsed="false" Name="Medium Shading 2"/>
UnhideWhenUsed="false" Name="Medium List 1"/>
UnhideWhenUsed="false" Name="Medium List 2"/>
UnhideWhenUsed="false" Name="Medium Grid 1"/>
UnhideWhenUsed="false" Name="Medium Grid 2"/>
UnhideWhenUsed="false" Name="Medium Grid 3"/>
UnhideWhenUsed="false" Name="Dark List"/>
UnhideWhenUsed="false" Name="Colorful Shading"/>
UnhideWhenUsed="false" Name="Colorful List"/>
UnhideWhenUsed="false" Name="Colorful Grid"/>
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
UnhideWhenUsed="false" Name="Light List Accent 1"/>
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>

UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
UnhideWhenUsed="false" Name="Light List Accent 2"/>
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
UnhideWhenUsed="false" Name="Light List Accent 3"/>
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
UnhideWhenUsed="false" Name="Light List Accent 4"/>
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
UnhideWhenUsed="false" Name="Light List Accent 5"/>
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
UnhideWhenUsed="false" Name="Light List Accent 6"/>
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

Related posts:

1. Susan Kistler on the AEA/BetterEvaluation Webinar Series and Making the Most of YouTube
2. Stephen McFarland on More Ted Talks for Evaluators
3. Susan Kistler on Who’s Linking to Your Website?

My name is Tanya Ostrogorsky, Assistant Vice Provost for Assessment and Evaluation at Oregon Health & Science University, and I’ve been involved with Oregon Program Evaluators Network since 2002.  I ‘grew up’ studying research methods and data analysis and looking back I was a functioning as an evaluator before I knew what that meant. It wasn’t until my doctoral program that I took my first program evaluation course and attended an OPEN conference. Since then I’ve held leadership positions on seven different occasions including a long stretch as OPEN President during a difficult time in the organization’s history.

The purpose of this post is not to tell you about my trajectory as a local affiliate leader, but to share lessons learned through my observations about the role and function of the local affiliates in supporting AEAs mission. I also want to remind us how critical the local affiliates are to the development of local talent as well as the national leadership pipeline. Finally, I want to highlight the under-realized sources of energy, excitement, and real diversity that are in our midst.

Recently, 126 conference attendees ranging from students to newly minted graduates to early careerist to long-timers gathered to hear about the Top 10 Trends in Evaluation with Dr. Stewart Donaldson.  My first reaction to that day was a strong sense of pride in watching a local affiliate consistently deliver significant professional development opportunities for 16 years. My second reaction, as I scanned the room, was on the diverse and exciting mix of attendees that represent our past, our present, and our future.

So, what’s my point? Just as AEA needs to leverage and develop the local affiliates, past local affiliate leaders need to ensure the next generations of evaluators are provided the organizational history and encouragement to pick up where we left off. In both cases, we have a professional responsibility to support and encourage our peers in taking the next step in their leadership development. We need to offer encouragement and harness their energy. Yes, they will stumble and they will re-create the wheel, but so did we.

Lesson learned: We must leverage the talents and energy of the local affiliates to develop the leadership pipeline needs. My hope is that AEA can bring their focus to the power of local affiliates to create a strong organizational legacy. At the same time, it is local affiliate leadership responsibility to ensure that we do our part and have a strong community to support AEA.

Hot Tip: Local/regional AEA affiliates offer many opportunities to build our evaluation community. Find yours here and take the next step!

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. OPEN Week: Kim Firth Leonard on Local Affiliates as Learning Communities
2. Susan Kistler on Local Affiliates and Managing Professional Obligations
3. CEA Week: Leah C. Neubauer on Adult Education, Planning for Learning & AEA 2012

I am Elizabeth O’Neill, Program Evaluator for Oregon’s State Unit on Aging and President-Elect for the Oregon Program Evaluators Network. I found myself on this unlikely route as an evaluator starting as a nonprofit program manager. As I witnessed the amazing dedication for producing community-based work, I wanted to know that the effort was substantiated. By examining institutional beliefs that a program was “helping” intended recipients, I found my way as a program evaluator and performance auditor for state government.  I wanted to share my thoughts on the seemingly oxymoronic angle I take to convince colleagues that we do not need evaluation, at least not for every part of service delivery.

In the last few years, I have found tremendous enthusiasm in the government sector for demonstrating progress towards protecting our most vulnerable citizens. As evaluation moves closer to program design, I now develop logic models as the grant is written rather than when the final report is due. Much of my work involves leading stakeholders in conversations to operationalize their hypotheses about theories of change. I draw extensively from a previous OPEN conference keynote presenter, Michael Quinn Patton, and his work on utilization-focused evaluation strategies to ensure evaluation is intended use by intended users. So you think I would thrilled to hear the oft-mentioned workgroup battle cry that “we need more metrics.”  Instead, I have found this idea to warrant more naval-gazing and less meaningful action.  I have noticed how metrics can be developed to quantify that work got done, rather than to measure the impact of our work.

Lesson Learned: The excitement about using metrics stems from wanting to substantiate our efforts and to feel accomplished with our day-to-day to activities. While process outcomes can be useful to monitor, the emphasis has to remain on long-term client outcomes.

Lesson Learned: As metrics become common parlance, evaluators can help move performance measurement to performance management so the data can reveal strategies for continuous improvement. I really like OPEN’s founder Mike Hendricks’ work in this area.

Lesson Learned: As we experience this exciting cultural shift to relying more and more on evaluation results, we need to have cogent ways to separate program monitoring, quality assurance and program evaluation.  There are times when measuring the number of times a workgroup convened may be needed for specific grant requirements, but we can’t lose sight of why the workgroup was convened in the first place.

Rad Resource: Stewart Donaldson with the Claremont Graduate Institute spoke at OPEN’s annual conference this year with spectacular response. Program Theory-Driven Evaluation Science: Strategies and Applications by Dr. Donaldson is a great book for evaluating program impact.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Tanya Ostrogorsky on Joining Your Local Affiliate
2. OPEN Week: Kim Firth Leonard on Local Affiliates as Learning Communities

My name is Kelly Smith, and I’m an evaluator, policy analyst and economist at ECONorthwest, a consulting firm in Portland, Oregon. I’ve been a devoted member of OPEN (Oregon Program Evaluators Network), an AEA affiliate, since I was introduced to it in graduate school in 2007. OPEN works hard to provide our members with a variety of learning opportunities, including brown bag lunch talks, workshops, networking events, conferences, and a book club. We do this mostly by tapping into our own members’ expertise. Until recently, I’d been instrumental in planning these events, and I’d certainly been an enthusiastic participant, but I hadn’t yet felt comfortable leading one.

This past year, I overcame my reticence and agreed to give a talk about cost analysis, a topic that seems to raise fear and confusion, if not hackles, among many evaluators. The event was free and the room was packed, far exceeding our expectations. Evidently, people were hungry for knowledge about this subject. Some attendees even expressed surprise that we didn’t charge for such a valuable class (lesson learned!). Having sensed a large untapped demand, we decided to offer two sessions about cost analysis at our annual conference in the spring, and both were well attended. We got great feedback about the usefulness of this topic.

These sessions were my first experiences “teaching”, and I found it both intellectually challenging and valuable (and if I’m honest, a bit nerve-wracking). I had to study the topic to refresh my knowledge, practice public speaking and presentation skills, and think on my feet. I’m confident that I got more out the experience than the attendees.

Get involved! If you’re a member of your local affiliate, don’t hesitate to step up and contribute your own knowledge! There is a real demand for learning opportunities, and certainly not enough supply. Your colleagues will benefit from your willingness to share, your affiliate will become stronger and more active, and you’ll come out ahead, too.

It’s hard to understate the value of learning from each other as evaluators. Not only do we grow as professionals when we contribute to our colleagues and our affiliate, but we expand the value and reach of the evaluation profession as a whole.

Rad Resource: If you are interested in learning more about cost analysis in evaluation, you’ll find a copy of the slides used at the conference sessions here.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. OPEN Week: Kim Firth Leonard on Local Affiliates as Learning Communities
2. CEA Week: Leah C. Neubauer on Adult Education, Planning for Learning & AEA 2012
3. OPEG Affiliate Week: Rob Fischer on an Effective Board

My name is Adrienne Zell and I work as an internal evaluator for the Oregon Clinical and Translational Research Institute, an organization that provides services to biomedical researchers at Oregon Health and Science University. I also volunteer as the executive director of a small nonprofit, Impactivism, which provides evaluation advice to community organizations. I have been a member of OPEN for over 10 years, and involved with the events committee for the past three years.

Many years ago, I was lent a collection of essays entitled, Yoga for People Who Can’t Be Bothered to Do It. Geoff Dyer’s essays are first-rate — humorous, amorous, and reflective – but it is his brilliant title that has stuck with me. Although OPEN members and volunteers have diverse roles within the field of evaluation, a common theme in our events and conversations has been the effort involved in convincing organizational leadership, staff, and stakeholders that evaluation is worth doing and that they should have a direct role.

This past year, one of our members, Chari Smith, successfully organized an OPEN event and a conference workshop designed to planfully connect evaluators and nonprofit staff and engage them in thinking about reasons why organizations may not “do” evaluation. As evaluators, we rarely can remove all identified barriers. But we can work to understand their complexity and re-focus on opportunities. Participation in OPEN, along with my experience as both an external and an internal evaluator, has inspired a list of tips on addressing evaluation gridlock in organizations and just “doing” it.

Hot Tip #1: Highlight current capacity. Most organizations are already practicing evaluation; they just aren’t using the term. They may collect data on clients, distribute feedback forms, maintain resource guides, or engage in other evaluation-related activities. Identifying and leveraging current accomplishments inspires confidence and makes evaluation seem less forbidding.

Hot Tip #2: Appeal to accountability. Program leaders, by definition, should be held accountable for program impact. The most recent issue of New Directions for Evaluation compares and contrasts the fields of performance management and evaluation. Program managers should regularly request and utilize both kinds of information when making decisions. Elements of these comparisons can be shared with program leadership, increasing understanding about the differences, commonalities, and benefits.

Hot Tip #3: Show them the money. Provide examples of how rigorous impact evaluation can result in stronger grant applications and increased funding. A recent EvalTalk post solicited such an example, and members were responsive. In addition, return on investment (ROI) and other cost analyses (see tomorrow’s post by OPEN member Kelly Smith) can demonstrate savings, inform resource allocation, and target areas for future investment.  A single ROI figure can “go viral” and motivate further evaluation work.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

1. OPEN Week: Erin Stack & Lindsey Patterson on Successfully Transitioning from Student to Professional
2. OPEN Week: Kim Firth Leonard on Local Affiliates as Learning Communities
3. Tanya Ostrogorsky on Joining Your Local Affiliate

Hi. We are Erin Stack and Lindsey Patterson. Erin is a Community Psychology doctoral student at Portland State University (PSU) and the current student liaison for the Oregon Program Evaluation Network (OPEN). Lindsey is a former OPEN student liaison and a soon-to-be graduate of the PSU Community Psychology program. Lindsey currently works at the Center for Improvement of Child and Family Services. Since Lindsey just transitioned into the workplace and Erin is about to, we thought it would be helpful to other transitioning students and new evaluators to share what we’ve learned about effectively transitioning from student to professional.

Hot Tip # 1: Identify and work with mentors. At PSU, we work closely with faculty advisors in a mutually beneficial capacity.  We have also found it helpful to develop relationships with other students and professionals within and outside of our program. These mentorships have resulted in research methods book clubs to help stay up-to-date on current statistical trends, publication and internship opportunities, and skills to navigate the ups and downs of graduate school and career preparation.

Hot Tip # 2: Network. Relatedly, we have found it helpful to always expand our professional networks by attending happy hours and social events sponsored by a variety of organizations (including OPEN). As a student, networking can be intimidating, but with an increasingly competitive job market, it is important to build relationships with hard-working contributors in the field that could one day be colleagues or collaborators.

Hot Tip # 3: Attend conferences. Conferences provide opportunities to learn about current happenings in the field and about career trajectories in evaluation, and to develop meaningful professional relationships through networking. Conferences have the built-in perk of providing the topics of conversation for you! For example, at the OPEN conference this past March in Portland, Oregon, Stewart Donaldson kicked us off with what he believed were the future trends in Program Evaluation. This lends us an exciting and easy topic to debate and to continue generating ideas with other conference attendees.

Hot Tip # 4: Get involved in local associations. We both volunteer for OPEN. This has been helpful for cultivating relationships with evaluators in the area, identifying diverse career options related to Program Evaluation, and work with a group of individuals who share a similar passion.

Rad Resource:  Early Career Listservs and Topical Interest Groups. Some organizations have dedicated listservs and/or topical interest groups that graduate students and new evaluators can join.  For example, the Society for Community Research and Action (SCRA) has created a listserv for early career individuals.  These resources can provide additional connections, job postings, and new or innovative ideas related to career opportunities.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

1. Viola, Patterson, Gray, and Boal on Reflecting on Students’ Self-Assessments of Knowledge Gained through a Program Evaluation Graduate Seminar
2. Sarah Hug on Evaluating STEM Pipeline Programs
3. Aubrey Perry on Ensuring a Positive Practica Experience

My name is Kim Firth Leonard, and I have the honor of authoring the first post on the aea365 blog for Oregon Program Evaluators Network (OPEN) week. I have been an AEA member since 2008, and am currently President of OPEN, a local affiliate of AEA founded in 1997. I work as Assessment Research Coordinator at Marylhurst University in Portland Oregon and do contract work in program evaluation via Leonard Research and Evaluation, LLC.

This week’s posts were by OPEN members who have played important volunteer and leadership roles for OPEN. The posts demonstrate the value of our local network by sharing lessons we’ve gathered in reflecting on our work together as evaluators and as volunteers with OPEN.

I have learned much about evaluation and about building learning communities through OPEN. The bulk of the work done by OPEN’s volunteer Council and Committees is in organizing and supporting local events. OPEN’s mission is to provide a regional, interdisciplinary forum for professional development, networking, and exchange of practical, methodological, and theoretical knowledge in the field of evaluation. It is through these events that we build learning communities, and in doing so strengthen our work individually, and as a field.  

Get Involved: Whether you have a local affiliate or just an informal network of other evaluators in your area, you too can host, lead, contribute to, or benefit from local evaluation events.

• Host: Events don’t have to be massive undertakings to be successful. Small, informal gatherings can be just as valuable as large conferences. “Have an idea? Go for it” is practically our events committee motto.
• Lead: Local events are great places to practice your presentation and training skills. Discussion groups, like OPEN’s new-ish Book Club are low-pressure and offer opportunities to discuss emerging topics.
• Contribute: Volunteer to help organize events for unique networking opportunities. Learning event planning skills is icing on the cake.
• Benefit: It’s all about learning together. Valuable learning about one another and the field can happen at any get together – so attend local events whenever you’re able.

Lesson Learned: OPEN has always been welcoming to community members who don’t identify as evaluators, exactly, but do related work or want to learn more about evaluation. In the last year or so we’ve been emphasizing this openness (ha!) and we’ve found that collaborating with and learning from others in related fields greatly enriches our evaluation learning community. Sessions at our recent conference intended to create opportunity to learn from and with others in our community, including non-profit leaders, were well received.

Rad Resource: Materials from our 2013 conference are available on our website.

Rad Resource: Your own learning community is at your local affiliate or among other local AEA members.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No related posts.

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Rad Resource – Juice Analytics Blog: The staff at Juice Analytics puts out one of my favorite blogs focusing on data use, analysis, and visualization. Their tag line alone “Your data is meant for action” is enough to get my vote.

Lessons Learned – Everyday Visualizations and Everyday Evaluations: This week, Ken Hilburn wrote about “Everyday Visualizations” – the laundry on the line that tells you that rain isn’t predicted for today, or the indicators on your laptop that tell you when your battery is about to die.

When I teach evaluation, I usually begin by noting that we are all evaluators. We collect data, analyze that information, and make decisions based on it. We read movie reviews and choose a show to see, peruse the meat counter at the grocery and perform a quick cost-benefit analysis and put down the tenderloin, and select a mate based on what might be seen as extensive interviews and evidentiary analysis. Professional evaluators merely increase the scope and systematization of the processes and apply them in situations with broader implications.

Lessons Learned – Identifying Life’s Key Indicators: My family lives at a boarding school where my husband chairs the science department, and (barring the extremely unexpected) from which my oldest daughter Emily will graduate this afternoon

As I sat through a baccalaureate service last night, my mind wandered to the mental calculus of the success criteria for the situation. She completed the program. Check. Did well, graduating cum laude. Check. Got into college to study computer engineering. Check. These are all indeed successes that have made me very proud.

Yet success over the past 18 years is far more difficult to quantify. A new tradition here is that, as a child of a faculty member, my husband will hand my daughter her diploma. To plan ahead, he asked “hug or handshake” – hug of course she exclaimed, excited at the prospect. I sat next to the headmaster’s wife at the service. She noted that Emily looked beautiful at the prom, no small feat for a child who had seen major health challenges. Earlier in the day, a friend had commented that Emily positively beams when standing beside her longtime boyfriend. Yes, said I, she is in love.

Ultimately, are these not what matter? Success is made manifest in health and happiness, confidence that you are loved and the capacity to love with others.

As I change my own work situation, I am setting goals for myself, looking at what should be my own key indicators of success, personally and professionally, in the coming years. If you have experience to share, I’d love to learn from you.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

1. Alice Hausman on Measuring Community -Defined Indicators of Success
2. René Lavinghouze on Sharing Your Program’s Story
3. Bloggers Series: Kevin Flora on the EdMatics Blog