American Evaluation Association 365 Blog

Syndicate content
A Tip-a-Day by and for Evaluators
Updated: 18 sec ago

GSNE Week: Kate Westaby and Valerie Moody on How to Hit the Ground Running in your New Evaluation Position!

10 hours 43 min ago

Greetings! We are Kate Westaby and Valerie Moody, new evaluators from two Clinical and Translational Science Award (CTSA) institutes. Kate is an Evaluation Research Specialist at the University of Wisconsin-Madison Institute for Clinical and Translational Research and Valerie is the Evaluation Coordinator at the University of Iowa Institute for Clinical and Translational Science. At the 62 CTSA institutes nationwide, program evaluation is a complex, dynamic, unpredictable environment, mandated by NIH, but implemented in a wide variety of ways by evaluators with diverse backgrounds.

Due to our personal efforts learning to adapt to these complicated surroundings, we wanted to know if there were best practices for new evaluators to orient themselves to their workplaces. Last year, we interviewed 16 new evaluators from 14 CTSA institutes to gather the most helpful strategies for learning about evaluation, thus allowing new evaluators to hit the ground running.

I felt it was like putting together a 1000 piece puzzle, but nobody gave you the cover,” — quote from a new CTSA evaluator.

Hot Tip 1: Learn the history of evaluation efforts at your workplace.New evaluators found this to be the most helpful strategy. Many suggested using programmatic documents (e.g., grant proposals, strategic goal documents, etc.) to find useful historical information. They were better able to understand evaluation needs and review progress towards those needs in a short period of time.

Hot Tip 2: Attend face-to-face meetings (or a conference) with evaluators who are doing similar work. This setting allowed new evaluators to hear what strategies others are using, what their struggles have been, and how they turned their struggles into successes. It also allowed them to establish face-to-face networks for future communication.

Hot Tip 3: Ask questions! Supervisors or colleagues can provide insight into program history, politics, and help you avoid reinventing the wheel. Don’t be afraid to speak up!

Rad Resource: For more tips on how to get comfortable in your new workplace or to look into which strategies were least helpful to our interviewees, check out our AEA 2013 poster below (or download a larger version from AEA’s public elibrary here).

AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Boris Volkov on What (Internal) Evaluators Can Do to Advance Evaluation Capacity Building
  2. OPEN Week: Adrienne Zell on Evaluation for People Who Can’t Be Bothered to Do It
  3. BLP TIG Week: Carla Forrest on Using Appreciative Approaches to Drive Workplace Performance

GSNE Week: Alice Walters on The Art and Science of Networking

Mon, 07/21/2014 - 01:15

I’m Alice Walters, a member of AEA’s Graduate Student and New Evaluator TIG.  I am a doctoral student in human services and work as a non-profit consultant in fund development, marketing, and evaluation.  I share some networking tips, below.

Networking is needed at every career stage.  Review tips and resources to increase your effectiveness.  Enjoy using your networking skills as both art and science to see what serendipitous outcomes transpire!

Hot Tip 1:  Networking is developing informal connections with other professionals.

Building informal connections can occur any time you meet other professionals.  Don’t exclude those outside your usual networks who can be a source of unexpected developments.

Rad Resource: Developing a Strong Professional Network” by the Penn State Alumni Association 

Hot Tip 2:  Networking is more than just about a job hunt. Networking is often associated with job hunting success but it can be much more than that.  Networking can lead you to new avenues, develop new collaborations, and bring attention to your own work in new venues.

Rad Resource: Tips for Successful Business Networking10 Advantages of Business Networking” bySusan M. Heathfield

Hot Tip 3:  Networking is not really an “activity,” it is a lifestyle. Networking is not an isolated activity you add to your calendar.  Instead, it is really a process, approach, and outlook on professional relationships.

Rad Resource: Cheat Sheet: 9 Professional Networking Tips” by Jillian Kurvers

Hot Tip 4:  Networking for the shy – is easier when you don’t think of it as “networking.” Even the most outgoing people can struggle with pressure to force a connection professionally.  Instead, it is better to explore relationships by asking questions that occur naturally to you.

Rad Resource: How to Network: 12 Tips for Shy People” by Meridith Levinson

Hot Tip 5:  Networking is an art.  It’s creative, flexible, and individualistic. Use your strengths to network.  Just as art appeals differently to individuals, networking can accommodate a variety of styles.

Hot Tip 6:  Networking is a science.  It deserves study and analysis. Science is study.  Networking is thoughtful.  It seeks to connect the random dots.  Networking requires analysis of input data.  It’s not an oxymoron to look for serendipity.  Serendipity is defined as finding something valuable but not sought for.  Still, if you are looking for connections and value, you will be more likely to find them.

Hot Tip 7:  AEA is a great resource for networking. AEA is the hub of evaluation professionals.  The AEA Topical Interest Groups, conferences, and local affiliates are a great place to start. On the AEA home page go to (third tab to the right):  Read>Links of Interest>Professional Groups   http://www.eval.org/p/cm/ld/fid=69

AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. OPEN Week: Erin Stack & Lindsey Patterson on Successfully Transitioning from Student to Professional
  2. Tamara Bertrand Jones on Finding and Working With a Mentor
  3. WE Week: Nick Hart on the Value of Affiliate Connections with the Academic Community

GSNE Week: Ayesha Tillman on Graduate Student and New Evaluator (GSNE) TIG Mentorship program

Sun, 07/20/2014 - 03:38

Hello, I am Ayesha Tillman, and I have all but deposited my dissertation for a Ph.D. degree in Educational Psychology with an evaluation specialization from the University of Illinois at Urbana-Champaign (Illinois). I along with Rae Clementz, Sarah Wilkey-Gordon, Tiffany Smith, Pat Barlow, and Nora Gannon-Slater are mentors in the Graduate Student and New Evaluator (GSNE) TIG Peer-Mentorship program. I have five mentees located across the U.S. including Louisiana, California, Michigan, Texas, and the Dominican Republic. So far, my participation as a mentor has been an incredibly rewarding and worthwhile endeavor.

Lessons Learned: All of my mentees joined the GSNE TIG peer-mentoring program because they were looking for someone to bounce ideas off of, share experiences with, and someone to give them tips/advice. Below is advice I have shared.

Hot Tip 1: Presenting. AEA, American Educational Research Association (AERA), and the Center for Culturally Responsive Evaluation and Assessment (CREA) are three conferences for evaluators to submit presentation proposals to. If you are uncomfortable submitting a paper, start with roundtable and poster presentations.

Hot Tip 2: Publishing. Publishing in evaluation can be tricky. Evaluation journals (and conferences) do not want submissions about the results of the evaluation. Research on evaluation and reflections on evaluation practice are well suited for publication. For example, the American Journal of Evaluation “explores decisions and challenges related to conceptualizing, designing and conducting evaluations.”

Hot Tip 3: Capacity building. The workshops at the AEA conference, the AEA summer evaluation institute, and AEA eStudies are great professional development opportunities for evaluators. The AEA Graduate Education Diversity Internship Program is an awesome opportunity for graduate students of color and from other under-represented groups who would like to extend their research capacities to evaluation.

Rad Resources:

  • GSNE mentorship program mentees. If you are interested in being a mentee, make sure you are a member of the GSNE TIG. You will receive an email once a quarter with the opportunity to become a mentee. If you are interested in being paired with a GSNE mentor, please send an email to Kristin Woods.
  • GSNE mentorship program mentors. If you are interested in being a mentor, you should have been an AEA member for two or more years and have attended at least one annual conference. If you are interested and willing to be a GSNE mentor, please send an email to Kristin Woods.
  • GSNE TIG Facebook page. The GSNE Facebook group is a great place to connect with other graduate student and new evaluators. We share resources, opinions, advice, and network on Facebook.

AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. GSNE Week: Kristin Woods on Gaining Practical Experience as a New Evaluator
  2. Tamara Bertrand Jones on Finding and Working With a Mentor
  3. STEM TIG Week: Ayesha Tillman on Advice to New Evaluators

Sheila B Robinson on EvalYear: A Taste of 2015 and a bit of Alphabet Soup to Whet Your Appetite

Sat, 07/19/2014 - 06:38

Good morning! I’m Sheila B Robinson, aea365′s Lead Curator and sometimes Saturday contributor. I love to share evaluation news and, ever the teacher, I look for opportunities to educate aea365 readers. Today’s lesson is about EvalYear, the International Year of Evaluation. If you haven’t yet heard about this, it’s time to get reading!

In October 2013 in São Paulo, Brazil at the Third International Conference on National Evaluation Capacities it was announced that 2015 would be the International Year of Evaluation (EvalYear). EvalPartners, the global movement to strengthen national evaluation capacities, is behind the effort, and it’s a big effort! Two leading partners and 47 core partners (of which AEA is one) along with 1580 evaluators/activists have joined or expressed interest in the declaration of EvalYear.

“The aim of designating 2015 as the International Year of Evaluation is to advocate and promote evaluation and evidence-based policy making at international, regional, national and local levels.”

Image credit: Hans Watson via Flickr

Lesson Learned: When you visit the EvalYear website and start reading, you will come across no fewer than 15 acronyms! Most are written out, but not all are (or are not written out on every page), so to prepare, have a little taste of alphabet soup!

  • IOCE – International Organization for Cooperation in Evaluation
  • IEG – Independent Evaluation Group
  • OECD/DAC – Organization for Economic Cooperation and Development / Development Assistance Committee
  • VOPE – Voluntary Organization of Professional Evaluators
  • MDGs – Millennium Development Goals
  • SDGs – Sustainable Development Goals
  • UNEG – United Nations Evaluation Group
  • ECG – Evaluation Cooperation Group
  • ALNAP – Active Learning Network for Accountability and Performance 
  • TF – Task Force
  • NECD – National Evaluation Capacity Development
  • QCPR - Quadrennial Comprehensive Policy Review
  • CSO – Civil Society Organization
  • ECD – Evaluation Capacity Development
  • EFGR - Equity Focused and Gender Responsive

“EvalYear will position evaluation in the policy arena, by raising awareness of the importance of embedding monitoring and evaluation systems in the development and implementation of the forthcoming Sustainable Development Goals, and all other critical local contextualized goals, at the international and national levels. EvalYear is about taking mutual responsibility for policies and social action through greater understanding, transparency, and constructive dialogue.”

Hot Tip: Visit EvalYear to learn more and consider how you will get involved!

Rad Resources: Check out the resource center for presentations and updates. The EvalYear logo and brochure is currently available in 18 languages and it is being translated in to many more languages.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Neha Karkara on the Evaluation Advocacy Toolkit
  2. Susan Kistler on VOPEs and the EvalPartners Innovation Challenge
  3. Amir Fallah on Resources for International Evaluators

LAWG Week: Amber Hill on Boosting Online Survey Participation

Fri, 07/18/2014 - 01:15

Hello! My name is Amber Hill and I am a research specialist at McREL International’s Denver, Colorado office. My work focuses on education research and my responsibilities include managing online surveys administered to state departments of education, districts, school staff, parents, and students in the United States, Pacific region, and Australia. Encouraging online survey participation can be tricky, which is why I use a variety of methods.

Hot Tip – Work with the IT Pros

Ensuring that participants receive the survey in the first place can be half of the battle. No matter your level of information technology (IT) expertise, it is helpful to coordinate efforts between the IT pros who work for your survey software provider, your own organization, and the organization for which you are administering the survey. Those three groups can help you with white listing, test emails, firewalls, and broken links.

Hot Tip – Communicate Early, Communicate Often

Participants are often leery of participating in a survey administered by a stranger, especially if the content is sensitive. Working with a partner organization that is familiar to participants helps increase understanding about the purpose, value, and trustworthiness of the survey and evaluator. Partner organizations may send an e-mail to participants with the evaluator’s name and contact information in advance of the recruitment e-mail. Follow-up and reminder e-mails from the evaluator that includes references to the partner organization shows participants the coordination between the organizations. Keeping surveys open for extended amounts of time also allows for more reminders and opportunities for participants to ask questions.

Cool Trick- Provide Participate Appropriate Incentives

Incentives such as monetary compensation or prizes can motivate participants to spend their time on the survey. Try to think of something that participants would genuinely enjoy or find useful. Incentives may go to participants or survey administrators, depending on how the survey is distributed. When funding is limited, a drawing for a prize among participants who elect to provide their contact information may be effective.

Rad Resources – Denver’s Urban Trails and Parks

While at Evaluation 2014, you will notice that Denver’s outdoor culture thrives everywhere from mountains peaks to downtown. The Colorado Convention Center bumps up against the Cherry Creek Trail, which if taken north leads to Confluence Park and south leads to Sunken Gardens Park and beyond. A quick exploration west will hook up with the South Platte River Trail and to Sloan’s Lake Park.  Longer treks east of downtown will reward visitors with mountain views at Cheesman Park (go to the Pavilion) and animal life at the Denver Zoo and City Park. Get outside!

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. Carla Hillerns on Thoughtful Tokens of Appreciation to Encourage Study Participation
  2. Jessica Foster on Maximizing Survey Response Rates
  3. LAWG Week: Antonio Olmos, Stephanie Fuentes, and Sheila Arens on Welcoming You to LAWG Week and Inviting You to Come to Evaluation 2014 in Denver this October

LAWG WEEK: Michelle Slattery on the Need for Evaluators in Problem Solving Courts

Thu, 07/17/2014 - 01:15

Greetings from Colorado!  I am Michelle Slattery, President and founder of Peak Research, a consulting firm specializing in program evaluation and STEM research, and evaluator for the 4th Judicial District Veteran Trauma Court at University of Colorado – Colorado Springs Trauma, Health & Hazards Center. I am writing about the need for evaluators in problem solving courts. These courts provide treatment and alternatives to incarceration for people with community health issues like substance use disorders or combat trauma. They are growing rapidly (more than 2,500 courts nationwide) because they show promise for increasing treatment engagement and reducing recidivism (relapse of negative behavior). They are easily criticized, however, because there is currently little evaluation research being done to inform their processes, measure their cost savings, and provide evidence of impact. My team recently concluded work on a 5-year Jail Diversion and Trauma Recovery – Priority to Veterans (JDTR) grant administered by the Colorado Office of Behavioral Health and funded by the Substance Abuse and Mental Health Administration (SAMHSA), which required and funded extensive process and outcome evaluation. During our tenure on the JDTR grant, we helped improve the court and obtain funding and sustainability by sharing and publishing results that document significant improvements in recidivism, post-traumatic stress disorder, substance use, depression, self-harm, and resilience. You can read more about the work here. Problem solving courts are a natural fit for evaluators, providing a rare opportunity to conduct evaluation that can help save lives while also improving communities.

Cool Trick: Check out Says-It.com to create your own custom signs like the Uncle Sam recruiting poster above.

Lesson Learned: Experimental studies with random selection and random assignment are very difficult to implement in the judicial system.  Quasi-experimental designs using propensity score matching may be the best alternative.

Rad Resource: Check out the Interactive Map provided by the National Association of Drug Court Professionals to find problem solving courts in your state.

Hot Tips:

  • While you’re in town, the Colorado Convention Center is a 5-10 minute cab ride from some fun diversions – Voodoo Doughnuts (1520 E. Colfax) and the Tattered Cover Book Store (2526 E. Colfax).  Voodoo is famous for their voodoo dolls, complete with a pretzel stake through the raspberry jelly heart, as well as an assortment of vegan doughnuts, throwback flavors to the 70s like Captain Crunch, and a giant glazed doughnut called the “Tex Ass.”
  • If you’re in the mood for a short road trip, you are just an hour’s drive from Colorado Springs, home of the U.S. Air Force Academy, Garden of the Gods park, and the Cog Railroad which goes to the top of Pikes Peak.  At the top, you will also find doughnuts being made – at 14,110 feet!

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. Margaret Braun and Shannon Myrick on Evaluating Drug Courts
  2. FIE TIG Week: Kathryn Sielbeck-Mathes and Rebecca Selove on Feminist Evaluation and Framing
  3. Olmos-Gallo, DeRoche, and McKinney on Increasing Stakeholder Sophistication

LAWG Week: Laureen Trainer on Enjoying Evaluation in the Moment

Wed, 07/16/2014 - 01:15

My name is Laureen Trainer and in January I became Principal of Trainer Evaluation in Denver; to say the past six months have been a mixture of excitement and terror would be an understatement. But, the good news is that I love the decision to start my own business and I’m excited to begin the next phase of my evaluation career.

As a one-person company, I do all of the data collecting, which leads to some really cool moments. Recently I’ve been observing the new school experience at the Clyfford Still Museum. My past life includes a MA in art history, so I could tell you all about Still and Abstract Expressionism and his role in the pantheon of American art. Yet, I was never a big fan of his work. But these past few weeks, I’ve listened to a lot of kids trying to describe their thoughts and feelings when looking at his art, and it has been awesome! Truly.

For example, when looking at one abstract work of a large field of orange covering 2/3 of the work and a mixture of purple, gray and black vertical lines covering the remaining 1/3 of the work, one 8th grader commented that it reminded of him of day and night. His was a good start, and I could see that, but he went further. He said that the orange took up the majority of the canvas because we remember more and live more during the day, so it is bigger, and even though we sleep a lot, we don’t remember that part of our lives, so that is why the dark part covers less of the canvas. I could go on with other examples of zen, autumn, war, hope, despair…but there isn’t time.

Their ideas have made me take another look at some of the artwork and have led to a new appreciation of the Clyfford Still Museum. Plus, it has been wild to combine my two worlds of art history and evaluation.

Hot Tip: It’s okay to live in the moment and enjoy what you are observing. I know that as an evaluator, I am there to document certain aspects of the tour and the interaction between the students and gallery teachers. However, I have totally enjoyed listening to the kids during my observations. I’ve even gone back through the galleries at the end of a tour to take a new look at an old painting.

Rad Resource: For additional great modern and contemporary art when you’re in Denver (all are close to the Colorado Convention Center), visit Museum of Contemporary Art, Denver Art Museum, Kirkland Museum and our great public art program, which includes mustangs, bears and brooms!

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. LAWG Week: Marley Steele-Inama on Collaboratively Building Evaluation Capacity
  2. LAWG Week: Antonio Olmos, Stephanie Fuentes, and Sheila Arens on Welcoming You to LAWG Week and Inviting You to Come to Evaluation 2014 in Denver this October
  3. Sheila B. Robinson on the Annual Call for Proposals – Evaluation 2014

LAWG Week: Mya Martin-Glenn & Lisa M. Jones on Evaluation in Public School Districts

Tue, 07/15/2014 - 01:15

We are Mya Martin-Glenn and Lisa M. Jones, and we work in the Division of Accountability & Research at Aurora Public Schools in Colorado. We will be sharing how external evaluators can learn some of the nuances of requesting school data. We also will give you a few hot tips for attending the AEA conference in Denver this October.

Lesson Learned: Know the district policies as well as the federal laws governing student data sharing. There are specific federal laws and rules that govern student data sharing, including the Family Educational Rights and Privacy Act (FERPA) and Children’s Online Privacy Protection Act (COPPA).

FERPA protects student education records and COPPA requires online sites and services (such as Survey Monkey and others) to provide notice and obtain permission from a child’s parents (for kids 13 years and younger) before collecting personal information from that child.

Hot Tip: Talk with someone in the district prior to requesting student data even if the evaluation is being conducted as a requirement of a grant. See if there is a central research and evaluation division that oversees data sharing with external entities. Also, check with the state – often the data you need is readily available.

Lesson Learned: Be sure you understand data coding. School district personnel download student data from data management systems such as Infinite Campus (IC). Frequently, data are stored in these systems using programmatic codes specific to the school district. It often takes considerable time to download and “clean” the data file for distribution to external evaluators.

Hot Tip: Ask for a “data dictionary” to help with any coding that may be unfamiliar to you.

Rad Resources: Currently our district is working on revising the external data request process, but here are some examples of other school district requirements for collecting data in schools.

Hot Tips: AEA Annual Meeting in Denver

  • Drink plenty of water – Start a week or so before arriving in Denver so your body has a chance to acclimate to the altitude which can be dehydrating.
  • Wear sunscreen and lip balm – Even in October, the mile high city is closer to the sun.
  • Bring your walking shoes – There are a lot of fun places within walking distance of the conference hotels (as well as a Light Rail system)

o   Comedy Works, 1226 15th St.

o   Denver Performing Arts Complex, 950 13th St.

o   Mercury Café, 2199 California St.

o   Denver Microbrew Tour, Great Divide Brewing Company – 303-578-9548

o   Brown Palace Hotel, 321 17th St, High tea is a lovely experienceor take a tour of the historic hotel

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. LAWG Week: Antonio Olmos, Stephanie Fuentes, and Sheila Arens on Welcoming You to LAWG Week and Inviting You to Come to Evaluation 2014 in Denver this October
  2. MNEA Week: Katherine Drake on Requesting Data from School Districts
  3. LAWG Week: Marley Steele-Inama on Collaboratively Building Evaluation Capacity

LAWG Week: Marley Steele-Inama on Collaboratively Building Evaluation Capacity

Mon, 07/14/2014 - 01:15

My name is Marley Steele-Inama, and I manage Audience Research and Evaluation at Denver Zoo. The Local Arrangement Working Group’s (LAWG) is excited to share with you the great evaluation work taking place in Colorado, as well as give you advice for making the most of Evaluation 2014 in Denver. Coloradoans are very proud of our state; don’t be shocked to notice many locals wearing clothing that dons the state flag’s emblem!

Denver harbors a spirit of collaboration, and this rings true for an initiative of which I’m a part – the Denver-area Evaluation Network (DEN). This network is made up of 15 different museums and cultural institutions, most of whom are a part of the Scientific and Cultural Facilities District (SCFD), a sales and use tax that supports cultural facilities through the seven-county Denver metropolitan area. DEN’s goals are to increase evaluation capacity building (ECB) in museum professionals through a multidisciplinary model that includes trainings with national evaluation experts, attending workshops and conferences, mentoring and technical assistance, dissemination and meetings, and engaging in institutional and pan-institutional studies. Thanks to a grant from the Institute of Museum and Library Services (IMLS), all DEN members will be attending this year’s AEA conference in Denver – a first for most of these participants.

Lessons Learned: Collaboration is core to DEN, however, working together is challenging. We’ve learned that to be successful, we need:

  • Champions to steer the project, and subcommittees to engage members and activate the work.
  • Frequent in-person meetings to stay motivated and connected.
  • Flexibility and the acceptance to make adjustments quickly when needed.
  • Leadership involvement at our institutions in the project to sustain such a large and time-consuming ECB effort. Value buy-in is critical.
  • Two members from each institution as part of the project – those institutions with two members in DEN, compared to one, are more successful at transferring ECB back in their institutions.
  • To accept that pan-institutional studies don’t always work with such a large and diverse group; we’ve learned that cohort studies often work better.

Hot Tip: Colorado is home to endless adventure, and that includes its exploding addiction to running. Start a training plan now and lace up for the Denver Rock n’ Roll Marathon and Half Marathon, scheduled for Sunday, October 19, one day after the conference ends. Prefer a “hoppy” adventure? Colorado is booming with craft breweries. You won’t have to walk far to taste some of Denver’s finest ales. Taprooms close to your hotel room include Denver Beer Company, Great Divide Brewing Company, Jagged Mountain Brewery, Prost Brewing, Renegade Brewing Company, and the legendary Wynkoop Brewing Company. Of course, feel free to stick around after the conference and sample from more of Colorado’s 230+ craft breweries!

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. LAWG Week: Antonio Olmos, Stephanie Fuentes, and Sheila Arens on Welcoming You to LAWG Week and Inviting You to Come to Evaluation 2014 in Denver this October
  2. AHE TIG Week: Sean McKitrick on Accountability Demands for Assessment in Higher Education
  3. Sheila B. Robinson on the Annual Call for Proposals – Evaluation 2014

LAWG Week: Antonio Olmos, Stephanie Fuentes, and Sheila Arens on Welcoming You to LAWG Week and Inviting You to Come to Evaluation 2014 in Denver this October

Sun, 07/13/2014 - 01:15

We are Antonio Olmos, Stephanie Fuentes, and Sheila Arens of the Colorado Evaluation Network (COEN), the local chapter of AEA. Welcome to the Local Arrangements Working Group (LAWG) week. We look forward to your visit to Denver this October for Evaluation 2014. We are working hard with AEA to plan a great event for you while you are in the Mile High City.

In the last year or so, our state has experienced some shifts which may interest evaluators. On January 1, 2014, we legalized the use of recreational marijuana. There have been many anticipated and unanticipated consequences, including possible reductions in crime and new sources of city and state revenues (along with potential ramifications in traditional evaluation arenas like public education and public health). It’s too early in this “experiment” to assess overall impact, but we know everyone is watching. Similarly, energy is on the minds of many Coloradans with intense debates over the oil and gas industry’s use of fracking driving initiatives to ban fracking in some jurisdictions due to of health and safety concerns. At the same time, there are calls to expand research and evaluation of green energies. These two examples speak to the balance between meeting energy demands and environmental consequences. Clearly, program evaluation may be in a position to help.

Hot Tips – What to Visit When You Come to Denver

In addition to the 2014 AEA conference, there are many other things worth keeping in mind when you plan your visit! Check http://www.denver.org/ for events in Denver. Be on the lookout for a local guide; in the meantime consider any of the following:

  • Outdoor activities: Jog or bike around the many trails through Downtown Denver. Denver has a B-Cycle program where you can borrow townies and explore the city. Get out of the city! There are plenty of hiking trails just a short drive away in Boulder, Red Rocks, or Golden.
  • Sporting events: Denver Nuggets, Colorado Avalanche, Colorado Rapids (soccer) and Denver Broncos will be in full swing. If Colorado Rockies are in the pennant race, baseball could be a fun option. Except for the Rapids, all venues are within walking distance or a short shuttle/light rail ride away.
  • Cultural events: Downtown Denver and its immediate surroundings are home to multiple art, nature and history museums, as well as theaters and music halls. Within walking distance are aquariums and historic buildings.
  • Explore the city … and beyond: There are multiple places in Downtown and Lower Downtown (LoDo) to go at night. Both are full of microbreweries – take a tour of them! Using the light rail/buses you can go to Golden, Boulder or Fort Collins. Or rent a car and explore the majestic Rocky Mountains.

We’re thinking forward to October and the Evaluation 2014 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration will soon be open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Related posts:

  1. Sheila B. Robinson on the Annual Call for Proposals – Evaluation 2014
  2. Susan Kistler on making the most of AEA2010
  3. Olmos-Gallo, DeRoche, and McKinney on Increasing Stakeholder Sophistication

Jayne Corso on Boosting Your LinkedIn Presence with Evaluation Keywords

Sat, 07/12/2014 - 09:50

Hello, my name is Jayne Corso and I work with Dan McDonnell as a Community Manager for the American Evaluation Association (AEA).

As you probably know, LinkedIn is the social platform for professional development, career hunting and thought leadership. It is an excellent resource for presenting yourself as an experienced, savvy evaluation professional and enables you to find resources and networking opportunities that will benefit your practices and strategies.

One of the most powerful features of LinkedIn is its ability to search people by name, profession, keywords, or location. Results from these searches are dependent on the strength of personal profiles. I’d like to share a few tips that will help you create a stronger personal profile and become better connected with your professional peers in the evaluation community.

LinkedIn Search

Hot Tip: Utilize all aspects of your profile.
Go beyond just including a photo, your work experience, and education. Add in your publications, skills, awards, independent course work, volunteer experience, and organizations you belong to. All of these features allow you to have a robust, well-rounded profile and will better highlight your expertise as an evaluation professional.

Hot Tip: Incorporate keywords.
Create a list of keywords that accurately communicate your expertise. Are data communications or data visualization or monitoring some of your greatest strengths? Improve your profile by incorporating these keywords repeatedly in your profile descriptions. This will allow your profile to be ranked high when the words are searched within LinkedIn (who you are connected to also influences these rankings). Placing keywords in your profile headline is also a great way to publicly show your expertise and helps other users make an informed decision about connecting with you.

Hot Tip: Customize your LinkedIn URL.
When you join LinkedIn, the site creates a generic URL for your profile that includes a series of numbers. Similar to a website URL, these numbers do not resonate high in a search. Placing your name or keywords into your URL will improve the visibility of your profile. Here are a few tips from LinkedIn on how to get started customizing your URL.

Rad Resource:
The search function of LinkedIn is also a great resource if you’re looking to expand your network and make connections. Searching industry keywords provides you with a full list of professionals and organizations dedicated to evaluation. You can also use advanced search to connect with colleagues, clients, and industry thought leaders. You’ll be surprised at how quickly you can expand your evaluation network with just a few searches. Try it out!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Dan McDonnell on 4 Recent Social Media Changes You May Have Missed
  2. Dan McDonnell on Adding a Photo to Your Blog Post in Google Search Results
  3. Nicole Porter on Looking for Evaluation Opportunities

Cultural Competence Week: Lisa Aponte-Soto and Leah Neubauer on Culturally Responsive Evaluation 101

Fri, 07/11/2014 - 01:15

We are Lisa Aponte-Soto and Leah Christina Neubauer, members of the AEA Public Statement on Cultural Competence in Evaluation (The Statement) Dissemination Working Group. Aponte-Soto teaches at DePaul University, and is an independent consultant. Neubauer is based in DePaul’s MPH Program.

The Statement reminds us that cultural competence is essential to all evaluation theory and practice. Being a culturally responsive evaluator requires a conscious effort of self-awareness and acknowledgement of our biases and the assumptions that we make about cultural groups. It also requires a willingness to attend to unique contextual dimensions and perspectives of a community, which requires open communication and dialogue. It is also a responsibility to contribute to the greater good of society. The following will help you assemble an evaluation team to apply CRE practices.

Four steps for building CRE practices:

  1. Attend to assumptions and context: Examine both cultural assumptions and conduct an analysis of the historical context, sociopolitical changes, and environmental strengths and weaknesses.
  2. Establish a CRE team and identity the team resources: Assemble a team that values and can attend to the unique cultural context of the community served and is inclusive of the key stakeholders and consumers of the evaluation as active agents.
  3. Apply CRE action steps to the evaluation plan: Work with stakeholders to develop, design and implement culturally sensitive and appropriate instruments.
  4. Disseminate evaluation results: Share findings of the program’s influence and impact with all stakeholder groups.

Becoming cultural “competent” is not a prescriptive goal. It is simply a way of being that requires a lifelong process of interactions and learning from experiences. The tips below will assist you in getting started.

Top 3 tips for personal growth and development:

  1. Practicing mindfulness and being present. Attend to what you say to others or even in how your body language may be inappropriate or insensitive.
  2. Journal your thoughts, perspectives, feelings and experiences. Reflect on these and revisit them to assess progress.
  3. Give yourself stretch assignments on your automatic processes once a week by listing 5 assumptions about someone you interacted with on a given day.

The following resources will help you continue to engage in developmental exercises.

Rad Resources:

Explore the Implicit Association Tests that allows you to check on your automatic assumptions your past-experiences, the media, and other cultural norms.

Visit the Teaching Tolerance website for resources on building greater understanding of diverse cultural groups among youth.

Gauge your emotional intelligence by taking the Body Language Quiz and attending workshops or reading books by The Wright Leadership Institute to hone your social and emotional intelligence to tune in to your negative assumptions, the root sources of these, and how to resolve these.

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

  1. Cultural Competence Week: Cindy Crusto and Osman Ozturgut on the Re-Introduction to AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group and Reminder to Examine the “Self”
  2. CC Week: Osman Özturgut and Tamera Bertrand Jones on Integrating Cultural Competence into Your AEA Presentation
  3. CC Week: Jori Hall on Integrating Cultural Competence into Everyday Practice, Part 1

Cultural Competence Week: Osman Ozturgut on Integrating Cultural Competency into Teaching Evaluation

Thu, 07/10/2014 - 01:15

Hi! My name is Osman Ozturgut. I am an assistant professor of doctoral studies at the University of the Incarnate Word. I have been teaching about culture and evaluation for some time now and I wanted to share my experiences.

Lesson Learned: Learning cannot happen unless we understand the social, cultural, economic, educational, and political spaces in which we live. Our further interactions and relationships with students, parents, and other members of the communities we live and work in shape our effectiveness as facilitators of learning. Whether we call it culturally competent, culturally sensitive, culturally responsive, culturally proficient, or simply good evaluation, to be effective evaluators and facilitators, we need to be able to communicate and negotiate among diverse cultures and understand the meaning systems of our audience.

What I have found to be missing in communicating with our audience is the emphasis on how the “self” influences our behaviors. Knowledge is constructed and valued through our past experiences. So, it is important to emphasize the “self” before teaching about specific context: “Cultural competence requires awareness of self, reflection on one’s own cultural position, awareness of others’ positions, and the ability to interact genuinely and respectfully with others” (AEA, 2011).

The focus of many training and teaching sessions on cultural competency has been teaching our audience on the “other”, simply focusing on the knowledge. Knowing does not necessarily translate into behavior. Once we create an awareness of the “self”, we can teach about the “other.” It is then we have a chance of moving beyond, “My best friend is African-American, of course I understand my stakeholders” or “You are Hispanic, you must be an expert on cross-cultural interactions. How about we add you to our evaluation team?”

The framework I use for teaching evaluation is, attitude (creating an awareness of the “self”), knowledge (teach about the “other”), skill (teach them about the skills needed, using case studies), and behavior (reflect on what culturally competent evaluation looks like in practice), of course, emphasizing the context for each case.

Hot Tips:

  • Have a framework and stick to it. Don’t let your audience get distracted. We each define and make sense of cultural competency through our past experiences.
  • Focus your teaching/training on creating an awareness of the “self.”  You can use “Diversity Iceberg” as a tool for beginners.
  • Go beyond the stereotypes when teaching about the “other.”  Focus on what is not obvious.
  • Use case studies to show what culturally competent teaching looks like in practice.

Rad Resources: Microagression is an area we rarely look at. Check out this video for an overview: http://www.youtube.com/watch?v=BJL2P0JsAS4

Check out “Evaluation Ethics for Best Practice”, by Michael Morris. It provides cases and commentaries.

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

  1. Cultural Competence Week: Karen Anderson on Cultural Competence in Evaluation Resources
  2. Cultural Competence Week: Rupu Gupta, Dominica McBride and Leah C. Neubauer on Critical Self-Reflection and Cultural Competence: From Theory to Practice
  3. CC Week: Tamera Bertrand Jones, Osman Özturgut, and Leah Nuebauar on Teaching Culturally Competent Evaluation

Cultural Competence Week: Asma Ali and Anthony Heard on Beyond the Findings: Reflections on the Culturally Competent Evaluator’s Role

Wed, 07/09/2014 - 01:15

Asma Ali and Anthony Heard, Professional Development Committee co-chairs for the Chicagoland Evaluation Association (CEA), say “Hello!” from the Windy City.

CEA’s professional development presentations in 2014 have focused on various strategies employed by professional evaluators to work with their clients and stakeholders. Defining “culture” to include environmental context, attendees agreed that evaluators have a responsibility to make findings relevant to diverse stakeholder groups. Our members also identified the following tips for thinking “beyond [research] analysis and findings”:

  • Understand evaluation context. Cultural competence includes reflection about the evaluators’ role and the evaluation circumstances. Contextualizing research findings and making them relevant to various diverse stakeholders is an important part of an evaluator’s work. This includes understanding the various political and social contexts of the program, as well as the purpose of the evaluation and potential use of findings.
  • Set evaluation expectations early in project and revise expectations as needed. Stakeholder relationships are an essential part of an evaluators’ work. To manage these relationships, evaluators must set expectations with their stakeholders in a manner that is relevant to them. Setting expectations can be managed formally with an evaluation plan or work process flow or with informal discussions or updates. In both cases, early and frequent communication with stakeholders is important.
  • Make the evaluation findings relevant to more than one stakeholder group. Evaluations are often commissioned by lead agency/program officials. Agency cultures may be very different in other stakeholder groups, leading to distinct perspectives about the findings. Evaluators may be required to disseminate findings in multiple methods (short or long reports, presentations, town hall meetings) to accommodate different learning and discussion styles among stakeholders.
  • Develop appropriate relationships with stakeholder groupsEvaluators have a distinct culture that influences their relationships. Internal evaluators may have easier access to or existing relationships with stakeholders but may be vulnerable to internal politics and expectations that don’t encourage best evaluation practice. External evaluators may need to spend additional time developing relationships with stakeholders, but may have a more clearly defined, evaluation-focused role.
  • Solicit input from fellow evaluators. Whether you are new to cultural competence or a seasoned expert on the topic, AEA and local AEA affiliates can be an extremely effective resource for perspectives you had not considered.

Lesson Learned: Understanding evaluation context, setting expectations, making findings relevant, addressing multiple stakeholder needs and developing appropriate stakeholder relationships are all involved in incorporating cultural competence in evaluation. Each strategy also fits into the others. For example, understanding your role as an internal evaluator and developing appropriate relationships with stakeholders will make it easier to understand the evaluation context, set expectations, and make findings relevant for stakeholders. Most of all, enjoy the discovery process!

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

  1. CC Week: Osman Özturgut and Tamera Bertrand Jones on Integrating Cultural Competence into Your AEA Presentation
  2. CC Week: Jori Hall on Integrating Cultural Competence into Everyday Practice, Part 1
  3. Cultural Competence Week: Tamara Bertrand Jones on Essential Tools for Training Culturally Responsive Evaluators

Cultural Competence Week: ¡Milwaukee Evaluation! on Dissemination of Cultural Competence

Tue, 07/08/2014 - 01:15

Hi! We are Nicole Robinson, Laura Pinsoneault, Nicky Bowman, and Emily Connors and we’re founding members of ¡Milwaukee Evaluation!. ¡Milwaukee Evaluation! is located in Wisconsin and is a professional development and field building collaborative of Wisconsin-based evaluators. We are also the statewide AEA affiliate. Our current field building initiatives center on diverse representation among evaluation professionals and culturally responsive evaluation. Our current work is focused on identifying ways to increase both the number of evaluators of color and other underrepresented groups (e.g., low-income, LGBT, rural, youth) and the number of pathways into the field of evaluation for these evaluators.

We are kicking off that work by bringing together evaluators of color to share how they entered the field. But, so far this work is off to a slow start. We’re having a hard time identifying evaluators of color across the state and we know there aren’t that many to begin with (hence the reason why we started this work). When we committed to doing this, you could say we knew on some level this was a long-term effort and, as result, we embedded this particular issue in to our mission, vision, and values – and we established ourselves as a legal entity to have an infrastructure to create change. However, we’re dealing with a major structural challenge in the field. More established groups with greater funding and influence could make major differences and have greater impact in the time it takes for us to locate and convene a handful of evaluators of color throughout Wisconsin. We’re actually grateful for AEA because it exemplifies the power of a large mainstream organization and their effect when they take on issues like this (e.g., AEA’s GEDI program).

At our 2013 Social Justice and Evaluation Conference, we held a brainstorming session with about a dozen talented evaluators across the state and it included evaluators of color and allies. Together, we generated key priorities or “headlines” to intentionally promote the inclusion of evaluators from diverse experiences and communities. Some of our best headlines were:

  • Not your Usual Suspects:  ¡Milwaukee Evaluation! launches a New Emerging Evaluators Curriculum
  • Meet the Graduates – The First Cycle of Emerging Evaluators
  • Nontraditional Funding for Nontraditional Evaluators
  • Turning Water into Wine: Transforming Evaluation Practice
  • Youth Gone Wild: Youth Evaluators
  • Cultivating Natural Evaluation Potential: Lay Evaluators

We’ll get there. These headlines will be a reality!

Rad Resources:

Check out the AEA GEDI PROGRAM as a resource for either getting experience with culturally responsive evaluation or recruiting evaluators who have been trained in this area.

Also check out the The Handbook of Leadership Development Evaluation edited by Kelly Hannum, Jennifer W. Martineau, Claire Reinelt

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

  1. Susan Kistler on the AEA Graduate Education Diversity Internship
  2. GEDI Week: Michelle Corbett on Working with Organizations to Build Their Internal Evaluation Capacity
  3. GEDI Week: Ashaki Jackson, Stewart Donaldson, and John LaVelle on the GEDI Program

Cultural Competence Week: Dominica McBride on The Importance of Cultural Competence

Mon, 07/07/2014 - 01:15

Hi, my name is Dominica McBride and I’m a member of the AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group, along with CEO of Become, Inc., a nonprofit using culturally responsive program evaluation as a tool in realizing social justice. I also teach Diversity at The Chicago School of Professional Psychology.

Lesson Learned: Unfortunately, too often, I see people pay “lip service” to cultural competence but don’t exert the effort needed to make the psychological changes necessary to make real structural change. Many people don’t see the profound importance of cultural competence. The purpose of this construct is not only so we (whoever the “we” is – evaluators, teachers, medical doctors) can communicate effectively and accomplish a given collective goal with people of any cultural background. It is also to achieve social equality so we can realize the US’s foundational philosophy; as the Declaration of Independence asserts, “We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.”

I want to take this time to reiterate the importance of cultural competence or responsiveness through showing its relevance today with a focus on macro level issues. Structural discrimination is still alive and well. Structural discrimination is the unintended discrimination that is embedded in institutions, perpetuating oppression. This type of discrimination continues to show up in institutions like the criminal justice system, education, housing and healthcare. For examples:

  • Michelle Alexandar writes, “More African American men were disenfranchised due to felony convictions in 2004 than in 1870.” The great majority of these convictions are due to drug-related felonies, despite the fact that African Americans and Caucasians use and deal at equal rates.
  • The tax structure lays the groundwork for unequal resource distribution to schools, with schools in more disadvantaged areas receiving less resources, thus perpetuating the cycle of disproportionate unemployment in more impoverished communities.

Our role as culturally competent or culturally responsive evaluators is not only to be aware of how we influence a situation and how culture can influence participants and a program but also to contribute to structural, societal changes for the greater, collective good. This means observing both the overt and the latent in programs and their contexts, saying the hard truths, and taking risks. This means being an advocate and catalyst.

Rad Resources: The New Jim Crow by Michelle Alexander provides a rich description and statistics showing the morphing of Jim Crow into present day structural discrimination in the criminal justice system.

Check out a video, The Unequal Opportunity Race, demonstrating some contextual barriers that have and continue to create obstacles to equality.

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Cultural Competence Week: Melanie Hwalek on the Adoption of the AEA Public Statement on Cultural Competence in Evaluation – Moving From Policy to Practice and Practice to Policy
  2. Cultural Competence Week: Karen Anderson on Cultural Competence in Evaluation Resources
  3. Cultural Competence Week: Dominica McBride on AEA 2013 and Spreading the Word on Cultural Competence

Cultural Competence Week: Rupu Gupta and Tamara Bertrand Jones on Cultural Competence Working Group Evaluation

Sun, 07/06/2014 - 01:15

Hello, we are Rupu Gupta and Tamara Bertrand Jones, members of the AEA Public Statement on Cultural Competence Dissemination Working Group (CCWG). We lead the Research and Evaluation subgroup and are developing strategies to understand how the larger working group is achieving its goals.

Lesson Learned: Purpose of Research and Evaluation subgroup. We started our work within this subgroup acknowledging that a systematic assessment of how the CCWG is achieving its goal is critical to document its accomplishments over time. We are a group of evaluators, so of course evaluation is part of our work.

In our initial discussions about the CCWG, we considered the group’s goals in general, and eventually decided to focus on two goals as a starting point:

  • Increasing awareness of the AEA Public Statement on Cultural Statement and resources; and
  • Increasing use and application of the concepts embedded in the Statement.

However, it was apparent that as we began our efforts we needed to learn, first and foremost, the different mechanisms through which the CCWG had been sharing the Statement.

Our post today is about our ongoing efforts examining the various ways the working group, with 22 members, has been disseminating the AEA Public Statement on Cultural Competence.

Within this group, we are currently pilot-testing a qualitative and quantitative online survey focusing on:

 

  1. Domains within which the Statement was shared (e.g., teaching, practice, and policy);
  2. Specific activities undertaken;
  3. Format of the dissemination (e.g., presentation, social media, and web material)
  4. Audiences with whom it was shared; and
  5. Outcomes of the activities.

Lesson Learned: Preliminary Findings. Based on the responses of a sample of the CCWG members, we learned that:

  1. The group primarily shared the statement equally in the domains of teaching, practice, and policy.
  2. The audiences with whom they shared the statement included federal policy makers, AEA members, pre-doctoral fellows, funders, and anyone interested in learning about evaluation.
  3. This was reflected in their activities ranging from disseminating the Statement at the federal level (e.g., through a letter to then Secretary of State Hillary Clinton), professional development efforts at AEA conferences, and webinars.
  4. The primary outcome of the group’s efforts was to increase knowledge of the statement and strategies and practices for culturally competent evaluation

Lesson Learned: Next Steps. We will continue to gather more information from the CCWG to get a complete picture of the ways in which the group has been disseminating the Statement. The results of the online survey will be used to understand the outcomes of interest for the group, so as to develop a longer-term evaluation plan for the CCWG’s efforts.

Rad Resource: See AEA’s Statement on Cultural Competence in Evaluation

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

 

Related posts:

  1. Cultural Competence Week: Melanie Hwalek on Cultural Competence Dissemination in Policy
  2. Cultural Competence Week: Melanie Hwalek on the Adoption of the AEA Public Statement on Cultural Competence in Evaluation – Moving From Policy to Practice and Practice to Policy
  3. Cultural Competence Week: Cindy Crusto and Osman Ozturgut on the Re-Introduction to AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group and Reminder to Examine the “Self”

Sheila B Robinson on Fast, Free, User-Friendly Eval Content at Your Fingertips!

Sat, 07/05/2014 - 06:11

Image Credit: The Hamster Factor via Flickr

Hi! Sheila B Robinson here, aea365′s Lead Curator and sometimes Saturday contributor with a few favorite eval sites with a broad range of content and free resources. Due to space limitations, I’ve left out descriptions of the most well known, but I do have them all and more here.

Lesson Learned: Start with: AEABetter Evaluation, Centers for Disease Control and Prevention – Office of the Associate Director for Program – Program EvaluationCommunity Solutions Planning and EvaluationFree Resources for Program Evaluation and Social Research MethodsInnovation Network Point K Learning CenterUniversity of Wisconsin-Extension Program Development and Evaluation, and Western Michigan University Evaluation Center.

Hot Tip: Check out these other sites for great content too!

Betty C. Jung’s Website

  • Huge collection of links organized into categories – Evaluation 101, Evaluation Guidelines, Evaluation Resources, Evaluation Templates, Logic Model Resources (government and non-government), Logic Model Templates, Public Health Program Evaluation, Evaluation of government programs, US Government Accountability Office, and more.

The Community Toolbox

  • This site is primarily for community development, but does include evaluation resources. Look at the table of contents and scroll down to Part J: Evaluating Community Programs and Initiatives.

Evaluation Toolkit for Magnet School Programs

  • Good basic resources for school-based programs; tools and resources may also inform evaluations of other types of programs.
  • Organized around six areas:
    • Set the stage for purposeful evaluation, Develop a theory of action for your program, Evaluate implementation to document what you are doing, Evaluate outcomes to show your program is making a difference, Get quality data into your evaluator’s hands, Take action in response to evaluation results.

The Free Management Library Basic Guide to Program Evaluation

  • “This document provides guidance toward planning and implementing an evaluation process for for-profit or nonprofit programs…”
  • Online guide to PE features sections on Where PE is helpful, Basic ingredients, Planning PE, Major Types of PE, Overview of methods to collect information, selecting which methods to use, Analyzing and interpreting information, Reporting evaluation results, Who should carry our the evaluation, contents of an evaluation plan, pitfalls to avoid.

Learning for Sustainability

  • “This LfS portal is a reference guide for those working to support social learning and collective action around sustainability issues.”
  • Though not devoted entirely to evaluation, this site features a wealth of evaluation resources including links to other sites, and downloadable papers and booklets.
  • Look for sections of the site devoted to “Planning, monitoring and evaluation” and “Research methods and approaches.”

Shaping Outcomes

  • While focused on Outcomes-Based Planning and Evaluation for Libraries and Museums, this website offers easy definitions of terms and examples with a free online course organized into 5 modules: Overview, Plan, Build, Evaluate and Report. It even includes instructor materials for teaching the course on shaping outcomes.

Something for everyone! Wouldn’t you agree?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Related posts:

  1. Margaret Riel on Learning Circles
  2. A. Rae Clementz on Evaluation Tech Resources
  3. Stewart Donaldson on Interactive Conceptual Modeling

Lisa Kohne on Developing Evaluation Newsletters to Disseminate Results To Multi-Site Participants

Fri, 07/04/2014 - 01:14

Hello! I’m Lisa Kohne, an independent evaluator and I work for SmartStart Consulting in Orange County, California. We specialize in conducting project evaluations for federally funded grants, primarily from the National Science Foundation. Most of our clients are math, science, and engineering professors from four-year universities. One of our big challenges is that some of our projects are multi-institutional, multi-state, and multi-country. It’s very difficult to bring multiple partners together at the same time to discuss evaluation findings – and most don’t have the time, inclination, or enough evaluation knowledge to read lengthy reports.

Hot tip:
To overcome these challenges we began to develop Evaluation Newsletters. They are usually two pages with lots of graphs, maps, and tables. We try to make them colorful, high-interest, and eye-catching. Some are wordier than others and our “skills” have evolved over the years. You can see the evolution from one of our earlier version (very wordy) to our more recent version (less wordy, more white space).

We only offer these to our larger, multi-site projects.  The reactions and feedback have been extremely positive.  No PI has ever turned down the offer to create a newsletter.  They are also great to distribute at advisory board meetings and project conferences.

Rad Resources:

  • Google Images works great for the simple clipart needed for newsletters. Simple is better. Just be careful to not use copyrighted ones.
  • Microsoft Publisher is our current choice of software.  We’ve tried Word but Publisher lines up the information much better.  Also, the new online subscriptions to MS Office 365 include Publisher.

SmartArt is our go to graphic developer.  Only available in MS Word, not Publisher.  So you need to create it in Word and paste it into Publisher.

Lessons Learned:

  • Less is more.  Less words, more pictures, lots of bullet points.
  • Make it personal and make it positive.  Add university and project logos, project goals, maps that indicate locations of participating institutions, funders’ logs, and anonymous quotes from participants.
  • Newsletters take a lot of time so build the cost into the budget.
  • Recruit your most artistic employee to create your newsletters – someone who understands color, balance, and brevity of words.
  • Send a sample to stakeholders and ask if it would be helpful to get evaluation results out.
  • Get commitment from your principal investigator to email the newsletters out to all stakeholders and/or project participants and post them on their project webpage.  Here is a webpage containing out newsletters on a NSF PIRE project.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Gwen Fariss Newman on Using Newsletters to Connect with Others
  2. Susan Kistler on Leaving Wordle for Tagxedo
  3. Tarek Azzam on Geographic Information Systems (GIS)

Bethany Laursen on How to Evaluate the Situation, Not Just the Program: It’s Complex!

Thu, 07/03/2014 - 01:15

I’m Bethany Laursen, Evaluation Outreach Specialist with the Solid & Hazardous Waste Education Center (SHWEC) at the University of Wisconsin. I’m also principal consultant at Laursen Evaluation and Design, LLC. At SHWEC, I help staff design programs that engage opportunities to achieve our mission. Opportunity hunting requires a form of situation assessment, which has not been widely or deeply discussed in evaluation—especially when it comes to evaluating opportunities in complex, dynamical situations.

Rad Resource: AEA’s EvalTalk and TIG group listservs as peer learning communities.

Through EvalTalk, several colleagues helped me distinguish among three approaches/tools that all claim to be useful in developing programs in complex situations: needs assessment (NA), developmental evaluation (DE), and strengths, weaknesses, opportunities and threats (SWOT) analysis.

Lesson Learned: NA, DE and SWOT are all necessary parts of evaluating complex situations and program responses.

To summarize this discussion so far, we have the following options, where () means “as a part of” e.g. NA is a part of SWOT:

  1. NA–>SWOT–>DE
  2. SWOT(NA)–>DE
  3. NA–>DE(SWOT)
  4. DE(NA, SWOT)

Any of these combinations is logical, although #4 might be difficult without one of the others occurring first. What is not logical is leaving one of the triumvirate out (NA, DE, and SWOT). Here’s why:

SWOT is inherently evaluative: it assigns data a certain value label (S, W, O, or T), based on the criteria “effect on our organization’s goals.” Clearly, we need data to do a reality-based SWOT, and this is why we must include a needs assessment. But a NA per se is not going to be enough data, because many clients think a NA is just about external stakeholders’ needs (Os), not internal capacity (Ss and Ws) or larger system realities (often Ts). (If preferred, one could also frame a NA as an ‘asset assessment.’) These external and internal ‘lessons learned’ from our situation should inform developmental program evaluation.

In complex situations, needs assessment is more usefully framed as ongoing situation assessment. This is what I see as the main evaluation task in the Creative Destruction phase of the adaptive cycle. Once we have a lay of the land (situation assessment) and we’ve evaluated the best path to start down (SWOT analysis), then we can jump into developmental evaluation of that path. Of course, what we find along the way might cause us to re-assess our situation and strategy, which is why #4 above is a logical choice.

Lesson Learned: Listen to the language your clients are using to identify relevant evaluation approaches and tools. In SHWEC’s case, our connection to the business sector led me to SWOT analysis, strategic planning, and Lean Six Sigma, all of which are evaluative without necessarily marketing themselves as evaluation approaches.

Figure 1: Augmenting a traditional logic model, this is a metaphorical picture of how SHWEC understands our complex, dynamical situation and our potential evaluation questions. (Each sailboat is a staff member.) Next, I had to find evaluation approaches that would fit.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Related posts:

  1. Michael Quinn Patton on Developmental Evaluation
  2. DE Week: Kate McKegg and Nan Wehipeihana on Talking to Clients and Communities about Developmental Evaluation
  3. MNEA Week: Pat Seppanen on Evaluating Complex Adaptive Systems