enhancing access for individuals with limited english profiency toolkit

Escape

Alerta de seguridad: si cree que sus actividades en la computadora están siendo monitoreadas, por favor accese este sitio web desde una computadora más segura. Para salir inmediatamente de este sitio, haga clic en la tecla “esc”. Si está corriendo peligro en este momento, llame al 911, a la línea de crisis local, o a la Línea Nacional Directa contra Violencia Doméstica al  1-800-799-7233 o TTY 1-800-787-3224.

Alerta de seguridad: si cree que sus actividades en la computadora están siendo monitoreadas, por favor accese este sitio web desde una computadora más segura. Para salir inmediatamente de este sitio, haga clic en la tecla “esc”. Si está corriendo peligro en este momento, llame al 911, a la línea de crisis local, o a la Línea Nacional Directa contra Violencia Doméstica al  1-800-799-7233 o TTY 1-800-787-3224.

Why evaluate a program?

It is not easy knowing whether a program is working as planned, or if it needs improvements or adaptations. Like using a recipe to make a dish, we need to document what ingredients we will need and what steps we need to follow.  Similarly in program evaluation, documenting your work allows you to see if the “ingredients and steps” of your program are working, or if they need improvements or adaptations. Unlike a recipe, evaluation allows you to understand how effective your program is, its strengths and weaknesses, its cost-effectiveness, opportunities to expand, and whether it should be eliminated or substituted by another program. This is valuable information when you need to apply for new or renewed funding.

What is my purpose in evaluating my program?

The reasons for conducting evaluation can be diverse and include both internal and external factors. When you plan to evaluate a program it is important to clearly identify the reasons why you are collecting information about the program. This will inform your evaluation activities. Table.1 consists of a series of questions that offer guidance in identifying the purpose for evaluating your program.

Table.1 Reasons to Evaluate Your Program

 

Who should I include in conducting evaluation activities?

Another essential element of evaluation work is engaging other people who can support your evaluation process. We usually refer to them as the stakeholders.  Stakeholders can be members of your community, members of your organization, or people who have connections with the communities you serve.  To help you to identify the key stakeholders of your program we suggest the following steps:

  1. Start by brainstorming a list of potential stakeholders in the community and in the organization.

  2. Consider reviewing the questions on the following table to narrow the list of stakeholders.

Table.2 Stakeholder Engagement 

In the following section–How do I Start?–we will explain in detail the first steps in conducting an evaluation.

 
 

How do I Start?

Program evaluation work is like cooking a dish from a recipe. As in all good recipes there are steps to follow, accompanied by a list of ingredients, and an image or photo of what the dish will look like in the end. As with all recipes, sometimes it is necessary to change or modify the steps or ingredients. Sometimes programs also need to be adjusted according to the realities of the community so we can maximize the benefits.

This process of collecting information allows us to reflect on the program (if it needs to be adjusted, if it lacks certain ingredients). This is what building the evidence consists of. Many models and strategies exist to support this process.

In the case of our toolkit, we have chosen to utilize the logic model as the visual guideline of the program or the list of ingredients and steps to follow, to evaluate your program or recipe. In the following section we offer more details about these tools, as well as examples of how to use it in our community work.

Remember there are also other ways to construct evidence and you can learn about them in the Resources section.

What is the logic model?

The logic model is like a map, a graph or a drawing that helps you understand the connection that exists between different parts of a program, project or initiative. Another way of looking at the logic model is like the page of a recipe, the lists of ingredients, the steps to follow, the order to take, and the time it takes to cook the dish. In other words, it is the sequence of events/activities required to shape the program.

How to develop a logic model?

There are many ways to create the logic model for a program. What is important is to include all the ingredients which will allow you to explain how it works and the rationale for the outcomes. We will use an example based on a common problem: you are thirsty and you do not feel well. Let’s look at this situation through a logic model.

Example: You are thirsty and you do not feel well.

What do you need? The first thing you have to do is find what you need—water or juice.

What do you need to do? You need to take action—drink the beverage.

What will happen after you drink the water or juice? The final result is that you will feel better and you will no longer be thirsty.

Below is this information in visual form.

Graph 1. Visual representation of “I am thirsty”.



This same example of water and thirst can also be explained in a different way and using the terminology (words) that evaluators use.

Graph 2. Logic Model of “I am thirsty”


As you can see, we are addressing the same situation or problem but using new terms. Next we offer more detail by describing each element of the logic model.

What are elements of a Logic Model?

The elements of a logic model are also known as components. Evaluators use different terms (words) to describe these components. These terms include:

Resources or Inputs: refers to the assets you have and need for the program to work. For example, these include people, money, educational materials, meeting space, etc.

Activities or Outputs: refers to what is done with the resources you have. These can be educational workshops, information campaigns, brochures, etc. It also refers to the beneficiaries of the outputs or activities, also known as the participants.

Impact or Outcomes: refers to the changes that occur as a result of the use of resources and activities. Changes can occur at an individual, family or community level. It is important to take into consideration that these changes can be positive, negative or unexpected and that the timeframe can vary between short, medium or long term.

Below we introduce an example and a graph which includes all of the previously mentioned elements.

Example: Working with animals and children at the shelter.

Mothers at a shelter have expressed to one of the advocates, their worries concerning the effects of having witnessed violence at home on their children. The mothers reported their children had problems with aggressive behavior and a lack of interest in participating in the recreational activities the shelter organizes for the children. In response to this situation, the decision has been made to offer therapy to the children through the use of animals. Let us look at this situation through the lens of a logic model.

Problem Statement: Children in the shelter are acting out the impacts of witnessing violence.

Resources or Inputs: local animal therapy program, child therapists, space at the shelter, animals, etc.

Activities or Outputs: offer animal therapy sessions with the children from the shelter, identify the ideal number of sessions and time offerings, etc.

Impacts or Outcomes: Short-term: the children engage in play therapy sessions. Medium-term: Children’s aggressive behaviors are reduced in the shelter. Long-term: children display healthier and more positive behavior all around.

Graph 3. Working with animals and children at the shelter logic model.


In the Resources section you will find several examples of program logic models, which vary according to the elements and the complexity of the situation in which they were used.

How do I Spread the Word?

Sharing the evaluation results is just as important as conducting an evaluation. Your organization already has great lessons about the program (program successes, its weakness and its strengths) that deserve to be shared as it will benefit others.  This is also a great opportunity to strengthen communication and build capacity among your allies.

The most common way to share this information is in a report. (Here, we provide an example of a report sharing the results of a community evaluation conducted in the form of listening sessions). We recommend to start by identifying who your potential audiences are, so you can decide the best ways to present your evaluation results.

You can use the tables below to help you get started.

Potential audience

Who would have an interest in knowing about the impact of your program?

Possible formats

What would be the most cost/time effective way to present the information?

Remember that sharing your evaluation results may require resources. Determine what resources your organization has or needs to distribute these results in a cost effective way.  Please check our Resources section for additional resources.

What do I need?

There are many ways to collect information about your program and each approach has its strengths and weaknesses. All of the data you collect forms part of the information you can utilize for your evaluation.

What are the different ways to evaluate a program?

There are many ways to conduct an evaluation, but it is the goal and the function of what you want to understand that determines the way an organization constructs its recipe. Generally speaking, the more widely known models or types of evaluations are:

  • Formative: occurs at the same time that the program is carried out

  • Summative: occurs at the end of the program

  • Process: occurs during the implementation of the program

  • Outcome: occurs just after the program has finished

  • Impact: occurs after the program has been carried out

To learn more about the different models or types of evaluation please visit the Resources section.

What information is needed for evaluation?

When we talk about collecting information for evaluation, we are not referring to something completely unfamiliar, on the contrary, it is something you already know, but perhaps you do not define it using these terms. Given that day to day we receive a lot of information from our participants, for example their age, place of origin, etc., we are sure that collecting this type information is already part of your process. Our contribution to your “recipe” will be to provide guidance on the ways to collect, organize, and analyze this information.

Types of Information

As a general rule, evaluation information is organized into two groups: numbers (quantitative) and words (qualitative).

When we talk about numbers (quantitative data), we mean information that helps us describe what exists and happens in terms of time, quantity, frequency. For example:

  • the average age of participants,

  • the number of people receiving services per week,

  • the number of sessions participants attended, etc.

When we talk about words (qualitative data), we mean information that helps us capture a deep understanding of participant’s experience in the program or their personal stories. This can include information about the why, the how, and the consequences of participating in the program. For example:

  • opinions about the program,

  • descriptions of experience within the program,

  • descriptions of impact on their personal stories, etc.

What are the sources of information?

The following table lists examples of potential sources of information that you might have and/or need for your evaluation.

Table.1 Sources of Information

Places where you can access information outside your organization include: local libraries, universities, government offices, research institutes, mass media (newspapers, radio), and social media (e.g., Facebook, Twitter, and Instagram).

We suggest you create your own list of information sources since this will help you organize your work and keep track of the information that you might have and/or need for your evaluation.  Utilize this worksheet to help you get started.

What tools can I utilize to collection information [data]?

There are a variety of tools that can be used to collect information. We believe that the best tool is always the one that corresponds to the circumstances in which it will be used. In other words, based on the realities of the organization, the resources available, the type of program, the participants, and the community. Consider the resources you have both in terms of personnel and funding, this will help you determine if you can use several tools—simultaneously or during different phases of the program. Using several tools allows you to obtain rich information from different perspectives.

The following table will help you understand each tool, its purpose and the possible situations we do not recommend using the particular tool. This will help you find the best “match” for your program and evaluation needs.

Table. 2  Types of Tools

When can I collect information?

Deciding when to collect information can appear to be a challenge, if we think of it as a new task. You may not have considered this, but much of your everyday work includes collecting information. For example, when people come to the office in need of services, they usually complete a form with their personal information. When collecting information about your program, the only differences are the goals and timing of your activities.

Imagine you want to collect information to evaluate a new program, in its initial phase.  You think you do not have all the necessary information to decide what the main activity will be in order for the program to work. One strategy could be to organize a focus group to brainstorm ideas about the program’s main activities. Another scenario could be that a program ended a couple of months ago, but you lack sufficient information to allow you to identify the program’s strengths and weaknesses. In this case asking former participants to fill out a survey could provide the extra information you need.

The most important point is to have a clear idea of the evaluation goals or the reason you want to collect evidence. This will also help you decide when is the right time to collect information.

Graph 1. Opportunities to Collect Information.

Other Considerations in Collecting Information/Building Evidence

What are the possible costs involved in conducting evaluation?

It is important to view evaluation activities in the same way you view any other job your organization performs, it will require a budget. As with any budget you will need to include potential and unexpected expenses.

Determine the essential components of your program and the ones you can substitute. Consider collaborating with other organizations or agencies to carry out the work, as a way to save money or reduce costs. Listed below are a few basic elements you should include in your budget:

  1. Personnel: consider looking within your organization to determine whether or not you have staff that have the skills or experience to help you with this task.

  2. Consultant: perhaps you do not have the personnel or their experience is limited such that you will need to hire someone outside the organization. This activity must not be taken lightly because it could end up with outcomes you did not hope for. To learn more about this task we offer you a complete section on Choosing an External Evaluator, which we believe will be very useful in helping you find the right evaluator for your evaluation needs.

  3. Transportation costs: you must consider that you might have to visit several places to obtain information or that your program activities take place at different locations.

  4. Printed materials: the program as well as the work of building evidence will require producing and printing documents, be they surveys, interviews, focus groups guides, etc. Take this cost into consideration in addition to the cost of obtaining information from places like libraries, government agencies whether in or out of state.

  5. Office supplies and electronic equipment: in this category include the computer, any software you might need to get the job done. And currently with the use of video cameras for Skype or other programs for video conferences, audio, graphic designs, etc.

  6. Communications: these include traditional mailing costs, professional newsletters and electronic mailings, internet, and local, national and/or international telephone calls.

Consider in detail all the activities and materials required to efficiently carry out this task. Prepare your budget considering the best and worst case scenarios. The key is to pay attention to all the details and plan for any scenario.

How can you conduct evaluation on limited funds?

  • Consider personnel and resources available to evaluate your program and re-allocate resources so that someone is working on evaluation activities.

  • Consider your current processes and how you can enhance these activities to gather evaluation information.

  • Consider partnering with graduate students/universities to get free or low-cost evaluation support.

  • Consider hiring an external evaluator and cost-sharing fees across multiple organizations.

  • Do not take an “all or nothing approach”…even starting with one evaluation question and then building evaluation activities over time can be useful.

  • Consider using free/low-cost tools that can help you gather and analyze information easily and quickly (e.g., SurveyMonkey, etc.)

  • TIP: Build evaluation activities into grant submissions.

How do I evaluate a program?

As we mentioned in the previous section, evaluating a program is in many ways similar to cooking. You start by selecting a recipe, gathering all the ingredients, and then transforming them into a delicious meal.  In evaluation, you start by deciding the type of evaluation that you will conduct, gather the information, and transform it. To be able to do this transformation you need to learn how to organize, analyze, and interpret data.  In this section we will review this process.

Organizing, Analyzing and Interpreting Data

You probably already collect a great deal of information about your program. For example, information about who is using your service and how many people you see every day. This information is called raw data. To make this raw data useful you need to transform it into meaningful, helpful information by organizing it, analyzing it and interpreting it. This raw data can be in the form of numbers (which we call quantitative data) or words (which we call qualitative data).  

Organizing, Analyzing and Interpreting Quantitative Data

Organizing Quantitative data

The first step is to organize the data to make it easy to understand and easy to find.

We might ask our participants to fill out a survey and tell us how old they are, their gender and their place of origin. To organize this data so that we can understand it and find it easily, we enter the data (information) from each survey into a table in a Word document or in an Excel spreadsheet.

Here is an example of a table we made using the quantitative data (numbers) from our participants’ survey:

This table makes it much easier to see all the data we have collected. If we did not make that table, we would have to read all of the surveys every time we wanted to find out something about our participants/clients.

Analyzing Quantitative Data

The next step is to analyze the data so that we can get information from it that actually tells us something we want to know.

For example, we can analyze the data in the table that we made from our participant survey to tell us about the people who use our services. Common forms of data analysis include frequency counts and averages .

Sample analysis:

  • 3 participants  (frequency count)

  • 40 years is the average age  (average)

  • 1 female and 2 males  (frequency count)

Interpreting Quantitative Data

The next step is to interpret the data so that it is useful to us.

Sample interpretation:

It appears as though from the three participants that the average age is 40 years old, and that we have more male participants than female participants.

As you can see, we were able to transform the raw data that we have in the table into a succinct paragraph that give us meaningful information about the program participants. Now let’s see how we can transform raw qualitative data (words).

Organizing, Analyzing and Interpreting Qualitative data

Most of the qualitative data (words) that we collect usually come from recordings of interviews or focus group discussions.  Sometimes we use data from notes either staff or participants collected about the program or specific program activities.  

Organizing Qualitative Data

Similar to the quantitative data, the first step is to organize the data. For qualitative data, this might mean transcribing the information from recordings to a Word document or from hand-written notes to a Word document.  This allows you to have all the information in the same format so you can analyze it.

Analyzing Qualitative Data

The next step is to analyze the data which consist of identifying the main themes. The following step-by- step process helps guide you through this process:

  1. First, you should write down your own observations of the activity, and any themes or topics that you remember.

  2. Next, review your notes and/or the transcriptions. Begin to highlight key statements and label these statements. For example, if you notice that several statements have to do with effective ways to engage the community, one label could be community engagement.

  3. Repeat the process with the rest of the information you collected.

  4. As you label the statements, you will notice that the themes will begin to repeat (this is what we mean by recurrent themes).

  5. Next, group together the themes based on those that are most similar.  You may need to merge theme to narrow down your results to a manageable number.

  6. This list of themes is the results of your analysis.

Let’s look at an example. This sample statement was obtained from focus group activities with youth about the Deferred Action for Childhood Arrivals (DACA) program.

“DACA allowed me to attend a university in another state. Without DACA I would have never applied to a university far from home as I knew I could not safely travel across country because of my documentation status.”

From this sentence we were able to identify the following themes:

  • THEME: Increased Educational Opportunities  

    • DACA widens the pool of potential universities youth can apply to

    • DACA allows youth to consider more educational opportunities

  • THEME: Safety

    • The program protects youth regardless of documentation status

Interpreting Qualitative Data

The final step is interpreting the data (inferring meaning from the data).

Sample interpretation:

From the testimony of one of our participants, we were able to learn that the implementation of Deferred Action for Childhood Arrivals also known as DACA has had a positive impact on the life of our youth participant by opening more doors to educational opportunities and increasing a sense of safety.

This task can be a great opportunity to build capacity among your staff, community leaders, and/or partner with other organizations by engaging them in the process.  

*If you do not have a person on your staff that can carry out the process, you should consider hiring an external evaluator. For more information that can guide you through that process please check out Choosing an External Evaluator section.