preschool literacy initiative

public librarian recruitment

certification

PLA and ALSC's Preschool Literacy Initiative

Evaluation FAQ

By Virginia Walter

There have been a number of questions about the evaluation component of the PLA/ALSCl Preschool Initiative. Here are answers!

1. Why do we have to do an evaluation?
   
   The people that President Bush has appointed to top positions in the Department of Education have pushed forward the principle of “research-based practice.” You have probably heard this phrase over and over again from George W. Bush as he talks about his education plan – which has been enacted by Congress.
   
   The people who devised the materials that you are using in this pilot program are now among those top decision-makers in the U.S. Department of Education. These materials are “research-based.” PLA contracted with the people who developed the materials based on research findings and who tested the materials with low-income parents. While the people responsible for creating the materials are pleased to see public libraries taking an active role in early literacy development, they are also skeptical that public libraries can really make a difference. They point to the lack of research that supports public library effectiveness in this area.
   
   One of the primary objectives of this initiative is to produce research findings that will demonstrate whether or not public libraries can produce positive outcomes in parent education interventions. Note that we are asking for both OUTPUT measures, the quantitative record of how many sessions you hold and how many people you reach. The intake and post-interviews, however, are designed to get at the more difficult OUTCOME measures, to determine if your intervention made any difference to the people you trained. Do they think differently as a result? Are they doing anything differently now? Since this is a national study with more than 20 participating sites, we think we will have some interesting results to share. It is possible that we will be finally able to demonstrate our credibility as a provider of early literacy caregiver education. (Please don’t try to skew the data to demonstrate outcomes that weren’t there, however! We’re counting on you to be reliable data gatherers! )
   
   The evaluation will also help us to judge the effectiveness of the materials themselves.

2. Won’t people just tell me what they think I want to hear? How can I trust what they say?
   
   Self-reporting is a valid form of data collection. In this case, since we are evaluating an education intervention, it will be a positive outcome if they have simply learned what the “right” answers are. That means that they learned something! However, by asking for an example of their behavior in several instances, you are acquiring concrete data. If they say the “right thing” but can’t provide an example, they probably have not yet changed their behavior to conform with the knowledge they have acquired.

3. I don’t feel comfortable asking library patrons personal questions like these. Can’t we change the wording?
   
   No. Sorry. In order for the research findings to be reliable, you all need to ask the same questions in the same way.
   
   I think it’s helpful if you think about the questions as being analogous to a reference or readers’ advisory interview. You often have to probe a little in order to find out exactly what the patron needs to know. Sometimes you have to ask WHY the patron needs the information or what level of information is required.
   
   These interviews will also give you valuable information about the parents and caregivers you are working with that will enable you to partner more effectively with them as they develop their knowledge about early literacy and their techniques in teaching the young children in their care. Think of the pre-interview as a diagnostic of the caregiver’s needs and the post-interview as an opportunity to check up on your own teaching skills.
   
   True professionals in all fields are accustomed to dealing with their clients on an individual basis (as we do too, at the reference desk). Doctors, social workers, attorneys, and therapists all conduct interviews that are designed to tease out the kind of information you are asking for here. People are accustomed to answering questions of service providers. I think if you approach these questions with the attitude of a caring professional, you will be surprised at how willingly people respond.

4. Do we have to do these interviews face-to-face? Can we do them over the telephone? Or give them a survey instead?
   
   Face-to-face interviews probably yield the most information. However, you can do them over the telephone if that works best for you. A written survey is NOT an acceptable substitute, however. The experience of completing a written survey is too different from answering questions orally; the results would not be comparable to those obtained in an interview situation.

5. Some of us are doing multiple sessions using multiple trainers. How can we judge the effectiveness of different trainers?
   
   Good point. We didn’t think of this when we devised the evaluation form. We suggest that you add a line at the top of the form on which you are recording data that reads: “Name of presenter or trainer.” When we analyze the data, we will probably not be able to determine the effects of different trainers at particular libraries; but you will have this information for your own use.

6. Do we have to record the sessions or make transcripts of them?
   
   No, you don’t have to record or make transcripts. Just take good notes of the respondent’s answers. Try to capture verbatim any particularly pertinent responses and put quotes around them so we know that these are the person’s own words. You can send us your handwritten notes if they are legible. Otherwise, please use a word processor to copy them before you send them.

Please feel free to contact Elaine Meyers or Virginia Walter if you have any particular questions that I didn’t answer here!