Friday, November 13, 2009

Was this a mistake?

I struggled with one of my questions, and the struggle continues...

I wanted to have a question that would draw out opinions and attitudes towards provincial curriculum documents, without the question biasing the response. Essentially I was fishing for "beefs".
I settled on the following question:

"What do you believe is the main goal of the course as presented by Provincial Curriculum documents? "


I expected responses that would either make an attempt at answering the question at face value and perhaps a few "the goal of the ELA curriculum is make students hate reading novels”. I hoped to pull those quotes for use in Professional Development discussions as conversation starters.

I am getting several of the first kind of response, but in addition I am getting a healthy handful of " I don't understand the question.

Hmmmmm....
What would have been a more clear question I could have asked to achieve my goal?

Thursday, November 12, 2009

Survey o Rama

I created a survey to gather some baseline data surrounding the theme of "Relevance" and tested it with a few non-teaching staff. It did not go over well. It was too wordy, and had several sneaky distractor phrases in it I had not caught.
example:

"I feel I have been given the freedom to modify my course content to make it as relevant as I feel is possible."

The "I have been given" statement didn't allow for those who had simply "taken" the freedom. In this case, I was less interested in wether teachers felt "allowed" to do something than if they felt "free" to do it.

I orignally tried to set the survey up on the U of S site, but abandoned it when I saw how less "user-friendly" it was in comparison to the unfortunately named "Survey Monkey.com"

Here's a draft of the final question set I sent out to Staff:




I have been employed as an educator for:

0-9 Years
10-19 Years
20+ Years

Rate your level of agreement with the following statements:
Strongly Disagree
Disagree
Somewhat Disagree
Somewhat Agree
Agree
Strongly Agree

I consider myself a subject expert in at least one of the courses I am teaching this trimester.

I feel students at my school are receiving an education that addresses their basic needs.

I feel students at my school are receiving an education that addresses their academic needs.

I feel students at my school are receiving an education that addresses their social and emotional needs.

I have the freedom to modify my course content to make it as relevant as I feel is possible.

I have the freedom to modify my teaching methodologies to make the course as relevant as I feel is possible.


During the last month, I have heard a student make the following statement, or one with similar meaning:

0 times 1-3times 4-8times 9 or more times


This is stupid!

Why do we have to do this?

How much do I have to do?

Can’t you just give us the answers?

Am I going to pass?

This is interesting!

Can we (insert a learning activity the student is suggesting)?

This will help me (insert a future application for the skill the student is learning)

Don’t tell me the answer!

When answering the following open-ended questions, please indicate the general subject area at the beginning of your response. You may opt out of these questions if you have any concerns regarding anonymity.

What portion(s) of your course are relevant?


When answering the following open-ended questions, please indicate the general subject area at the beginning of your response. You may opt out of these questions if you have any concerns regarding anonymity.

What portion(s) of your course are relevant?

What portion(s) of your course lack relevance?

What portion(s) of your course lack relevance? 6. What do you believe is the main goal of the course is presented by Provincial Curriculum documents?

What do you believe is the main goal of the course as presented by Provincial Curriculum documents?

What barriers, if any, are there to presenting the course in a way that is most relevant to students?


What barriers, if any, are there to presenting the course in a way that is most relevant to students?


I will show you all the on-line version of the survey in class. I would include the link here for you all to peek at, but I'm afraid of empty response number creeping into my data....

Friday, October 23, 2009

Planning a Program Evaluation: Worksheet

Engage Stakeholders

Who should be involved?

Board Administration:
School based administration:
School based student services:
School based Learning Coordinators/ Learning Leaders : School based teaching and Educational assistance staff
Students

How might they be engaged?
Through participating in the design, implementation, and reporting of results.

Focus the Evaluation

What are you going to evaluate? Describe program (see logic model).

What is the purpose of the evaluation? _To determine the effectiveness of the relevance potion of the collegiate renewal program


Who will use the evaluation? How will they use it?

Who/users How will they use the information?

Students and teachers To inform their practice (what they are doing
during the evaluation)

Teachers and school based To inform and guide future practice
administration (what they will do post-evaluation)

School based administration To account for programs that are successful
and board administration so that they can be replicated, or
unsuccessful so that they can be addressed.


What questions will the evaluation seek to answer?

Does ____________ programming have an effect of student perception of their learning purpose and, if so, what is the effect and how was it made.
OR
Main Question:
Did students learn the relevance of schooling?

Sub Questions:
How aware were students of the relevance of schooling before it was addressed through collegiate renewal?

How often are teachers informing students of the relevance of what is being studied?

In what ways are teachers informing students of the relevance of what is being studied?

What support materials are provided and used by teachers and students to connect topics being studied and real-world application?

Is the attempt to make schooling more relevant successful?

Should increasing students’ awareness of relevance be a focus for 2010-2011?




What information do you need to answer the questions?

What I wish to know Indicators – How will I know it?

What are students’ understandings of their purpose in regarding to their education?

 Quotes will include indicator terms and phrases such as “I need my grade 12 to get a job” (indicated ritualistic participation) or personal relevance terms such as “like”, “enjoy”, “interested”, “understand” and “help”

How often are teachers informing students of the relevance of what is being studied

 Frequency charts, self reporting, interviews

In what ways are teachers informing students of the relevance of what is being studied?
 Frequency charts, self reporting, interviews

What support materials are provided and used by teachers and students to connect topics being studied and real-world application?
 Material and document review, observation

Is the attempt to make schooling more relevant successful?
 Students will be able to articulate how their engagement in school will affect their lives outside of school and into the future.
 Students will be able to articulate what skills they are being offered and how those skills are/will be useful in the future
 Students may be able to articulate how learning continues beyond the classroom and beyond their school years.
 Teachers will perceive a decreased need to “prod” or “cajole” students into engaging in a task or activity.
 Success/retention rates shown in number of credits earned year to year

Should increasing students’ awareness of relevance be a focus for 2010-2011?
 The success of moderate intervention during the school year

When is the evaluation needed? May 2010

What evaluation design will you use? Mixed Method Design

Collect the information
What sources of information will you use?
Existing information
: _______Tell them From Me data__________________

People: ____ School based administration: School based student services:
School based Learning Coordinators/ Learning Leaders 1 :
School based teaching and Educational assistance staff
Students

Pictorial records and observations: __Observation logs from __________and _________

What data collection method(s) will you use?

Survey
Interview
Observation
Document review
Testimonials
Journal, log, diary
Other (list) ___Credit review

When will you collect data for each method you’ve chosen?

Method Before program During program Immediately after Later
Survey X X
Interview X X
Observation
(Administration
and School
counselor
to keep
visitation
logs) X X
Log X
Document review X
Credit Review X X

Instrumentation: What is needed to record the information? Computer
Flip Camera
Digital Audio Recorder
Transcription software

Will a sample be used? No___________________

Analyze and Interpret

How will the data be analyzed?
Data analysis methods:,
video, textual and numerical data analysis

Who responsible: ___

How will the information be interpreted—by whom? The information will be interpreted in a report written by Lesley Walters and vetted by a selection of stakeholders

Use the Information

How will the evaluation be communicated and shared?

This has yet to be determined. While I can expect that the report will be shared with teaching, administration and para-professional staff, I has not been decided if this information will be appropriate beyond that scope. I have been counseled to keep the project in a limited sphere until it is completed and has been viewed by administration.


Notes:

1) Learning Coordinator is a position formerly referred to as Department Head
Learning Leader is a position created in 2007 to support the collegiate Renewal Initiative
2) Relevance
"Motivation is the reason for being engaged and relevance helps to provide that compelling reason. Learning is often engaging to the extent that students deem the learning as meaningful and interesting to self and world. “Contextualized learning involves being able to see the value and relevance of the skills [and understandings] being learned as opposed to learning that is abstract and divorced from real life (Feuerstein, 1980, Haywood, 1993). Learning is best placed in meaningful contexts that show its inherent utility and capitalizes on students’ interest” (McLean, 2003).

When will I ever use this in the “real world”? This is a perennial student question and thoughtful teachers consider relevance when viewing curriculum and considering instruction. The sentiment is reflected in our instructional planning question: “Why would students give their hearts and heads to this work?” For we know, learning is ultimately the choice of the learner. Still, we also understand that students must take responsibility “for generating interest, if they find the work boring, by finding ways to make it more challenging and worthwhile for themselves” (McLean, 2003). "
Taken from the Saskatoon Public schools Collegiate renewal website on October 15, 2009 http://schools.spsd.sk.ca/collegiaterenewal/

Saturday, September 19, 2009

Assignment #2

An Approach to Evaluating ECS Programming for Children with Severe Disabilities



Overview:

The ECS Programming for Children with Severe Disabilities document describes the programming offered from Alberta Education to Children with severe/profound disabilities aged 30 months to 6 years of age. The programming is provided in both community and home-based settings and involve educators, therapists, assistants and families in a blended program. The programming is regulated and must meet strict standards involving initial diagnosis and application, programming oversight, and time.


Assessment:
In my experience, the issues surrounding the effectiveness of a program often come down to the fit of a bureaucratic configuration to the system it seeks to guide. If a professional bureaucracy is to be effective, rules and regulations need to be understood as relevant, reasonable and acceptable to all parties involved. The quality and delivery of the organization’s professional development is also an important factor. Professional development is often seen as a process of accreditation, rather than pool of resource. When examining a program’s professional development, we ask if the stakeholders have the relevant information and support to make the best use of the resources available to them.

When I read the program description, the majority of questions I had were related to the effectiveness of the organizing system to apply an iterative process of exploring the effectiveness and relevance of its policies. My questions were:


Timelines:
What is the expected timeline between a diagnosis, and the child receiving the required additional assessment by qualified personnel, such as a Speech/Language Pathologist, Pediatrician, Chartered Psychologist or Child Psychiatrist?
Does this match the actual time line?
Applying:
Is the process of being qualified for the funding severely hindering the effectiveness of the program? If so, could suggestions on how to expedite the process be gathered in the process of evaluation?
Is the process of re-applying each year onerous to the point of hindrance and, if so, what suggestions can be presented by stakeholders.
Is the process not thorough enough? Are there a proportion of children in the program who don’t fit it?
Coordination:
How well coordinated are combined programs? Is there a cogent facilitation plan in place to allow to communication between programs provides in the case of a combined program?
Professional Development
Are the teachers in charge of developing the Individual Program Plans (IPP) comfortable with their level of training and expertise to be able to create these plans?
Are the other stakeholders comfortable with the level of teacher training and expertise to be able to create these plans?
What resources are available to teachers for building and implementing IPPs?
Are the Teachers aware of resources available for the child, and for the building and implementing of IPPs?
What level of training and resource is being provided to home caregivers to allow programming benefit to extend beyond the home visits?
Regulation:
What constitutes the visit time measurement of 1.5 hours?
Is it 1.5 hours of direct contact with the child, or does that time include documenting the visits, briefing and/or debriefing with the caregiver?
Morale:
What attitudes and opinions do supervising teachers and programming providers have towards their level of responsibility?

Approach:

I would take a two-pronged approach. Firstly, I would engage the stakeholders with a questionnaire that would explore the various groups ‘answers to these questions. This would give me a reading of the overall institutional health of the organization.

Often, when measurements of attitudes towards things like authority structures, enrollment management, remuneration and professional development are attempted; they are done so using survey rating scales. I disagree with applying them in this case. Asking about degrees of agreement or dissent doesn’t provide the opportunity for stakeholders to provide their input into building a better system, or lending argument to why beloved systems should remain unchanged. This could be considered a participant-oriented model.

Secondly, I would invite stakeholders to begin the process of creating a logic model. The following image from The University of Wisconsin’s Program Development and Evaluation Center gives and overview of how a program is interpreted through a logic model. The ability of a logic model to provide a common language and clarified, commonly determined outcomes would be of benefit to this organization




In closing, I would caution against attempting to quantify outcomes in an organization such as this. This form of feedback can be misleading as the results depend as much on the children and their individual abilities and circumstances, as it does on the program and its activities.






-

Wednesday, September 16, 2009

Step 1: my list of questions (focusing required)

-What is the expected timeline between a diagnosis, and the child receiving the required additional assessment by qualified personnel, such as a Speech/Language Pathologist, Pediatrician, Chartered Psychologist or Child Psychiatrist?

-Does this match the actual time line?

-Is the process of being qualified for the funding severely hindering the effectiveness of the program? If so, could suggestions on how to expedite the process be gathered in the process of evaluation?

-Is the process of re-applying each year onerous to the point of hindrance and, if so, what suggestions can be presented by stakeholders.

-Is the process not thorough enough? Are there a proportion of children in the program who don’t fit it?

-How well coordinated are combined programs? Is there a cogent facilitation plan in place to allow to communication between programs provides in the case of a combined program?

-Are the teachers in charge of developing the Individual Program Plans (IPP) comfortable with their level of training and expertise to be able to create these plans?

-Are the other stakeholders comfortable with the level of teacher training and expertise to be able to create these plans?

-What resources are available to teachers for building and implementing IPPs?

-Are the Teachers aware of resources available for the child, and for the building and implementing of IPPs?

-What level of training and resource is being provided to home caregivers to allow programming benefit to extend beyond the home visits?

-What constitutes the visit time measurement of 1.5 hours?

-Is it 1.5 hours of direct contact with the child, or does that time include documenting the visits, briefing and/or debriefing with the caregiver?

-What attitudes and opinions do supervising teachers and programming providers have towards their level of responsibility?

Sunday, September 13, 2009

I'm obviously a little touchy

Looking at my assigned case study I found a quote, whose purpose was obviously to inspire.
Instead it irked me, and I found myself looking back at the case more critically.

"A child can see a painting, but it takes a teacher to unlock the beauty that is contained within it.”

Is it just me, or does that seem a bit dismissive?

Anyhow, I like to take clear note of when ever I find myself being biased. I'll try and put more weight to the notes I took on the first reading, rather than the second pass.

Social Science Research vs. Program Evaluation

I found this little ditty entitled Michael Scriven on the Differences Between Evaluation and Social Science Research

"Evaluation determines the merit, worth, or value of things" while "Social science research does not establish standards or values and then integrate them with factual results to reach evaluative conclusions"