University at Buffalo Course and Teacher Survey: UBCATS

9. Who has access to evaluation reports?
The teacher and the department chair may view the evaluation report. The Dean may also review reports. (Courses with multiple instructors have more complex choices which are still under development.) Excerpts from the report are customarily available as follows: tenure and promotion dossiers contain summative means and may include excerpted written comments; the Student Association publishes summative means of undergraduate lectures and seminars only (no written comments, no data on lab and recitation sections); a “Chair's Report” which summarizes findings is also produced; and additional summary reports are under development (e.g., top-rated large lecture courses; sections with high response rates; sections with too few responses).

12. Which factors may bias student ratings?
According to campus experience and publications about evaluation, student ratings are not free of bias and may vary with factors other than course content, learning outcomes, assigned readings or grading policies.

13. What is the ‘real’ and ‘accurate’ course rating?
Generally, students rates courses and instructors highly (above average.) Students voluntarily rate their perceptions of courses and instructors on a 5-point scale and with written comments. Student evaluations are only a part of evaluating courses and instructors.

14. Is there bias related to general education courses?

Courses typically used to complete general education requirements are rated slightly lower as are large, required, prerequisite, lower division courses. Note that a single course may meet all or some of these criteria. Department means and overall means are stable from semester to semester.

15. Is there bias related to ‘courses in the major evaluated by majors’?
Under review.

16. Is there bias related to small courses and very few responses?
When sections are small, the results are more variable because each response is a larger portion of the total. The consequence is that such courses show volatile ratings from semester to semester. Very small enrollment courses also constitute a high percentage of the very highly rated courses. Caution in reporting these results is appropriate.

17. Is there bias related to response rate?
Survey completion rate was 43% of surveys by 60 % of the students for the first online administration and 42% for the second semester administration. Higher response rates are associated with higher mean ratings but the means seem otherwise comparable. (However, please note that a high percentage of these high response rate courses are also very low enrollment courses where the ratings are already typically higher.) The results remain tentative and the association does not imply causation.

18. What can faculty do to raise response rates?
Teachers of classes where over 75% of the students participated in UBCATS were asked what they had done to encourage students to respond. Some answered that good courses, well taught, were sufficient to get high response rates (i.e., they did 'nothing'.) Nearly all the instructors who had a large response rate stressed to their students that their opinions mattered and that their feedback could help the instructor improve the course. Most who achieve a high response rate simply remind their students.often. They verbally remind their students, send e-mail messages to their students, and post announcements about UBCATS on UBlearns. Response rate is not related to grading nor is response rate related to the percentage of students with strong opinions (i.e. a form of retribution by mad students.)

19. Is there bias related to Online vs. In-Class Administration?
Comparing data from fall ’03 (in-class, bubble sheet administration) with fall ’04 (on-line UBCATS administration) the distribution of course means is comparable. Similarly, the same courses taught by the same faculty have comparable mean ratings, as do departments. There is variation, but not consistently, in any direction.

20. Is there bias related to departments or programs with few sections and few students?
The course and department means tend to be high.

21. How many sections and faculty were evaluated by how many students?
In Spring 2005, 68867 potential surveys were evaluated by a maximum of 16851 students taught by 1202 faculty/TAs in 2428 sections. Evaluation of sections increased from about 1700 to about 2200 between 2000 and 2005 while the number of completed surveys remained about the same.

	Surveys	Sections	% Completed
Spring 2000	19,400 (27%)	1,017 (46%)
Fall 2000	25,300 (35%)	1,069 (49%)
Fall 2003	33,400 (46%)	1,313 (60%)	(70% within reporting sections)
Fall 2004	31,100 (43%)	2,450 (100%)	(43% within reporting sections)
Spring 2005	28,867 (42%)	2,621 (100%)	(42% within reporting sections)

Note: In Spring 2005, 42% of students completed the surveys, 6% logged in but did not complete any surveys, and 18% logged in and completed some surveys.

22. How do departments and programs differ on student generated ratings?
On a scale from 1 (poor) to 5 (best), for questions about ‘recommending’ the course and instructor, department means ranged from 3.4 to 5.0 in this semester sample. Note that departments and programs with just a few evaluated sections (marked with *) tend to be rated very highly; all departments are rated ‘above average’; and departments with a high percentage of enrollments in beginning courses, several large enrollment sections, and more required courses, are more likely to receive lower overall mean ratings.

Fall 2003 UBCATS	Question 10 (Mean = 4.2)	Question 9 (Mean = 4.0)
	"I would recommend instructor"	"I would recommend course"
African-American Studies	4.4	4.5
American Sign Language*	4.7	4.7
American Studies/Ctr. For Americas	4.1	4.1
Anthropology	4.3	4.2
Arabic*	4.7	4.6
Art	4.2	4.1
Art History	4.4	4.2
Asian Studies	4.2	4.1
Biological Sicences	3.6	3.4
College of Arts and Sciences*	3.9	3.6
Chemistry	4.0	3.7
Chinese	4.2	4.2
Classics	4.5	4.4
Communicative Disorders	4.3	4.1
Comparative Literature	4.5	4.3
Danish*	4.6	4.8
Economics	4.0	4.0
English	4.3	4.2
French	4.4	4.1
General Education Core (UGC)	4.0	3.8
Geography	4.3	4.2
Geology	4.6	4.3
German	4.6	4.3
Greek Ancient*	4.5	4.0
Greek Modern*	5.0	4.8
Hindi*	4.2	4.2
History	4.4	4.2
Humanities*	3.5	3.8
Interdisciplinary Degree Program (SSC)	4.6	4.5
Irish*	4.9	4.9
Italian	4.7	4.5
Japanese	4.7	4.6
Legal Studies*	4.2	4.3
Judaic Studies Program	4.4	4.3
Korean	4.4	4.4
Latin*	4.8	4.5
Latina/Latino Studies	4.8	4.8
Linguistics	4.1	3.9
Mathematics	4.0	3.9
Media Study	4.2	4.1
Medicinal Chemistry*	3.6	3.5
Music	4.2	4.1
Music Theatre*	4.9	4.9
Philosophy	4.4	4.2
Physics	4.2	3.8
Planning and Design	4.3	4.1
Polish*	4.9	4.8
Polish Studies*	4.2	4.1
Portuguese*	5.0	4.9
Political Sicence	4.2	4.0
Psychology	4.1	4.0
Religious Studies Program*	4.6	4.5
Russian*	4.7	4.6
Sociology	4.3	4.2
Spanish*	4.4	4.2
Theatre	4.5	4.4
Theatre and Dance	4.7	4.7
Undergraduate Learning Center	4.5	4.1
Women's Studies	4.1	4.1

Note: *= fewer than 5 sections evaluated
Source: Fall 03 Summary of 1,313 sections and 33,367 student responses.

23. How do survey responders and non-responders differ in class performance?
Students who complete their surveys are more often the more highly graded students in the section. The results of a pilot study are presented below. Comparing the percentage of A and F grades and class grade point averages between responders and non-responders shows that the responders had higher grades and higher GPA regardless of course level (Lower Division, Upper Division, Grad), regardless of discipline (GLY, PHY, PSY, COM, HIS, SOC), and regardless of class response rate.

			Responders	Non-Resp	Responders	Non-Resp	Responders	Non-Resp
Course Level	Enroll	Resp Rate	(R) GPA	(NR) GPA	(R) w/ 'A'	(NR) w/ 'A'	(R) w/ 'F'	(NR) w/ 'F'
HIS 1XX	173	39%	C+	C-	24%	10%	3%	16%
HIS 1XX	47	34%	B+	B-	63%	39%	0%	10%
GLY 1XX	192	39%	C	C-	17%	12%	7%	9%
GLY 1XX	72	38%	B	C+	33%	16%	4%	11%
SOC 1XX	47	36%	B+	B-	53%	27%	0%	3%
SOC 1XX	245	36%	B-	C	26%	8%	2%	13%
1XX Avg	129	37%	B-	C+	36%	19%	3%	11%
HIS 3XX	20	45%	B	C+	44%	27%	0%	27%
HIS 3XX	45	24%	B	B-	46%	15%	9%	9%
GLY 3XX	20	45%	B-	C	33%	9%	0%	9%
GLY 3XX	34	44%	B-	B-	47%	26%	7%	5%
SOC 3XX	32	44%	B	C+	43%	28%	0%	11%
3XX Avg	30	40%	B	C+	43%	21%	0%	12%
HIS 5XX	13	54%	B+	B	57%	17%	0%	0%
GLY 5XX	12	83%	A-	B	60%	49%	0%	0%
SOC 5XX	12	50%	B+	A-	67%	67%	0%	0%
5XX Avg	12	62%	B+	B+	61%	44%	0%	0%
DMS 3XX	20	90%	B+	B+	67%	0%	0%	0%
COM 1XX	41	88%	A-	A-	86%	61%	0%	0%
CDS 6XX	21	86%	A-	A-	66%	68%	0%	0%
PHI 2XX	30	83%	B-	F	40%	20%	0%	78%
PHY 1XX	26	77%	A-	B	70%	0%	0%	0%
PSY 6XX	17	76%	B-	A-	46%	74%	0%	0%
HighResp	26	83%	B+	B	63%	37%	0%	13%

Questions Faculty/Chairs/Administrators Ask