Teagle Grant Project:

The Philosophy Program at Beloit College

Phil Shields, Matt Tedesco, Heath Massey

June, 2008

I. Task

Shortly before undergoing an external review in fall 2006, the members of the Department of Philosophy

and Religious Studies agreed to participate in “Assessing the Value Added to Liberal Education by

Academic Majors.” For the departmental self-study required for this review, we were charged with

considering how the philosophy and religious studies majors support the general education goals expressed

in Beloit College’s recently-revised mission statement. So, it appeared that we would be in an excellent

position to contribute to the Teagle Foundation project by examining “the complementary relationship

between general education and the major[s].” This examination, we learned, would focus on three

particular emphases of a liberal arts education: critical thinking, civic engagement, and quantitative

reasoning. Our task would be to determine how our coursework and our advising contributes to the

realization of these general education goals and then to devise ways to pursue these goals more effectively.

Our self-study in the fall of 2006 gave us the chance to begin discussing the role of our department with

respect to the mission and goals of the college. Then, in the spring of 2007, we focused on what kind of

information we needed and what instruments could provide it. We decided to use examples of student

writing from different stages of their coursework as a means of direct assessment, as well as exit surveys

and transcript breakdowns for indirect assessment. By the end of the academic year 2006-7, we possessed

writing portfolios and questionnaires provided by graduating seniors, as well as transcript breakdowns for

philosophy majors and minors dating back to 2000. Over the summer of 2007, we began to process this

information by reading all the materials and discussing possible strategies for addressing what we learn

from them. This work has continued through academic year 2007-8, which has yielded a new batch of

survey and portfolio data, and during which we began to implement some of the changes that followed

from our early work on this project, as well as our self-study.

II. Methods and Instruments

In order to assess the value of our department’s contributions to liberal education at Beloit, it was necessary

to identify methods for determining how our major benefits students with respect to the three areas of

emphasis specified in the grant project. Next, it was necessary to produce instruments for gathering the

information needed for this assessment. In the humanities, this is somewhat challenging, for much of what

we teach—critical thinking, as a prime example—does not readily allow for quantitative measurement. We

were somewhat skeptical about the possibility of measuring outcomes with respect to our educational goals.

So, the challenge was to determine how to find data that would help us accomplish the task of this project

without substituting a reductive, piecemeal form of analysis for the holistic approach to student learning

that is central to philosophy and religious studies.

As our department is comprised of two distinct programs, we approached our task in different ways,

resulting in the employment of different instruments. On the philosophy side, we chose to use three

primary instruments:

1. Questionnaire for Graduating Majors

In 2005, we developed a brief but relatively detailed exit questionnaire for our graduating philosophy

majors so that we could learn from their experiences in the program. The questionnaire, as initially

developed, was comprised of seven open-ended questions, such as: “If you were given free reign to

redesign the department (including classes offered, requirements for the major or minor, etc.), what

changes would you make, and why?” Student responses to these questions therefore focus on their

particular concerns. In spring 2007, in response to the charge of this assessment project, we added six

“circle-the-number” questions which ask students to rate their agreement or disagreement with a statement.

In spring 2008, we further modified the questionnaire, expanding the “circle-the-number” questions to ten

in order to get better, more precise feedback from our graduating students. We designed these questions to

address the specific issues of critical thinking, civic engagement, and quantitative reasoning, or the more

general topic of the relation of coursework and advising in philosophy to all-college learning objectives.

For example, students are asked whether they agree or disagree with this: “Advising sessions with

philosophy professors helped me to make good choices about courses to take in other fields.” The purpose

of these questions is to help us to gauge our students’ perception of how well the philosophy program

supports the general education goals in question.

2. Transcript Analysis

Critical thinking is a skill that we aim to develop in most if not all of the courses we offer. We are less

focused, however, on promoting quantitative reasoning and civic engagement, except to the extent that

courses in ethics will enable students to be more reflective when they do engage in political or community-

based activities. Consequently, we need to know whether students who major or minor in philosophy are

cultivating these competencies in their other coursework. In order to help us determine this, we requested

from the registrar a breakdown of course distribution by division for all majors over the last decade, with

interdisciplinary studies and study abroad included separately. Our intention was to use this data to track

general trends in the kinds of classes our students are (and are not) taking over the course of their time at

Beloit. Since this data does not include which particular courses were taken by each student, it cannot

provide a detailed picture of whether and how students pursue general education goals in their coursework

outside philosophy. To address this concern, we supplemented what we have gleaned from the registrar’s

data by reviewing actual transcripts of recent philosophy majors with attention to particular kinds of classes

that stress quantitative reasoning and to study abroad and internships which demonstrate a concern with

civic engagement.

We began our transcript analysis by asking the Registrar’s Office for copies of the transcripts of all

philosophy majors from the graduating classes of 2000 to 2008. Then we examined these transcripts for

courses and certain co-curricular experiences related to quantitative reasoning and civic engagement. In

addition, we looked at courses in foreign languages and courses related to creative expression, which are

two other areas we believe are central to a good liberal education. We created a spreadsheet including each

student’s year of graduation, major (in addition to philosophy), minor, and coursework related to

quantitative reasoning, civic engagement, and languages and arts. Under each category, we listed courses

that the student took which promote these general education goals. Since Beloit has no college-wide

requirement addressing these goals specifically, there are no general standards for which courses promote

them. Consequently, we had to use our judgment in determining which courses to include in our analysis.

Under quantitative reasoning, we included most courses in mathematics, computer science, physics, and

economics, as well as courses in biology, psychology, sociology, and anthropology that focus on statistics.

Under “civic engagement,” we included courses in ethics, political science, and a variety of other

disciplines, as well as internships, study abroad, and domestic, off-campus study. In order to determine

whether to include a course in this category, we considered whether it appears to prepare students to act

thoughtfully and ethically. Finally, we created our own category “language and arts,” under which we

included art, dance, music, theater, and creative writing. After creating the spreadsheet displaying each

student’s cousework in these areas, we counted the total number and the percentage of students who had

taken courses in these areas. The results of our transcript analysis are presented in Appendix B.

3. Writing Portfolio

By the time they are seniors, all philosophy majors will have taken Introduction to Philosophy, one or more

ethics courses (including Ethical Theory, Environmental Ethics, Biomedical Ethics, and other topics), and

Colloquium in Philosophy, a capstone course in the major. These common experiences provide an

opportunity for the direct assessment of student learning. In philosophy, as in other areas in the humanities,

writing assignments provide not only a basis for student evaluation, but also a means for their reflective

engagement with the subject matter. Writing carefully and thinking critically go hand in hand. Also, most

major assignments in our courses are papers or essay exams. Through our discussion of how to directly

assess the contributions of coursework in philosophy to critical thinking, civic engagement, and

quantitative reasoning, we decided that it would be beneficial to review student writing at various stages.

In order to make this possible, we asked all senior philosophy majors to submit a writing portfolio

comprised of a paper submitted in their first philosophy class, a paper from an ethics class, and the research

paper they produced in the capstone colloquium. Over the summer of 2007, we reviewed the portfolios

from that year’s majors with the aim of discovering how the quality of critical thinking they exhibit

improves (or does not). Rather than beginning with rubrics for assessment of the paper assignments, which

is one approach taken by our colleagues in religious studies, we first read the papers, focusing on what

signs of good thinking they do (or do not) display. We compared each student’s early, middle, and later

work to determine what kind of development is apparent. At the end of the process, we discussed our

findings and attempted to articulate what, in our collective view, characterizes sound critical thinking and

to identify ways that our courses appear to be more and less successful in cultivating it. More recently, we

have collected those same portfolios from our 2008 graduating majors, with the goal of repeating the

review undertaken last summer. Two sample portfolios are included in Appendix C.

III. Self-assessment

Participating departments in this grant project have been charged with assessing their majors with respect to

three particular learning goals: critical thinking, quantitative reasoning, and civic engagement. We will

address each of these individually.

1. CRITICAL THINKING

Critical thinking is rightly thought to be central to a philosophical education, though what it means to think

critically takes many different forms depending on the course and the instructor. We try to teach students

to recognize arguments, to analyze them, and to respond critically in a variety of ways. We are all

committed to the idea that thinking critically involves being able to recapitulate a line of reasoning and

engage in dialogue, raising objections and concerns, to see if that reasoning approaches the truth. At one

extreme we teach deductive reasoning, where students learn to evaluate the validity or invalidity of

inferring one statement from another or other statements. A valid deductive argument is one whose

premises provide conclusive grounds for the truth of the conclusion. Many arguments and inferences made

in philosophy, as in life, do not lend themselves to such definitive analysis, and most of our courses involve

inductive reasoning as well, where inferences are not simply valid or invalid, but gray and probabilistic.

Inductive arguments can only be evaluated as strong or weak, depending on the degree of support the

premises lend to the conclusion. Critical thinking can also involve unpacking the assumptions behind a

statement or argument, uncovering unstated premises and making these underlying assumptions more

explicit. While this in itself need not involve evaluation of the statement or argument, it may at least help

us recognize the implicit complexity and ambiguity, thereby making our understanding and assessment of it

more adequate and responsible. While the deductive reasoning is explicitly addressed in the formal logic

course, it remains implicit in our other courses as a precondition for other types of critical thinking. The

second and third types of critical thinking, inductive reasoning and presupposition analysis, are explicitly

addressed in all our other courses from the introductory level to upper-level seminars.

The senior exit questionnaire provides some indirect means of assessment of critical thinking in general.

On a scale of 5 from “Strongly agree” to “Strongly disagree,” 8 out of the 9 senior exit questionnaires in

2007 were marked 1 for “Strongly agree” to the statement, “The courses I took in philosophy promoted my

critical thinking abilities.” The response to this question was even more effusive in 2008, with all ten

respondents offering strong agreement. And this sentiment was echoed in the other questions focused on

critical thinking that were added to the survey in 2008 (see Appendix A for full details). Taken together,

these responses signal strong agreement among our majors and minors that their study of philosophy

developed their capacity for critical thinking.

Prior to the quantitative questions were some open-ended qualitative questions. In 2007, over half of these

bring up critical thinking as a strength of the department or a capacity they have gained from their

experience in the department. 2007 graduates write, for example: “The department trains its students to

think in a very critical way, which is extremely valuable to me, because I now apply that method in many

other areas of my life;” “I found the critical thinking skills learned in my philosophy classes crossed over to

my science classes (especially in organic chemistry lab);” and “I feel as though the subject, while it did not

prepare me directly for any particular job, I still think about things critically every day, and I feel as though

it has really helped me become a more balanced and calm person.” All of the other four questionnaires

identify closely related skills like “clear analysis” and “effective argumentation” in answer to the open-

ended question as to what they had gained from their experience in the department. Clearly the students

perceive that they are acquiring critical thinking skills.

The open-ended questions in the 2008 surveys reflected these sentiments as well, where the vast majority of

them once again explicitly pick out critical thinking as a clear benefit of their philosophical studies. These

graduates write, for example: “I’ve also become a more analytic thinker and I’m able to identify and

articulate assumptions in arguments and flaws in reasoning. This has helped me in other disciplines as well

as philosophy;” I have gained a love of philosophy and the search for truth, a set of tools for critically

evaluating my own and other’s arguments and beliefs, and an expanded view of the nature of reality;” and

“I’ve gained the ability to think critically, examine arguments, and see situations from many perspectives.

The thought-provoking discussions that took place in my classes in the department, even in Logic,

expanded the way I think and analyze arguments.” Comments like these can be found in virtually every

questionnaire.

Our direct method of assessment involved portfolio analysis. This is both more meaningful and more

difficult to implement. We are looking to track progress in critical thinking skills by examining three

papers from three different stages in the students’ study, ranging from their introductory course, a 200-level

course and a 300-level seminar. We have identified several criteria for tracking progress. First,

intelligibility. To what degree does the paper maintain a coherent focus? Second, critical distance. Is the

writer too attached to initial intuitions and does he or she fail to seriously consider obvious or important

objections? Third, sophistication. Does the paper make inappropriately sweeping generalizations and fail

to qualify statements carefully? And fourth, judiciousness. Does the writer fail to perceive and prioritize

the relative significance of criticisms and objections, or if doing research, the relative significance of the

sources?

2007 was the first year in which we have collected portfolios from graduating seniors in philosophy, and

unfortunately our results are hampered by the fact that we only received four complete portfolios from

graduates. For a variety of reasons, we have struggled with gathering these portfolios in 2008 as well.

Given the limited sample from 2007, we still discerned clear progress in the four areas listed above, though

there were also some disappointments where students seem to have been ill-prepared for the challenges of

writing an independent research paper for colloquium and regressed to earlier levels according to some of

our four criteria. [EXPAND WITH REFERENCE TO APPENDIX C, INCLUDING REFERENCES TO

SPECIFIC PORTFOLIO EXAMPLES] We have implemented changes in our colloquium in attempt to

address these weaknesses. We also plan to be more proactive in collecting portfolios in the future so this

important direct measure of progress in critical thinking will be more significant.

2. QUANTITATIVE REASONING

Of the three learning goals that comprise the emphasis of this Teagle grant project, quantitative reasoning is

the one goal that the philosophy department least engages. That is, while we believe that we are intensely

engaged in advancing our student majors with respect to both critical thinking and civic engagement, very

little of what we do within our departmental courses qualifies as quantitative reasoning. What little we do

with respect to quantitative reasoning can be found in the logic course taught each semester, to the extent

that logic is a kind of precondition for mathematics, and the kinds of symbolic proofs emphasized in formal

propositional and predicate logic model the procedures for derivation underlying mathematical proofs. The

connection here, however, is tenuous at best.

Given the dearth of quantitative reasoning offered in our departmental courses, we have employed two

different strategies for learning whether or not our majors seriously engage this learning goal during their

time at Beloit College. First, we have investigated the transcripts of recent graduates to find evidence of

quantitative reasoning in their choice of courses outside of the philosophy major. There is, however, a

serious limitation to this kind of analysis that should be highlighted here. Some courses, by name and

catalog description alone, provide sufficient reason to believe that the student is engaged in quantitative

reasoning in the course. The clearest examples here are most mathematics courses, as well as discipline-

specific statistics course that are offered in many departments in the natural sciences and social sciences.

But importantly, not finding these kinds of classes does not demonstrate that the student is not engaged in

quantitative reasoning. This is because many courses, depending on the particular instructor and particular

syllabus, may prominently feature quantitative reasoning even if the course title or description fail to make

this clear.

Using our best judgment in what to include under coursework promoting quantitative reasoning, we

discovered that a majority of recent philosophy majors have taken at least one course that we would expect

to demand significant quantitative reasoning, but few of them have taken more than a handful of such

courses. According to our transcript analysis (see Appendix B), Only 22% of philosophy majors

graduating between 2000 and 2008 did not take any courses related to quantitative reasoning. In

comparison, 78% of all students took one or more of such courses, 44% took two or more, and 24% took

three or more. Thus, it appears that four out of five recent philosophy majors have had at least some

training in quantitative reasoning at Beloit, but fewer than half have taken multiple courses in which this

competency is taught, and only about a quarter of them have pursued math, physics, computer science,

economics, or statistics in depth. If we suppose that students who take fewer than two courses in these

areas are likely not to develop good quantitative reasoning, then it is incumbent on us to use advising

sessions to encourage students to make coursework in these areas a higher priority.

Besides transcript analysis, we have also added objective questions that specifically address advising to our

recently-developed senior exit questionnaire. These questions emerge from the idea that, because our

major does not substantially address quantitative reasoning (among other valuable learning goals), the onus

is on us as advisors to see that, as best we can, these goals are being met in the classes that our advisees

choose outside of the major. In discussions about advising prior to the return of the questionnaires, we had

agreed that, generally speaking, we had paid insufficient attention to these kinds of considerations in

advising. The questionnaires confirmed this: in comparison with other objective questions about our

students’ experiences in the major, the scores on questions concerning advising outside the major generally,

and concerning quantitative reasoning specifically, were uniformly lower. The two advising questions in

the questionnaire received the lowest scores (that is, the most disagreement) from our majors and minors,

and the question concerning our advising with respect to quantitative reasoning in particular clearly stood

out from the rest (see Appendix A). Tellingly, with 16 respondents over two years, only two strongly

agreed that their advisor encouraged them to take courses outside of the department that focus on

quantitative reasoning. While the sample size here is admittedly small—at present, just two graduating

classes—the results were clear enough to suggest that we need to pay more attention to our advising in this

respect. These preliminary results raise further, deeper questions about the nature and purpose of

advising—questions also briefly gestured towards in Appendix A—but those questions are largely beyond

the scope of this project.

At this point, it is also worth mentioning that a tension exists in advising between classes that emphasize

quantitative reasoning and classes that emphasize other valuable learning goals that our department does

not significantly address. While this particular grant project concerns three specific learning goals, it is

worth noting that Beloit College’s mission statement explicitly mentions six essential skills that students

are expected to develop, of which quantitative reasoning is merely one. And this list is not exhaustive: it

does not include, for example, creative expression, which seems on the face of it to be no less important a

learning goal than quantitative reasoning. This creates some puzzles about advising that do not have a clear

resolution: advising certainly should not be reduced to a checklist of all imaginable learning goals, and

given that different classes presumably engage these learning goals differently, it is not clear that a student

should be steered away from one kind of class with one kind of learning goal towards a different kind of

class with a different learning goal. And this puzzle, of course, sits against the backdrop of the problem

raised above: namely, that it is not at all obvious which courses meet learning goals such as quantitative

reasoning, given that a course’s name and catalog description very inadequately reveal the actual content of

the class.

To better understand and measure our majors’ progress in meeting these other valuable learning goals

beyond those proscribed in this grant project, we did not limit our transcript analysis to classes with an

emphasis on quantitative reasoning. Instead, we also gathered data from transcripts on the arts and

languages courses taken by our students. It is interesting to compare the two sets of data. For instance,

while only 22% of students took no quantitative courses, an alarming 52% took no courses in a foreign

language. On the other hand, 37% took two or more language courses, compared to 44% in quantitative

reasoning, and 24% took three or more, the same as in quantitative reasoning (see Appendix B). Thus, the

portion of philosophy majors who study foreign languages in some depth—about one quarter—is

comparable to the portion who study mathematics or math-based fields in depth. However a smaller

portion of students take at least one or two language courses, as compared with those who take one or two

quantitative courses. Since there is no general education requirement to study a foreign language, and

Beloit’s “breadth requirement” provides many alternatives for students who wish to take languages, this is

another area that we need to address in our advising. It is particularly disturbing how few philosophy

majors study languages that would enable them to read philosophy in other languages in which it has been

written, especially Greek, Latin, German, and French. In addition to noting courses in the foreign

languages, we tracked courses that encourage students to express themselves creatively. In this area, 32%

of recent philosophy majors did not take a single course in art, dance, music, theater, or creative writing,

while 43% took two or more such courses, and 29% took three or more. So, while a slightly higher

percentage of students pursue the arts in more depth than those who do so in courses related to quantitative

reasoning or foreign languages, almost a third of all philosophy majors have completely missed out on this

part of a liberal education.

3. CIVIC ENGAGEMENT

Of the three learning goals emphasized by this Teagle grant project, civic engagement is in some respects

the most complex. With respect to the other two goals—quantitative reasoning and critical thinking—there

is a sense in which practice is sufficient for learning. That is to say, for students to develop their critical

thinking or quantitative capacity, it is enough that they engage in the practice: we want students simply to

think critically and work with numbers, whatever the context. These skills are like muscles that need to be

exercised, where presumably any number of exercises would be sufficient for this purpose. Civic

engagement, on the other hand, is quite different: we don’t want students to simply act, but we went their

actions to be well-aimed. Presumably, in other words, we want students to direct their civic engagement

well. Civic engagement is valuable only when it is informed throughout by thoughtful and critical

reflection; when civic engagement fails to be thoughtful in this way, it reduced to mere action. This

complexity can be seen most clearly when we imagine contexts where a carefully engaged student would in

fact refrain from taking action. Unlike the other two learning goals emphasized in this project, then, critical

thinking has a qualitative component that the others lack. We don’t merely want our students to act; we

want their actions to be careful and considered, and we want them to know when to act and when not to act.

It is in this respect that civic engagement, appropriately understood, is closely related to critical thinking:

civic engagement requires a certain kind of critical thought on the part of the engaged student.

Given this complexity, different sorts of learning experiences will develop the different features of civic

engagement. One kind of learning experience, certainly, that involves civic engagement is the learning

experience that is practical and requires action on the part of the student. Examples include certain study

abroad experiences and internships. In those two respects, while we explicitly encourage both (particularly

the former), transcript analysis reveals that students have not engaged in either opportunity in recent years

as often as we would like. Our data show that only 22% of recent philosophy majors participated in an

internship or field term, and only 30% participated in a study abroad or domestic, off-campus study

program. Our department, though, is particularly well-suited to cultivating the reflective component of

civic engagement, for the way that many of our courses deal specifically with moral reasoning. Beyond the

fact that ethics has traditionally been considered one of the core areas of philosophy, our department has

particular strengths in ethics. There is an ethical component to many classes that we teach in the

department, and our curriculum is weighted towards courses that are primarily ethical—all of our

introductory classes emphasize ethics, and other examples include Ethical Theory, Environmental Ethics,

Biomedical Ethics, Philosophy of Law, as well as a range of seminars. What these classes all have in

common—and what makes them ideally suited to cultivating the reflective component of civic

engagement—is the way that they challenge students to critically examine the grounds of their values and

their views on a wide range of moral issues.

In our transcript analysis, we tracked the courses taken by our students in both philosophy and other

disciplines that should prepare students for thoughtful, ethical action, and we found that 100% of

philosophy majors who graduated betweeen 2000 and 2008 took at least one such course, and 97% took

two or more (see Appendix B). While this by no means demonstrates that we have been successful at

promoting civic engagement, it does at least suggest that philosophy majors are taking the kind of courses

that will help prepare them to be socially responsible actors in their communities. In our transcript

analysis, we also took note of activities or programs that may serve as indications of philosophy majors’

studies going beyond their regular classroom experiences, such as leadership programs, special projects,

symposium presentations, and even teaching assistantships; however, we have not attempted to draw any

conclusions from their participation in such activities. In connection with our transcript analysis, we also

began collecting anecdotal data about philosophy majors’ participation in student government, community

service, and other extra-curricular activities related to civic engagement. In the future, we will use advising

sessions to gather more information of this kind in order to supplement what we glean from our transcript

analysis.

As a part of this grant project, we have attempted to measure our majors’ developed capacity for well-

aimed civic engagement both indirectly and directly. Indirectly, we have added objective questions to our

senior exit questionnaire that specifically ask whether or not their experiences in ethics courses in the

department developed their capacity for civic engagement. All majors are required to take Ethical Theory,

so every senior will have had at least that one ethics-oriented class; practically, given the number of classes

we offer each semester and the number of classes they need to take to satisfy the major requirements, many

student have had one of more ethics-oriented classes besides Ethical Theory. Again, while the sample size

is small, results here were on the whole much more positive than the results of the objective questions

added to address advising and quantitative reasoning: we were given generally high marks that our ethics

courses did meet this goal. To the statement, “The ethics course(s) I took helped to enhance my capacity for

social responsibility,” our mean score from 18 respondents over two years was 1.67, where marking 1

signals strong agreement (see Appendix A).

While there was a consensus in this respect, a review of the data in Appendix A shows that the agreement

here was not quite as strong as with other statements in the questionnaire. One possible explanation for this

is the deep way in which the philosophical study of ethics is connected with civic engagement, such that it

might not be immediately obvious to very recent graduates. This hypothesis could conceivably be explored

by collecting survey data, delivered via departmental newsletter, from graduates several years after

graduation. We propose such a survey in our action plan (below). Currently, however, in the absence of

this comparative data, a more direct measure of the developed capacity among our majors for well-aimed

civic engagement is needed.

In this spirit, more directly, we have utilized the senior writing portfolios to qualitatively measure our

students’ developed capacity for well-directed civic engagement. These portfolios asked for three papers

from our graduating majors: a paper from their first philosophy class (presumably, though not necessarily,

Introduction to Philosophy), a paper from a 200-level ethics class, and a senior paper from a 300-level

colloquium or seminar. While these portfolios are also being used to measure critical thinking, it is not

surprising that the portfolios are useful here as well, given the close correlation between general critical

thinking skills and the particular critical skill of well-aimed action. Certainly here, the emphasis is on the

200-level ethics paper, where we were looking for evidence of a developed ability (as measured against

their first paper) to carefully articulate a position, to recognize important distinctions, and to anticipate and

respond well to concerns and objections, all fundamentally connected with the notion of acting rightly. As

mentioned above in our discussion of critical thinking, our portfolio analyses are hindered by the fact that

the sample size is so very small, with just four submitted to us in 2007; a further hindrance worth noting

here is that one portfolio of the four submitted in 2007 failed to include a 200-level ethics paper (though

worth noting is the fact that two of the portfolios featured multiple ethics papers from the various levels).

We are still in the process of gathering portfolios from our 2008 graduates, and so have not begun to

analyze them yet in any detail. Our findings from our analysis of the 2007 portfolios map very closely onto

the discussion of our portfolio analysis found in the section of this report focused on critical thinking—

again, this is not surprising, given their similarities. [PROVIDE SPECIFIC TRANSCRIPT EXAMPLE(S),

REFERENCE THOSE INCLUDED IN APPENDIX C] We therefore refer you to the section on critical

thinking in this report for further details.

IV. Summary of Self-assessment

With respect to the three learning goals emphasized in this grant project, then, the philosophy department

sees itself as intensely engaged in advancing its majors with respect to both critical thinking and the moral

reasoning that is essential to well-aimed civic engagement. These two goals are closely related, insofar as

critical thinking lies at the core of moral reasoning, and as such, the single tool of the senior writing

portfolio serves as a direct measure for both of these learning goals. While our initial sample size of

portfolios was quite limited, they have pointed towards the common experience of our senior colloquium as

one area where we might enhance our majors’ development in meeting these two learning goals.

Indirectly, our developing exit questionnaires indicate that we have not, in our advising, adequately

emphasized our attention to learning goals that are largely or entirely not addressed within the philosophy

major, including (though certainly not limited to) quantitative reasoning, the third and last goal emphasized

in this project. These surveys have been encouraging, however, for the ways that our students themselves

highlight the special training we have offered them in developing their critical thinking skills. Certainly,

with both the portfolios and with our survey data, we would benefit from more and better information

going forward.

There is an important tension in our use of writing portfolios that needs clarification at this point. We want

to say that portfolios do, or will when we get a better return, provide meaningful evidence of improved

critical thinking in our students. We discussed several criteria that we may use to evaluate the quality of

the critical thinking reflected in these samples of student writings. But earlier when introducing the

instrument of senior writing portfolios in Section II, Part 3, we refused to develop rubrics in advance to

quantify our evaluation of the writing portfolios. This raises the question: What is the difference between

the criteria we discussed and the rubrics we rejected? Could we not simply use our criteria as rubrics of

assessment? The difference is that the criteria we listed are ad hoc and cannot replace our holistic

judgment as experts. We are merely trying to use the criteria to gesture toward the kinds of things our

discernment picks up, but if there is a discrepancy, we would criticize and modify our criteria to fit our

intuitions/judgment before we would criticize or modify our judgment to fit these criteria. In other words,

the criteria depend on and follow from our holistic assessment instead of the assessment following from a

mechanical summing of pre-established rubrics. That is not to say

that our intuitions/judgments cannot be criticized and that we are not accountable, but only that we are

accountable to the intuition/judgment of other experts in our field (this is the point of "peer-review").

These experts may in turn try to articulate criteria to support or explain their judgment. In fact, we expect

an attempt at articulation and it has an important role to play, but it is still the judgment that carries the

weight. So we do not want to imply that critical thinking eludes adequate assessment, but only that it may

elude assessment by formalized criteria. We believe that there are no mechanical shortcuts which could

bypass expertise and be meaningfully employed by non-experts. Therefore we believe it is inappropriate to

produce rubrics for critical thinking to serve as an algorithm for assessment, preferring instead a qualitative

assessment of critical thinking by experts in the field.

Finally, to return for a moment to the guiding question of this grant project, what exactly is the value added

to liberal education by a philosophy major? At Beloit, we stand committed to the pursuits of “knowledge

through free inquiry” and “personal, social, and intellectual development through multiple paths.” We

share a conviction in “the educational benefits of engaging diverse perspectives” and the vital importance

of “active, responsible citizenship.” We strive to cultivate “a passion for learning within and beyond the

classroom” and to impart the skills of “effective written and oral communication, logical thinking,

…problem-solving, [and] judgment.” While such values and learning goals are represented across the

college in a variety of ways, we believe that a major in philosophy emphasizes them particularly well.

Thinking carefully and responsibly is, of course, one of the primary focuses of the study of philosophy.

Without our having to thematize “critical thinking” as what we teach students to do, there appears to be a

consensus among philosophy majors that our courses train them well in this area. A significant part of our

mission is to provide students with opportunities and topics for sustained self-reflection, and to guide them

in thinking, writing, and speaking about their answers to some of the enduring questions of human

existence. Not only do we train our students in critical reflection on their own views, but we also guide

them in the study of intellectual history, providing them with the vocabulary, concepts, and logical

framework necessary to understand it. In other words, we both encourage our students to engage in free

inquiry and educate them in many of its traditional forms. Yet we recognize that the unity of “our

intellectual tradition” is a problematic one, and we aim to foster students’ awareness of the many points of

view included in and excluded from it. Finally, by maintaining a variety of offerings in ethics, we ensure

that students are exposed not only to interesting theoretical issues, but practical ones as well. Even at the

introductory level, we challenge students to analyze and evaluate different solutions to moral problems, that

is, to exercise their sense of moral responsibility and their capacity for moral reasoning. So, we aim to

ensure that they adopt a critical stance not just to their own beliefs, and not just to the theories of “great

philosophers,” but to their own values and the values embodied by their society and its history.

To summarize, it is primarily by means of these core competencies—critical thinking, knowledge of

intellectual history, and moral reasoning—that the philosophy major promotes the objectives that are

articulated in Beloit’s mission statement and learning goals, and thus adds value to general education at

Beloit.

V. Action

As a result of our self-assessment, we have begun to develop a plan of action for improving student

learning and strengthening the relationship between the philosophy major and Beloit’s learning goals. Our

plan is still in its early stages and subject to revision. At this point, our strategy involves meeting each

semester after advising week, developing a newsletter to foster contacts with alumni, restructuring the

capstone course in philosophy, and employing a variety of activities to further strengthen the philosophical

community at Beloit College.

1. Post-advising debriefings

Our exit questionnaires and transcript breakdowns, while not providing the most detailed information about

student needs, have given us at least an indication of several ways that advising in the major could be

improved. For example, it appears that philosophy majors participate in study abroad less than the average

Beloit student. While the college has set the goal of a 50% rate of participation in study abroad, only about

30% of recent philosophy majors have participated (see Appendix B). Also, many of our recent graduates

did not agree that their major advisors encouraged them to pursue courses devoted to quantitative reasoning

or helped them to make good choices in general about coursework outside of philosophy.

As advisors, the three of us already engage in a lot of informal, “in-the-hallway” discussions about student

needs. One further course of action we plan to pursue is to schedule at the end of each semester’s advising

period a meeting devoted to dialogue about our advising sessions with students. This will help us to

establish and maintain an ongoing conversation about our questions and concerns, surprises and

disappointments, and successes and failures in student advising. During these post-advising debriefings,

one thing we will focus on is how the learning goals of the college as a whole factored into our most recent

meetings with our advisees. We hope that setting aside a time each semester to take stock and exchange

ideas will help us to ensure that we are encouraging students to stretch themselves intellectually and to

make wise choices about their schoolwork outside of the philosophy major.

2. Departmental newsletter

Surveying recent graduates of Beloit about their perceptions of how the philosophy major contributed to

their education as a whole provided us with some helpful information, but students’ perspectives on their

college education will obviously change and hopefully sharpen over time. Especially since there are

typically ten or fewer philosophy majors graduating each year, we concluded that we need to tap into our

alumni in order to obtain more information that will be useful for the ongoing process of self-assessment.

Sending questionnaires to students, say, five years after their graduation from Beloit will serve the dual

purpose of providing another, possibly more mature perspective on the job we are doing and increasing the

size of our sample, thus the amount of available information.

However, the philosophy program has never had any organized way of remaining in contact with our

majors after graduation. To address this lack, we plan to put together a brief annual newsletter that will

allow us to stay in touch and keep our alumni informed of things happening with us and our students. First,

we will turn to the Office of External Affairs to help us organize contact information. Then, the three of us

will design the newsletter, drawing on the experience of other departments who already produce an annual

publication. Finally, we will use our students and alumni with whom we are already in touch to contribute

ideas and material. With any luck, the newsletter will open a channel of information flowing not just from

us to alumni, but in the opposite direction as well, providing us with more feedback on the strengths and

weaknesses of the major.

3. Colloquium renovation

Our first batch of writing portfolios and our questionnaires indicate that while some students clearly

flourish and grow as thinkers over the course of pursuing a degree in philosophy, others appear to need

more guidance. In the writing portfolios submitted by graduates in the class of 2007, the research papers

that they produced for the capstone course, Colloquium in Philosophy, were a mixed bag. A number of

these papers, as well as others we have received in recent semesters, do not demonstrate the level of

intelligibility, critical distance, sophistication, and judiciousness that we expect from a senior philosophy

major. We have identified several ways to assist students in making a final leap to careful research on

philosophical issues and the quality of critical thinking that attends it.

Our renovation of the colloquium, begun in the fall of 2007, changes the capstone experience from a ¼-

credit course that majors are required to take twice, to a ½-credit course that they take once, in the fall of

their senior year. The amount of work that we expect them to undertake in this course alone justifies this

increase. Another change is that we now encourage students to meet with the advisor of their research

project earlier in the semester and more often than students have been in the habit of doing in the past.

Moreover, the three of us now begin the semester by devoting several class periods to discussing key

aspects of the research process with and modeling it for our students. For example, we will bring them to

the library for a preliminary research session using electronic resources like the Philosopher’s Index, we

will devote one session to turning a philosophical concern into a researchable question or thesis, and we

will demonstrate how to recognize and incorporate good sources into a research paper. For future

semesters, we plan to develop sessions on documenting sources, using footnotes, constructing a

bibliography, and other important issues, which we can include interchangeably based on student need as

we perceive it. As the renovation proceeds, we will solicit student input in order to maintain a “feedback

loop” for assessing the value of the changes made and the need for attention to other areas in the future.

We will also consider whether and how to integrate some of the material we prepare for colloquium into

lower-level courses when it would be appropriate. Academic year 2007-8 was the first to employ these

changes, and we were pleased to find that the papers as a group were generally stronger than those

submitted in recent years.

4. Strengthening our community

One comment frequently made in earlier incarnations of our graduate questionnaire concerned a desire for a

stronger, more active philosophical community at Beloit College. There has been a long-standing ice

cream social organized each semester, but beyond that event and classes, opportunities for students to

engage with both faculty and each other were minimal. The benefits of a stronger community are many,

but at least one benefit connects closely with the spirit of this project. We have here largely focused on

direct and indirect measures of assessing our program, but more informal means of feedback can

sometimes be more salient and effect more change. Finding more opportunities in new settings to interact

with our students can, among other things, help us to get a better anecdotal sense of ways to improve their

learning experiences in our department. It also remains the case that much of our advising happens

informally with students, rather than in planned advising appointments, and so increasing the opportunity

for informal engagement with students should increase the opportunities for this informal advising.

To this end, we have recently put in place several activities all aimed at creating a friendlier and more

intellectually stimulating philosophical community at Beloit. Beyond our aforementioned ice cream

socials, we have begun organizing an annual cookout for majors aimed at celebrating our graduating majors

and minors. We also have begun, in the fall of 2007, a “Philosophy and Film” series, where one several

evenings throughout the semester we view a film together with our students and then discuss its

philosophical implications afterwards. And, in conjunction with the change in our colloquium described

above, we began in the spring of 2008 a philosophy reading group for interested faculty and students.

While there have been some stumbles with the reading group as we grapple with the best ways to organize

it, we plan on continuing it in the spring of 2009 and beyond.

While none of these activities will yield the sort of data that assessment typically calls for, the knowledge

we hope to gain about ourselves and our program through these opening up new lines of communication

with our students is consistent with the goals at the heart of the project at hand.

Appendix A: graduate questionnaire and resulting data

A.1: Questionnaire (current version, spring 2008)

PHILOSOPHY DEPARTMENT GRADUATE QUESTIONNAIRE

Congratulations on graduating! You made the wise choice of a philosophy major or minor, and now we

want to benefit from your wisdom. Please take the time to answer the following questions, so that we can

best evaluate our curriculum in light of your experiences with it. Please provide as much detail as possible,

and also be as frank as possible. It will help us to know both the things you found most effective in your

experience in the department, as well as the things you found least effective. When you’ve completed this

questionnaire, please return it to Matt Tedesco at tedescom@beloit.edu if you fill it out electronically, or

else put in his office door at MI 210 if you print it and fill it out by hand. Thank you!

Name:

Year of graduation:

1) What classes did you take, and with whom, in the philosophy department?

2) What were the most effective, rewarding, or otherwise beneficial aspects of your experiences in the

department?

3) What were the least effective, rewarding, or otherwise beneficial aspects of your experiences in the

department?

4) What do you think you’ve gained from your experiences in the department?

5) If you were given free rein to redesign the department (including classes offered, requirements for the

major or minor, etc.), what changes would you make, and why?

6) If you were given free rein to redesign the department (including classes offered, requirements for the

major or minor, etc.), what would you be sure to retain, and why?

7) All things considered, given present facts about the department, would you recommend that someone

interested in majoring or minoring in philosophy at Beloit College do so? Why or why not?

Please circle the number that most accurately reflects your answer to these questions, or, if you’re filling it

out electronically, place an “x” next to the appropriate number:

Strongly Strongly

agree disagree

8) The courses I took in philosophy

promoted my critical thinking abilities 1 2 3 4 5

9) Studying philosophy has increased my ability

to analyze and evaluate arguments 1 2 3 4 5

10) Studying philosophy has contributed to my ability

to find and challenge assumptions 1 2 3 4 5

11) In my philosophy courses, I was taught

to think through opposing viewpoints 1 2 3 4 5

12) Studying philosophy has made me better

at building arguments to support my claims 1 2 3 4 5

13) The ethics course(s) I took helped to enhance

my capacity for social responsibility 1 2 3 4 5

14) My philosophy advisor encouraged me to take

courses outside the philosophy department

that focus on quantitative reasoning 1 2 3 4 5

15) Advising sessions with philosophy

professors helped me to make good choices

about courses to take in other fields 1 2 3 4 5

16) The philosophy curriculum successfully

advances the general education goals of

Beloit College (as I understand them) 1 2 3 4 5

17) My overall educational experience at Beloit

benefited from my study of philosophy 1 2 3 4 5

18) Any additional comments? Provide them here:

A.2: Questionnaire data: results

Before considering the data gathered so far from our questionnaires, a few clarifications need to be made.

First, because we have modified the survey further each year that we have used it, some of the objective

questions have two years of data, while others come from just a single year of feedback. Second, where

relevant, we have summarized some of the findings from other comments provided by our students in their

questionnaires. Third, the question numbers below correspond to the numbers as they appear on the above

version of the survey. And fourth, because of the scoring system used on the questionnaire, the lower the

score, the greater the disagreement, where perfect strong agreement would be a 1.0 and perfect strong

disagreement would be a 5.0.

Question #8: 1.05 mean (19 respondents)

2007: 8 (1), 1 (2)

2008: 10 (1)

Question #9: 1.00 mean (10 respondents)

2008: 10 (1)

Question #10: 1.11 mean (10 respondents)

2008: 9 (1), 1 (2)

Question #11: 1.38 mean (9 respondents)

2008: 8 (1), 1 (3), 1 (unanswered)

Question #12: 1.00 mean (10 respondents)

2008: 10 (1)

Question #13: 1.67 mean (18 respondents)

2007: 6 (1), 1 (2), 1 (3), 1 (unanswered)

2008: 5 (1), 1 (2), 4 (3)

Question #14: 2.81 mean (16 respondents)

2007: 2 (1), 4 (2), 1 (3), 1 (4), 1 (5)

2008: 2 (2), 2 (3), 2 (4), 1 (5), 3 (uanswered)

Question #15: 2.17 mean (18 respondents)

2007: 2 (1), 5 (2), 1 (4), 1 (5)

2008: 4 (1), 3 (2), 1 (3), 1 (5), 1 (uanswered)

Question #16: 1.21 mean (19 respondents)

2007: 7 (1), 1 (2), 1 (3)

2008: 9 (1), 1 (2)

Question #17: 1.11 mean (19 respondents)

2007: 8 (1), 1 (2)

2008: 9 (1), 1 (2)

A.3: Questionnaire data: preliminary findings

The findings, though preliminary given the relatively small sample size (particularly for questions only

answered by 2008 graduates), are revealing in a number of ways—regarding us as faculty, regarding the

students, and regarding the survey itself.

Regarding the survey specifically, it is an apparent weakness of it that no “not-applicable” option is

explicitly available for the “circle-the-number” questions. In some cases, students took it upon themselves

to avail themselves of such an option regardless; in other cases, however, the results may be skewed by

students signaling disagreement where applicability is dubious. For instance, with graduating minors in

particular, it is not clear that some or all of the advising questions are appropriate, yet their responses are

contained in the data here alongside the responses of majors. This is further complicated by the fact that

some minors had once planned or declared majors (and been advised accordingly), while others were

minors all along; the responses of the former, with respect to the advising questions, may be more relevant

than those of the latter.

Other students clearly struggled with how to respond to the questions about advising. It is not clear

whether this is simply a weakness of the survey tool, or rather raises much deeper questions about the

nature and purpose of advising. Some of the student comments in this respect are revealing. One 2007

graduate observed: “The previous questions regarding advising don’t quite match up with how I

approached advising. I always had my classes all figured out before I talked to Matt, so those questions are

fairly irrelevant. However, Matt provided EXCELLENT advising in areas not related to classes. He

advised on such topics as the future, how to get into grad schools, how to pick grad schools, making sure I

was finishing my requirements, how to approach teaching philosophy, etc. So while I did not find much

relevance for advising in class choices, Matt was a great advisor. That doesn’t seem to be reflected in the

questions, so I wanted to make it explicit.”

Several of the themes from this long comment are reflected in other responses from different students, and

with different departmental advisors. Another 2007 graduate commented, with respect to advising

connected with long-range life-planning, “I would have liked some kind of preparation for the future or

possible things one can do with a philosophy major—in general, more advising/life planning.” A 2008

graduate emphasized the fundamental importance of self-direction in planning a college career, observing,

“I always felt that when I had a question or needed support, it was available.” Additionally, several of the

2008 graduates felt that the questions about advising really weren’t relevant. One, who had a second major

and came to the philosophy major afterwards, noted, “I needed more advice about in-department course-

taking than out.” Another, who chose to fill in a “not-applicable” answer to question #14 regarding

quantitative reasoning, quickly explained, “I was doing that anyway.”

There is undoubtedly much more to consider with respect to the aims of advising, and what it means to be

an advisor at a liberal-arts college like Beloit. Yet, in the end, we suspected all along that our attention to

advising with respect to courses outside the department had generally been lacking, and these preliminary

results support that suspicion.

Appendix B: transcript data

Philosophy majors, 2000-2008

Total number of students Percentage of majors

Took no courses in an area related to quantitative literacy 14 22.22%

Took one or more courses in an area related to quantitative literacy 49 77.78%

Took two or more courses in an area related to quantitative literacy 28 44.44%

Took three or more courses in an area related to quantitative literacy 15 23.81%

Transferred one or more credits in an area related to quantitative literacy 19 30.16%

Took no courses in an area related to civic engagement 0 0.00%

Took one or more courses in an area related to civic engagement 63 100.00%

Took two or more courses in an area related to civic engagement 61 96.83%

Took three or more courses in an area related to civic engagement 53 84.13%

Participated in a field term or internship 14 22.22%

Participated in a study abroad or domestic off-campus study program 19 30.16%

Took no courses in a foreign language 33 52.38%

Took one or more courses in a foreign language 30 47.62%

Took two or more courses in a foreign language 23 36.51%

Took three or more courses in a foreign language 15 23.81%

Took no courses in an area related to creative expression 23 36.51%

Took one or more courses in an area related to creative expression 40 63.49%

Took two or more courses in an area related to creative 27 42.86%

expression

Took three or more courses in an area related to creative expression 18 28.57%

[INSERT EXCEL APPENDIX, PAGE 1]

[INSERT EXCEL APPENDIX B, PAGE 2]

[INSERT EXCEL APPENDIX B, PAGE 3]

[INSERT EXCEL APPENDIX B, PAGE 4]

[INSERT EXCEL APPENDIX B, PAGE 5]

[INSERT EXCEL APPENDIX B, PAGE 6]

[INSERT EXCEL APPENDIX B, PAGE 7]

Appendix C: sample portfolios

[FORTHCOMING]