IS Course Success in Liberal Arts Institutions – What’s...

IS Course Success in Liberal Arts Institutions – What’s the Formula?

Suvankar Ghosh Bijayananda Naik

Department of Decision Sciences, Information Systems, and Economics University of South Dakota

Vermillion, SD 57069 [email protected], [email protected]

Xiaolin Li Department of E-Business and Technology Management

College of Business and Economics Towson University Towson, MD 21252

[email protected]

ABSTRACT

Much of IS pedagogy research has focused on IS programs in business schools or in computer science departments. Insufficient attention has been given to assessing IS pedagogy in business schools without an IS major and in a strong liberal arts environment where skepticism about IS education is high. We describe a newly-designed IS core course that succeeded in such an environment. Our formula for success comprises of inculcating the notion that IS knowledge has both a business and a technology dimension to it, treating these two dimensions as co-equals, and closely integrating the two dimensions while continuing to deliver technical education using the time-tested active learning approach. The active learning component of this course included working on a set of three software development projects focused on an e-business theme. Furthermore, given our perspective that IS knowledge consists of a business and a technology dimension, we also developed an entirely new approach to measuring learning outcomes. Student outcomes were measured in terms of movements in the two-dimensional IS knowledge space and detected via multivariate analysis of variance. Data on student outcomes were collected from three classes held in consecutive semesters. This study breaks new ground both in terms of how IS learning is conceptualized and measured and in demonstrating the success of a technology-driven pedagogical approach in an essentially non-technical culture.

Keywords: Pedagogy, Instructional pedagogy, Computer literacy, Active learning, Learning goals and outcomes

1. INTRODUCTION

This study focuses on teaching the information systems (IS) core course in a business school at a midsized university that does not offer an IS major and where there is much skepticism among the business majors as to the relevance of IS education. The issue of the relevance of IS education particularly to students not pursuing IS careers has been noted by various educators (Baugh, 2011; Hoffman and Blake, 2003; Law, 2003). Baugh (2011) puts the focus squarely on this issue in posing the question: “How do you teach it (IS) and keep the interest of your students?”

Law (2003) talks about the challenges of designing IS courses for students who have no intention of pursuing

vigorous IS training. The students in the business school of this institution indeed have no intention of pursuing a career in information systems. The university does have a computer science department as part of the college of arts and sciences, but that is completely separate from the business school. The target audience for the IS core course is business students in their senior or junior year, with seniors comprising about half the class, pursuing diverse majors ranging from marketing to finance to human resource management. It is this set of students who question the relevance of technical IS education to their careers, given that the prospect of starting their careers is indeed imminent for the seniors.

The particular characteristics of the institutional environment are not the target of this research and are

Journal of Information Systems Education, Vol. 25(3) Fall 2014

189

assumed as given. Rather, this study focuses on the efficacy of an IS core course that we designed and delivered, which employed a rather technical active learning approach given the non-technical culture of the institution, and on our novel approach to conceptualizing and measuring IS literacy. The motivation for the selected emphases of this research is not difficult to understand. In spite of the questions raised about the relevance of technical IS education to general business majors, it still remains incumbent upon business schools to produce students that are sufficiently IS-literate and can face the challenges of a complex, technological world. This view of the necessity of making all business students IS-literate regardless of their particular career inclinations is echoed by Tsai (2002) and the National Association of Colleges and Employers (2011). However, how to go about achieving this important objective in non-technical or less-technical institutional environments does not seem to have been addressed adequately in the published literature. We believe that this article provides valuable insight in this matter and would stimulate further research.

In this article, we begin by presenting a review of the relevant literature in Section 2 where Section 2.1 discusses the notion of IS literacy and Section 2.2 reviews active learning in IS. With the discussion in Section 2.1 as a backdrop, we develop a novel holistic conceptualization of IS literacy and present it in Section 3. An innovative design of the IS core course for non-IS business majors that implements our active learning approach, which we refer to as the disaggregated mode of technology development, is presented in Section 4. A set of hypotheses to assess the effectiveness of the newly designed IS core course is developed in Section 5. The research methodology and the results on learning outcomes and the hypotheses tested are described in Sections 6 and 7, respectively. Finally, Section 8 presents the conclusions of this study.

2. LITERATURE REVIEW

2.1 IS Literacy IS literacy is not about knowing how to use Microsoft Excel or PowerPoint. Instead it is about gaining knowledge of a range of topics from aligning the technology with the business to telecommunication networks to computer security (Laudon and Laudon, 2012). These topics that every business student must know have been identified by Ives et al. (2012), a group of 40 distinguished IS faculty. Based on the list of topics identified by Ives et al. (2012) as well as a broader examination of standard undergraduate IS textbooks (Kroenke, 2011; Laudon and Laudon, 2012; Valacich and Schneider, 2010), we propose that the knowledge that an IS core course provides to business students can be characterized using two principal dimensions, viz., a business dimension and a technology dimension. The business dimension of IS knowledge includes topics such as IS strategy, business value, competitive advantage, business-technology alignment, globalization, ethics, and the impact of IS on the business processes, culture, values, and structure of the organization. The technology dimension includes topics such as hardware, operating systems, software applications, middleware, networks, communication protocols, the Internet, information technology (IT) architecture, database management systems, and security.

Our envisioning of IS knowledge in terms of having a business and a technology dimension is also consistent with the IS Model Curriculum that has been developed over the years from 1997 to 2010 (Davis et al., 1997; Topi et al., 2010). Although the model curriculum was formulated for students in IS degree programs, the topics and areas they defined can also fit into our business and technology dimensions of IS knowledge. Some topics in the 2010 model curriculum such as systems analysis and design could be viewed as having some portions, such as the definition of business requirements in the systems analysis phase, belong to the business dimension whereas other portions that pertain to the design and implementation of the system would belong to the technology dimension of IS knowledge. Furthermore, undergraduate IS textbooks such as Laudon and Laudon (2012) characterize this duality of the IS discipline by stating that it has a “behavioral approach” encompassing disciplines such as psychology, sociology, and economics, and also a “technical approach” that spans computer science, management science, and operations research.

Our conceptualization of IS knowledge as having a technology and a business dimension leads to a new and holistic way of measuring IS literacy and learning outcomes as discussed in Section 3. Given this epistemological view of IS knowledge as having a technology and a business dimension, the next question is the best way for delivering IS education.

2.2 Active Learning in IS Active learning is a time-tested approach in IS pedagogy. A review of the active learning literature in IS pedagogy reveals that there are essentially three different modes of active learning employed in IS courses: 1) using technology, 2) integrating technology, and 3) developing technology. Representative sets of active learning studies in these three modes are shown in Tables 1, 2, and 3. IS courses in the technology-use mode, as shown in Table 1, include those where students are taught how to use ERP systems such as SAP or OpenERP (Ayyagari, 2011; Davis and Comeau, 2004; Drajner and Schenk, 2004; Fedorowicz et al., 2004; Leger, 2006; Sager et al., 2006). The technology-use active learning approach actually requires the least amount of prior background and sophistication with technology. The technology is essentially simply presented to the students as a black box.

The next level up in terms of technical sophistication is technology integration-based active learning. Examples of this type of course are given in Table 2. This mode is often found in e-commerce courses where major projects typically involve setting up a substantial website, such as a website for a new business (Abrahams and Singh, 2010; Changchit et al., 2006; Neck and Stoddard, 2006). Hand-coding the website is not the path taken in such courses; instead various tools, such as Joomla for web content management, Google Checkout for the shopping cart application, and Google Analytics for web traffic measurement, are employed in setting up the e-commerce site.


190

Study Technology Used Davis and Comeau (2004) ERP system (SAP) Draijer and Schenk (2004) ERP system (SAP) Fedorowicz et al. (2004) ERP system (SAP) Leger (2006) ERP system (SAP) Sager et al. (2006) ERP system (SAP) Ayyagari (2011) ERP system (OpenERP)

Table 1: Technology Use Mode

Table 2: Technology Integration Mode

*course taught in the computer science department Table 3: Technology Development Mode

The final mode of technology development involves

building a complete and working software application or a web application such as described in Fox (2002), Janicki et al. (2007), Mitra and Bullinger (2007), and Scott (2004). Clearly, this mode of active learning requires the most sophistication in software technology and this type of course is generally offered for IS majors in a business school at an

advanced stage or by a computer science department for its students. Table 3 shows examples of courses using this mode. A closer look at the institutional context in which these three different types of courses are offered clearly shows that the more technical the active learning mode employed, the more technical is the institutional context and the background of the students the course is intended for.

In designing our IS core course, although offered in a non-technical environment, we took the bold step of choosing the technology development mode of active learning. We chose the technology development mode as opposed to something else, such as using an ERP system, because we believe that this mode stretches the students’ capabilities the farthest, imparts a greater amount of technical knowledge, and boosts their confidence in dealing with technical topics. The mere interaction with software systems, such as an ERP system, as a black box does not provide the user with an appreciation of the layers of technology underlying information systems and e-business processes. We also felt that working with the technology at a hands-on level, such as building a website using HTML and ASP.Net, would allow a better integration of the material on new technologies taught in the lectures such as XML, which VanLegen (2010) has argued should be part of the IS Model Curriculum, with something tangible that the students did such as the website they built. XML is presented in the lectures as a generalization of HTML. We describe the design of our course in Section 4, but next we discuss our novel conceptualization and measurement of IS literacy.

3. A HOLISTIC CONCEPTUALIZATION AND MEASUREMENT OF IS LITERACY

Based on our discussion of IS literacy in Section 2.1, we address here the broader question of what we view as IS learning. We have argued in Section 2.1 that IS knowledge should be viewed as having a business and a technology dimension. The IS knowledge of a business student can therefore be considered as his or her position in the two-dimensional knowledge space represented by a point as shown in Figure 1. We define IS learning as an upward and rightward shift in the student’s position in this two-dimensional IS knowledge space. The instrument for measuring students’ self-perceptions of their knowledge in the business and technology dimensions is given in the Appendix. It is important to note that since we are taking a holistic view of knowledge, the instrument is not deconstructed into specific items in the technology dimension such as knowledge about security, network protocols, or enterprise architecture. Similarly, the instrument is not deconstructed in the business dimension into specific items such as for strategy, competitive advantage, or organizational impact.

Study Integration Scenario Changchit et al. (2006)

E-commerce site for on-line business

Abrahams and Singh (2010)

Web-site for non-profit organization

Braender et al. (2009)

Creating a community using blogging

Williams and Chinn (2009)

Web 2.0 technologies for market promotion

Lenox (2008) Improving website of local service provider

Abrahams (2010) Web-site for online business Neck and Stoddard (2006)

Online business (Babson’s acclaimed FME course)

Study Technology Developed De Brock (2001) Small IS Fox (2002) Database application in Microsoft

Access/Visual Basic Scott (2004) ASP.Net web application with

VB.Net backend for industry sponsor

Janicki, Fischetti, and Burns (2007)

ASN.Net web application for real user in MIS capstone course

Lim (2002) Web development with Java programming

Tan and Jones* (2008)

IS for external client

Mitra and Bullinger (2007)*

IS for local health care provider

Chen (2006)* IS for college’s IT department


191

Our research model, which employs large and aggregated constructs such as knowledge in the business or the technology dimension and associated instruments, is essentially a parsimonious model. Parsimonious models such as the Technology Acceptance Model (TAM) (Davis, 1989; Venkatesh et al., 2003) are a staple of IS research. The TAM also employs aggregated constructs such as perceived usefulness without deconstructing usefulness into more specific constructs that contribute to usefulness such as performance, manageability, serviceability, scalability, and extensibility of the technology. On the other hand, there are also technology-task fit models that are more granular as they focus on how the tasks that need to be performed fit with the characteristics of the technology (Goodhue and Thompson, 1995). Clearly, both parsimonious and granular models have their own applications in research and they both provide useful insights, albeit at different levels.

An upward and rightward shift in the students’ position in the knowledge space as a consequence of taking the course can be detected using multivariate analysis of variance (MANOVA) with statistics such as Wilk’s lambda, Pillai, Hotelling-Lawley, and Roy (Huberty and Olejnik, 2006). Our particular conceptualization of IS knowledge is pragmatic in that both an instrument (Appendix) and a statistical technique, MANOVA, exist for measuring IS learning in this holistic two-dimensional view of IS knowledge.

Our approach stands in contrast to how IS learning has typically been measured. In the traditional granular approach, there are many learning outcomes specific to a course which are measured individually and separately. Daigle et al. (2007) define six types of learning outcomes including decision modeling, risk analysis, and leveraging technology in their accounting information systems course. Similarly, Surendran et al. (2005) define ten measures for assessing learning in their systems analysis and design course. Attaway et al. (2011) measure student performance in their IS course with regard to eight learning objectives that fall in the areas of software applications, technology infrastructure, and information systems strategy. As we see later in Section 5, our parsimonious research model involves

testing fewer hypotheses related to the main learning goals of the course. The design of our course is presented next.

4. IS CORE COURSE DESIGN

The main goal of this IS core course is to provide a broad understanding of the business and technology aspects of IS. Non-IS business majors do need to understand the business impact of IS and at least have a high-level and general understanding of various technical areas. Given these broad learning goals, the textbook for the course is Management Information Systems – Managing the Digital Firm by Laudon and Laudon (2012), which is notable for its emphasis of the business side of IS and is rich in real-world cases. As part of the assessments, students work in groups on three major case assignments that focus particularly on the business side of IS.

We established in Section 2.2 that technical IS education is best delivered in an active learning format, albeit with differences in the mode of delivery which can range from technology use to integration to development. We chose the technology development mode where the students worked on a set of software development projects. The key question is the choice of these software projects that would best integrate with the business side of IS knowledge. Clearly, e-commerce or e-business is a major and dominant topic in IS courses. From e-commerce applications for selling products and services to e-procurement applications for purchasing parts and supplies to Internet-based customer relationship management (e-CRM) and supply chain management (e-SCM), business truly is about e-business. Given this large focus on e-business and all its ramifications in this course, we felt that giving students an understanding of how an e-business application actually works from a technology perspective would be a good accompaniment to the knowledge they were gaining about the business importance of these applications. Consequently, we put together a set of projects labeled the “Anatomy of an e-Business Application.”

We recognized that building a complete and working web application was beyond the capabilities of students with no previous programming experience, which was the case for

Figure 1: Shift in Students’ Position in Business-Technology Knowledge Space

Shift in average class position

Technology Dimension Knowledge

Business Dimension Knowledge

Student’s position in knowledge space before course

Student’s position in knowledge space after course


192

the majority of the class. Hence, we did not require the students to build a complete and working application, but instead to work on a set of three discrete software development projects which explored the three layers of a web application – the presentation, business logic, and the data layer. The three projects which the students did in groups of three to four students, with the groups being the same as in the case assignments, are described next. Microsoft Visual Studio was the software development environment used in this course.

Presentation: In this project, the students had to build a website using HTML and ASP.Net elements and they were free to choose the topic of the website. However, given the overall imperative of integrating the technical with the business, students were encouraged to develop a website on some idea for a potential business or to show in a compelling fashion some product or service that they were interested in. The instructor demonstrated in class how to use Visual Studio to build websites. In their project, the students were required to use ASP.Net elements such as <asp:Table>, <asp:TableRow>, <asp:TableCell>, and <asp:Image> for sectioning the web page and appropriately positioning the text, images, and video. The <iframe> element was used for including video in the website. They also had to provide links to other related websites using <asp:HyperLink>. Their website had to consist of a minimum of three linked web pages.

Business Logic: In this project, the students learned about the basic elements of computer programming such as arithmetic statements, loops, and conditional statements. The programming language used was C#. The instructor introduced the concept of an algorithm by flow-charting a simple algorithm that finds the sum of a series of numbers as in: sum = 1 + 2 + … + N. The instructor then showed how to translate this simple algorithm into a C# program that utilized the basic assignment, looping, and conditional statements of C#, along with inputting the value of N and validating the input. The notion of the type of variables and their declaration was also discussed. The project that the students had to do was to modify the sum program demonstrated in class into a product program for finding the product of the first N integers. This basic ability to program in C# was then used to add a simple animation to the website the students had built in the first project. The animation involved employing a C# script along with the <asp:Timer> and <asp:ScriptManager> elements. The animation that the students had to add was to take a web page with four images in it and make those images rotate in a clockwise fashion.

Data: We debated whether to teach ADO.Net which would have allowed a database to be connected to the website built in the first project and then enhanced in the second project. However, it was felt that ADO.Net was at a level of complexity that was simply beyond the reach of business students, the majority of whom had no programming background whatsoever. Consequently, we set learning SQL as the main goal of the last project. The goal of the data layer project was to learn how to query relational databases to extract pertinent information using SQL. For this project, SQL Server Express was used as the database, which is bundled in with Visual Studio.

The instructor demonstrated how to create a database with some tables and then to use SQL to extract information

from these tables. The SELECT statement along with various clauses such as WHERE for filtering records, GROUP BY for aggregating records, as well as functions such as COUNT, MAX, and AVG were shown. The notion of the INNER JOIN of two tables and the capability of storing a SQL query as a VIEW in the database were also taught. The students worked on a project where they created a database for a course registration system which contains tables describing courses, courses that students have taken, and information about the students. They had to define SQL statements for various types of queries launched against the database and store these queries as views to demonstrate to the instructor.

As we did not require the students to build a full working e-commerce application, we refer to our approach as the “disaggregated mode of technology development” in contrast to the traditional technology development mode that involves building a complete software application reported in Fox (2002), Janicki et al. (2007), Mitra and Bullinger (2007), and Scott (2004). In order to assess the effectiveness of the course, several hypotheses on learning outcomes were developed based on our holistic conceptualization of IS-literacy. The development of these hypotheses is presented next.

5. HYPOTHESIS DEVELOPMENT

Following Section 3 where IS literacy consists of knowledge in both the business and the technology dimension, Hypothesis 1 aims at assessing the change in IS knowledge that results from the course when both the business and technology dimensions of IS knowledge are considered together. Hypotheses 2 and 3 look at each of the two dimensions separately.

Hypothesis 1: There is a change in the students’ IS knowledge after taking the course when both the business and the technology dimension of IS knowledge are considered together. Hypothesis 2: There is an increase in the students’ knowledge in the business dimension of IS knowledge after taking the course. Hypothesis 3: There is an increase in the students’ knowledge in the technology dimension of IS knowledge after taking the course. Hypotheses 1, 2, and 3 are simply hypotheses of student

performance with respect to the learning goals of the course. Measuring student performance, either objectively or via student perceptions, relative to the learning goals of the course is a standard practice in IS pedagogy research (Surendran et al., 2005; Daigle et al., 2007; Attaway et al., 2011). The only difference is that in these other studies the learning goals have been defined more specifically whereas in this study we have defined the learning goals holistically. Hence, we have fewer such hypotheses as compared to, for example, Daigle et al. (2007). For their accounting information systems course, they tested six hypotheses with respect to their six learning goals of decision modeling, risk analysis, measurement, reporting, research, and leveraging technology.


193

While our holistic Hypotheses 1, 2, and 3 may on the surface appear to an IS teacher as though they should be true, it must be borne in mind that much of IS pedagogy research has occurred in more technical institutional contexts (De Brock, 2001; Fox, 2002; Janicki et al., 2007; Mitra and Bullinger, 2007; Scott, 2004; Tan and Jones, 2008). This is perhaps the only empirical study, to our knowledge, on IS pedagogy conducted in a non-technical liberal arts environment with high student skepticism towards IS education and its relevance to the students’ chosen careers. Consequently, these hypotheses cannot simply be taken for granted and must be subjected to empirical assessment.

The role of prior knowledge has long been recognized as important in learning in general (Ausubel, 1968; Novak, 2010) and in IS learning in particular (Drake, 2012). A key precept of the Assimilation Learning Theory (Ausubel, 1968; Novak, 2010) is that when new concepts build upon and are integrated with prior knowledge, the learning is more meaningful and effective. Consequently, students coming into this course with some prior background in computers, such as computer courses taken from any other department or institution including high school, should be able to better master the content of this course. Hence, we posited:

Hypothesis 4: There is a difference in the students’ IS knowledge after taking the course, when both business and technology dimensions are considered together, between those students with some prior IS background and the students without this background.

Furthermore, we felt that the difference in IS knowledge at the end of the course between the students with some prior background in computers and those without would stem mainly from the ending position in the technology dimension being higher for the students with some background in computers and information systems. Prior courses in computers, software applications, or programming feed directly into the technology dimension of IS knowledge. Hence, the hypothesis:

Hypothesis 5: The students’ knowledge in the technology dimension is higher after taking the course for those students with some prior IS background relative to students without this background.

Admittedly, the notion that the students’ with prior knowledge in IS end up at a better position could stem in part from their starting out at a higher level of technical knowledge. However, if Hypotheses 1, 2, and 3 test strongly positive indicating that the course significantly benefitted everyone, then the difference between the ending positions of students with prior IS background and those without cannot be entirely due to the difference in the starting positions. Assimilation Learning Theory (Ausubel, 1968; Novak, 2010) would argue that the students with prior background got more out of the course as their background helped them to assimilate the material better.

We debated whether students with some prior technical background, as evidenced by having taken some computer or IS courses previously, would necessarily be able to master the business ramifications of information systems and technology better than students without this background. It is

true that we have taken an integrated approach in treating the business and the technology dimension of knowledge in this course, and this integration would argue that a student who understands the technology would perhaps also understand the business implications of IT better. However, the business implications of IT also draw upon non-technical knowledge that a non-technical student may understand better. For example, a marketing major may better understand the general theories of competitive advantage, such as the Resource-Based View (Barney, 2001) or Porter’s Five Forces Model of competitive advantage (Porter, 1980). Hence, they may actually be able to grasp the competitive advantage of IT better than a student with a better technical background in computers and information systems. Pitting the integrated nature of the course against the off-setting influences of non-technical business knowledge that a non-technical student may have a better handle on, we posited:

Hypothesis 6: There is no difference in the students’ knowledge in the business dimension after taking the course between the students’ with some prior IS background and the students without this background.

Course success rates were assessed in post-hoc analysis. Success rates were based on the percentage of students who succeeded in this course where a successful student outcome is defined at two levels: 1) the student improved knowledge in both dimensions of IS literacy, and 2) the student improved knowledge in at least one of the two dimensions. A classification of students with regard to whether they succeeded was built using the survey data. An increase in the Likert-scale score on a given dimension after taking the course over the value prior to taking the course was deemed a successful outcome in that particular dimension. The classification data were then used to find the percentages of students who succeeded for the two levels of success.

6. RESEARCH METHODOLOGY

After the students completed this course, a survey (see Appendix) was administered to them at the end of the semester to assess learning outcomes in the technology and business dimensions of IS literacy. Students were asked to rate their knowledge and understanding of information systems and technology from a business and also a technology perspective both before and after taking the course using a 5-point Likert-scale as shown in the Appendix. The survey was administered in three offerings of the course held in consecutive semesters.

The survey data obtained from the three semesters were consolidated and then analyzed to assess learning outcomes from this newly designed course. Between the three semesters that the survey was administered, 138 surveys were returned out of a combined enrollment of 222 students in the three semesters corresponding to a 62% response rate. A 62% response rate is sufficiently high to mitigate concerns of non-response bias. The composition of the class in Table 4 shows that close-to-graduating seniors constitute about half the class.


194

UG Standing Percentage Senior 45% Junior 47% Sophomore 8% Freshman 0% Total 100%

Table 4: Class Composition

We followed a retrospective pretest-posttest design for this study and administered the survey at the end of the semester. We asked students to reflect back to their level of understanding and knowledge about the technology and business aspects of information systems and technology at the beginning of the semester as they filled in the survey. The retrospective method is to be contrasted with the traditional pretest-posttest approach where the survey is administered both at the beginning of the semester to collect the pretest data and also at the end of the semester to collect posttest data. The two main reasons for using a retrospective rather than a traditional pretest-posttest design in this study were to minimize response-shift bias and subject resentment.

Response-shift bias arises from the fact that subjects do not have sufficient knowledge at the beginning of an intervention to correctly evaluate the variables being measured in the study, such as their knowledge and understanding about a given area or discipline (Aiken and West, 1990; Campbell and Stanley, 1963; Howard et al., 1979). It is only after the intervention, or taking the course, would they be able to correctly assess how little or how much they knew about that area at the beginning of the course. Since the subjects in effect “don’t know what they don’t know” at the beginning of the course, what they report about their knowledge at the beginning can be biased. This is the response-shift bias and it poses a major threat to the internal validity of the study. In this case, response-shift bias was deemed to be a major problem since business students at a liberal arts institution without an IS major, and where this is the only IS course that students take, clearly could not know enough about the field at the beginning to provide a reliable indication of their knowledge about IS at that time. Often, subjects may have an inflated view of their knowledge of some field when they really don’t know the field. Hence, a retrospective pretest-posttest design is typically used in these situations and this has long been viewed as an effective approach to addressing this problem of the response-shift bias (Howard, 1980; Howard et al., 1979; Lam and Bengo, 2003; Pratt et al., 2000).

Another benefit of the retrospective pretest-posttest method is that it is generally viewed in the literature as less intrusive compared to the traditional pretest-posttest approach (Howard, 1980; Howard et al., 1979; Lam and Bengo, 2003; Pratt et al., 2000). A less intrusive research design is less likely to arouse subject resentment. Subject resentment can also pose a threat to the internal validity of a study as has been seen in previous experimental research where, when different groups are given different treatments, the group that perceives having received an inferior treatment resents the whole research. Subjects harboring resentment may not provide accurate answers on surveys (Onghena, 2005; Sanchez and Medkik, 2004). In this study, if the pretest was administered at the beginning of the course in the traditional pretest-posttest format, a possible source of

resentment could be that students might view the course as simply a professorial research experiment. Given existing student skepticism towards a course that might not be relevant to their careers, we wanted to be very cautious about introducing any other factor that could increase student discomfort with the course.

7. RESULTS

7.1 Analysis of Variance Table 5 gives the descriptive statistics for the Likert-scale before-the-course and after-the-course knowledge scores in the two dimensions of business and technology. As seen from Table 5, the class average scores after the course were larger than those before, thereby providing a prima-facie case for improvement in the students’ knowledge in both the business and the technology dimension. This was then examined via the MANOVA and the univariate ANOVA tests. Table 6 reports values of the Pillai, Hotelling-Lawley, Wilks, and Roy statistics along with their p-values for the MANOVA test. The extremely significant p-value for each of the four test statistics supports Hypothesis 1, therefore indicating that the course indeed shifted the class average position in the business-technology knowledge space.



Before After Before After Class Average

2.3188 3.5326 2.1739 3.2754

Table 5: Class Averages for Before- and After-Course Knowledge

Table 6: MANOVA Knowledge Enhancement Results

Next, ANOVA tests were performed to determine if the knowledge enhancement was significant in each dimension. The ANOVA test results are shown in Table 7. As seen from Table 7, the F-statistic value for the business dimension of IS knowledge is extremely significant. This test result supports Hypothesis 2. Similarly, Table 7 shows that the F-statistic for the technology dimension of IS knowledge is also extremely significant. This test result supports Hypothesis 3. Thus, we can conclude that the univariate ANOVA test results indicate that there was an increase in the class averages of IS knowledge in both the business and the technology dimension after the course was taken by the students. The fact that Hypotheses 1, 2, and 3 tested positive indicates that the newly designed IS core course was indeed successful in enhancing IS literacy.

Business Dimension Technology Dimension F statistic p-value F statistic p-value 130.7 2x10-16 *** 112.8 2x10-16 *** Table 7: ANOVA Knowledge Enhancement Results

Statistic Value p-value Pillai 0.3479 2.2x10-16 *** Hotelling-Lawley 0.5334 2.2x10-16 *** Roy 0.5334 2.2x10-16 *** Wilks 0.6521 2.2x10-16 ***


195

7.2 Role of Prior Knowledge The group averages for after-the-course knowledge in the business and the technology dimension for the two groups, one with some prior IS background and the other without this background, are given in Table 8. The one-factor MANOVA test results for the role of prior knowledge in determining the after-the-course positions of the students in the IS knowledge space, where both the business and the technology dimension are taken into account, are given in Table 9. The MANOVA results are significant thereby supporting Hypothesis 4 that prior IS background indeed determines how much IS knowledge students end the course with.



With No Prior

With Prior

With No Prior

With Prior

Group Average

3.4438 3.6939 3.1236 3.551

Table 8: Statistics for After-Course Knowledge Based on IS Background

Table 9: MANOVA Test Results for Role of Prior Background

We had speculated in Hypotheses 5 and 6 that the differential gain in IS knowledge that students with some prior IS background enjoy from this course relative to those without the background originates from the technology and not the business dimension of IS knowledge. The significant ANOVA test in Table 10 in the technology dimension of knowledge indeed bears out Hypothesis 5. Regarding the business dimension, we felt there were certain offsetting influences where, although better technical grounding can help a student understand the business side better because of the integrated nature of the course, this could be offset by a non-technical student having a better grasp of the business issues. Table 10 shows that the ANOVA test for differential gain of knowledge in the business dimension for students with prior IS background has a p-value of 0.0882. This is not significant at the 0.05 level but is significant at a lower 0.1 level. Hence, while it would appear that prior IS background may help students acquire more knowledge about the business side of IS, one cannot make such an assertion with a great deal of confidence.

Business Dimension Technology Dimension F statistic p-value F statistic p-value 2.949 0.0882 8.947 0.0033* Table 10: ANOVA Test Results for Role of Prior

Background

7.3 Post Hoc Analysis of Course Success Rates Table 11 provides the sample percentages of course success rates. We computed two course success rates, one for the percentage of students who increased their knowledge in both the business and the technology dimension of IS knowledge, and another for the percentage of students who increased their knowledge in at least one dimension. We found that 88% of the students in the sample reported increasing their knowledge in at least one of the two dimensions while 71% reported having increased their knowledge in both dimensions. This also means that, had we defined two a priori hypotheses stated in null form as H0: θ1 ≤ 82% and H0: θ2 ≤ 64% where θ1 is the population percentage of the students that increased their knowledge in at least one dimension and θ2 the corresponding population percentage in both dimensions, these hypotheses would have been rejected at a 0.05 level of significance. The imputed thresholds of 82% and 64% in the above hypotheses are the maximum that they can be for rejection of the null at a 0.05 level of significance. This provides some insight, albeit within the framework of post hoc analysis, into the population percentages of the students that succeeded in this course. Clearly, course success rates of 64% and 82% in terms of population percentages, depending on whether enhancement in both dimensions of knowledge or at least one dimension is the criterion, can be viewed as meritorious outcomes as was done in this institution.

At Least One Dimension

Both Dimensions

Sample Percentage 88% 71% Imputed Maximum for Population Percentage 82% 64% p-value 0.0412* 0.043* Table 11: Post Hoc Analysis of Course Success Rates

8. SUMMARY AND CONCLUSIONS

Most IS pedagogy studies have focused on courses offered by IS programs in business schools or computer science programs (De Brock, 2001; Fox, 2002; Janicki et al., 2007; Mitra and Bullinger, 2007; Scott, 2004; Tan and Jones, 2008). This is one of the very few studies that focuses on IS learning in a liberal arts environment where there is built-in resistance to IS learning. The business school in question does not have an IS major and has historically not had a technical culture. Business majors ranging from finance to marketing to human resources who take this course do not find the course to be all that relevant to their careers, particularly those who are seniors and are imminently embarking upon non-technical careers. This cultural resistance was a well-known characteristic of the environment and hence was treated as a given in this study.

This article essentially evolves a formula for the success of IS courses in such less-technical or non-technical liberal arts environments. The success formula combines technical active learning with an enhanced focus on the business. IS knowledge is essentially viewed as comprising of two co-equal dimensions, business and technology, and the course was informed and structured from that perspective.

Statistic Value p-value Pillai 0.0636 0.01183* Hotelling-Lawley 0.06795 0.01183* Roy 0.06795 0.01183* Wilks 0.93637 0.01183*


196

Integration of the knowledge presented in the technology dimension with that in the business dimension was a key characteristic of the course. As the course spent a good deal of time on e-business given that the contemporary business runs on the Internet, the hands-on laboratory component of the course was structured around three software development projects that explored the three layers of an e-business application.

As we wanted to challenge the students’ abilities to absorb technical knowledge and skills to the maximum extent, we followed the technology development mode of active learning even though the students by and large had non-technical backgrounds. However, given the adage that a course must be “hard but not too hard” (Martin et al., 2008), we chose not to make the students build a complete and working e-business application. We felt that this goal would perhaps be an unreachable stretch for non-technical business students with no prior programming experience. Consequently, the laboratory-based active learning component engaged what we refer to as the “disaggregated mode of technology development”. In this mode, the three projects that the students did were loosely coupled and provided insight into the internal workings of an e-business application without requiring the building of a full working application.

The technical knowledge gained from the labs also helped students better understand related technical content such as XML, SOA and Web Services taught in the lectures. XML was presented as a markup language that is a generalization of HTML where the presentation tags in the document are replaced by more general tags used for documents such as purchase orders and invoices exchanged between the business entities in a supply chain. Hence, dry and esoteric topics such as XML were ultimately related to something personal, or the websites built by the students as part of the labs. Learning the alphabet soup of technical terms such as XML and SOA is something that many non-IS majors are skeptical of since they don’t see its relevance to their future careers. However, the conversation in the business world is increasingly laced with technical jargon as business processes inexorably move towards e-business and the Internet. Operations managers, marketing managers, and finance managers will have to come to terms with terminology and concepts such as XML and Web Services as customers, partners and financial institutions are linked with these technologies.

A major contribution of this research is the way we have conceptualized and measured IS learning. We take the view that IS knowledge is fundamentally comprised of a business and a technology dimension. Positive learning outcomes are upward and rightward movements in this two-dimensional IS knowledge space. Multivariate analysis of variance (MANOVA) can then be used to detect the movement of students in this knowledge space. This is a novel way of conceiving of IS knowledge and this study is the first application of MANOVA techniques in IS pedagogy research. Hypotheses 1, 2, and 3 tested strongly significant for our course indicating that the course indeed enhanced students’ knowledge in both the technical and the business dimension of IS knowledge. Furthermore, post hoc analysis of course success rates showed that the maximum imputed population percentage for learning success in at one least

dimension was 82%, and for both dimensions it was 64%. These were viewed as strong success rates thereby bolstering the overall conclusion of course success.

Some prior background in computers and IS did help students to get more out of this course, but it must be recognized that this course unequivocally helped all students. The knowledge enhancement hypotheses tested extremely significant for the student body as a whole. Students with prior IS background were simply able to end the course in a more favorable position compared to students without this background with regard to the technology dimension of IS knowledge.

In closing, we note that this IS core course has had the positive impact of moving the institution to a more technical culture. Other liberal arts institutions wishing to initiate this type of change can benefit from the approach taken in this course. Furthermore, the conceptualization of IS knowledge as comprising of co-equal business and technology dimensions along with the measurement techniques described herein can be replicated in other IS courses.

9. REFERENCES

Abrahams, A. S. (2010). Creating E-Commerce Startups with Information Systems Students: Lessons Learned from New Venture Successes and Failures. Information Systems Education Journal, 8(35), 3-24.

Abrahams, A. S. & Singh, T. (2010). An Active, Reflective Learning Cycle for E-Commerce Classes: Learning about E-Commerce by Doing and Teaching. Journal of Information Systems Education, 21(4), 383-390.

Aiken, L. S. & West, S. G. (1990). Invalidity of True Experiments: Self-Report Pretest Biases. Evaluation Review, 14(4), 374-390.

Attaway, A. N., Chandra, S., Dos Santos, B. L., Thatcher, M. E., & Wright, A. L. (2011). An Approach to Meeting AACSB Assurance of Learning Standards in an IS Core Course. Journal of Information Systems Education, 22(4), 355-366.

Ausubel, D. P. (1968). Educational Psychology: A Cognitive View. New York, NY: Holt, Rinehart, and Winston.

Ayyagari, R. (2011). Hands-On ERP Learning: Using OpenERP, an Alternative to SAP. Journal of Information Systems Education, 22(2), 123-134.

Barney, J. B. (2001). Is the Resource-Based “View” a Useful Perspective for Strategic Management Research? Yes. Academy of Management Review, 26(1), 41-56.

Baugh, J. M. (2011). Make it Relevant and They Just May Learn it. Information Systems Education Journal, 9(7), 14-20.

Braender, L. M., Kapp, C. M., & Yeras, J. (2009). Using Web Technology to Teach Students about their Digital World. Journal of Information Systems Education, 20(2), 145-154.

Campbell, D. T. & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research on Teaching. Chicago, IL: RandMcNally.

Changchit, C., Cutshall, R., & Gonsalves, G. C. (2006). Designing an Electronic Commerce Course: An Effort to Balance Theory and Practice. Information Systems Education Journal, 4(108), 1-7.


197

Chen, B. (2006). Teaching Systems Analysis and Design: Bringing the Real-World into the Class Room. Information Systems Education Journal, 4(84), 1-8.

Daigle, R. J., Hayes, D. C., & Hughes, K. E. (2007). Assessing Student Learning Outcomes in the Introductory Accounting Information Systems Course Using the AICPA’s Core Competency Framework. Journal of Information Systems, 21(1), 149-169.

Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340.

Davis, C. & Comeau, J. (2004). Enterprise Integration in Business Education: Design and Outcomes of a Capstone ERP-Based Undergraduate E-Business Management Course. Journal of Information Systems Education, 15(3), 287-299.

Davis, G. B., Gorgone, G. T., Couger, J. D., Feinstein, D. L., & Longenecker, H. L. (1997). IS’97 Model Curriculum and Guidelines for Undergraduate Degree Programs. New York, NY: ACM and Park Ridge, IL: AITP.

De Brock, E. O. (2001). Integrating Real Practical Experience in ICT Education. Journal of Information Systems Education, 12(3), 133-140.

Draijer, C. & Schenk, D.-J. (2004). Best Practices of Business Simulation with SAP R/3. Journal of Information Systems Education, 15(3), 261-265.

Drake, J. R. (2012). A Critical Analysis of Active Learning and an Alternative Pedagogical Framework for Introductory Information Systems Courses. Journal of Information Technology Education, 11, 39-52.

Fedorowicz, J., Gelinas, U. J., Jr., Usoff, C., & Hachey, G. (2004). Twelve Tips for Successfully Integrating Enterprise Systems across the Curriculum. Journal of Information Systems Education, 15(3), 235-244.

Fox, T. L. (2002). A Case Analysis of Real-World Systems Development Experiences of CIS Students. Journal of Information Systems Education, 13(4), 343-350.

Goodhue, D. L. & Thompson, R. L. (1995). Task-Technology Fit and Individual Performance. MIS Quarterly, 19(2), 213-236.

Hoffman, M. & Blake, J. (2003). Computer Literacy: Today and Tomorrow. Journal of Computing in Small Colleges, 18(5), 221-233.

Howard, G. S. (1980). Response-Shift Bias a Problem in Evaluating Interventions with Pre/Post Self-Reports. Evaluation Review, 4(1), 93-106.

Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, S. W., & Gerber, S. K. (1979). Internal Invalidity in Pretest-Posttest Self-Report Evaluations and a Re-Evaluation of Retrospective Pretests. Applied Psychological Measurement, 3, 1-23.

Huberty, C. J. & Olejnik, S. (2006). Applied MANOVA and Discriminant Analysis. Hoboken, NJ: John Wiley and Sons.

Ives, B., Valacich, J. S., Watson, R. T., Zmud, R. W., Alavi, M., et al. (2002). What Every Business Student Needs to Know about Information Systems. Communications of the Association for Information Systems, 9, 467-477.

Janicki, T. N., Fischetti, D., & Burns, A. T. (2007). Incorporating Real-World Projects and Emerging Technologies into One MIS Capstone Course. Information Systems Education Journal, 5(24), 1-18.

Kroenke, D. (2011). Using MIS. Upper Saddle River, NJ: Pearson Education.

Lam, T. C. & Bengo, P. (2003). A Comparison of Three Retrospective Self-Reporting Methods of Measuring Change in Instructional Practice. American Journal of Evaluation, 24(1), 65-80.

Laudon, K. C. & Laudon, J. P. (2012). Management Information Systems – Managing the Digital Firm. Upper Saddle River, NJ: Pearson Education.

Law, W. K. (2003). Challenges for Curriculum Design in IT Education. In M. Khosrow-Pour (Ed.), Information Technology and Organizations: Trends, Issues, Challenges, and Solutions, (1103-1104). Hershey, PA: Idea Group Inc.

Léger, P.-M. (2006). Using a Simulation Game Approach to Teach Enterprise Resource Planning Concepts. Journal of Information Systems Education, 17(4), 441-447.

Lenox, T. L. (2008). The Value of Service Learning in the CIS Curriculum: A Case Study. Information Systems Journal, 6(66), 1-9.

Lim, B. (2002). Teaching Web Development Technologies: Past, Present, and (Near) Future. Journal of Information Systems Education, 13(2), 117-124.

Martin, J., Hands, K., Lancaster, S., Trytten, D., & Murphy, T. (2008). Hard but not too Hard: Challenging Courses and Engineering Students. College Teaching, 56(2), 107-113.

Mitra, S. & Bullinger, T. A. (2007). Using Formal Software Development Methodologies in a Real-World Student Project: An Experience Report. Journal of Computing Sciences in Colleges, 22(6), 100-108.

National Association of Colleges and Employers (2011). Retrieved December 25, 2011, from www.naceweb.org

Neck, H. & Stoddard, D. (2006). Babson College Nominates the Foundation Management Experience (FME) for USASBE’s Innovative Entrepreneurship Education Course. Retrieved July 14, 2013, from http://www.docstoc.com/docs/13452782/Babson-College-The-Foundation-Management-Experience-_FME

Novak, J. D. (2010). Learning, Creating, and Using Knowledge (2nd ed.). New York, NY: Routledge.

Onghena, P. (2005). Resentful Demoralization. In B. Everitt & D. Howell (Eds.), Encyclopedia of Statistics in Behavioral Science (1744-1746), Hoboken, NJ: Wiley-Blackwell.

Porter, M. E. (1980). Competitive Strategy. New York, NY: Free Press.

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring Program Outcomes: Using Retrospective Pretest Methodology. American Journal of Evaluation, 21(3), 341-349.

Sager, J., Mensching, J., Corbitt, G., & Connolly, J. (2006). Market Power of ERP Education – An Investigative Analysis. Journal of Information Systems Education, 17(2), 151-161.

Sanchez, J. I. & Medkik, N. (2004). The Effects of Diversity Awareness Training on Differential Treatment. Group and Organization Movement, 29(4), 517-536.

Scott, E. (2004). Systems Development Group Project: A Real-World Experience. Information Systems Education Journal, 4(23), 1-10.


198

Surendran, K., Ehie, I. C., & Somarajan, C. (2005). Enhancing Student Learning across Disciplines: A Case Example Using a Systems Analysis and Design Course for MIS and ACS Majors. Journal of Information Technology Education, 4, 257-274.

Tan, J. & Jones, M. (2008). A Case Study of Classroom Experience with Client-Based Projects. Journal of Computing Sciences in Colleges, 23(5), 150-159.

Tsai, N. (2002). Measurement of a College Computer Literacy Course. In M. Dadashzadeh, A. Saber, & S. Saber (Eds.), Information Technology Education in the New Millennium, (49-53). Hershey, PA: IRM Press.

Topi, H., Valacich, J. S., Wright, R. T., Kaiser, K., Nunamaker Jr., J. F., Sipior, J. C., & de Vreede, G.-J. (2010). IS 2010: Curriculum Guidelines for Undergraduate Degree Programs in Information Systems. Communications of the Association for Information Systems, 26(18), 359-428.

Valacich, J. & Schneider, C. (2010). Information Systems Today – Managing in the Digital World. Upper Saddle River, NJ: Pearson Education.

VanLegen, C. A. (2010). XBRL: Beyond Basic XML. Information Systems Education Journal, 8(21), 3-6.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly, 27(3), 425-478.

Williams, J. & Chinn, S. J. (2009). Using Web 2.0 to Support the Active Learning Experience. Journal of Information Systems Education, 20(2), 165-174.

AUTHOR BIOGRAPHIES

Suvankar Ghosh is an Assistant Professor of information

systems and quantitative methods with the Beacom School of Business at the University of South Dakota. A veteran of the IT industry, he received his Ph.D. in Business Administration from Kent State University. His research focuses on the valuation of investments in technology, decision-making models, and the adoption and

diffusion of innovations. His research has appeared in Information Systems Research, European Journal of Operational Research, Journal of Organizational Computing and Electronic Commerce, Journal of Information Systems Education, among others. Bijayananda Naik is an Associate Professor Emeritus with

the Beacom School of Business at the University of South Dakota. He received his PhD from the University of Wisconsin-Milwaukee. He has published in several journals including Information Systems Frontier, International Journal of Intelligent Systems in Accounting, Finance, and Management, and

International Journal of Production Research.

Xiaolin Li is an Associate Professor of e-Business and Technology Management with the College of Business and Economics at Towson University. He received his PhD in Management Systems from Kent State University. Dr. Li’s current research emphases are the adoption, continuation, and valuation of technological innovations and buyer-supplier relationships. His research has been published in

Information Systems Research, Journal of the Association for Information Systems, Decision Sciences, International Journal of Production Economics, Journal of Organizational Computing and Electronic Commerce, Journal of Computer Information Systems, Journal of Information Systems Education, among others.


199

APPENDIX: INSTRUMENT

Assessment of Learning along Business and Technology Dimensions Business Dimension

Before the course I would rate my knowledge and understanding about the business aspects of information systems and information technology (IT) prior to taking this course as:

1. Having no knowledge whatsoever 2. Having very little knowledge 3. Just barely knowledgeable 4. Moderately knowledgeable 5. Highly knowledgeable

After the course I would rate my knowledge and understanding about the business aspects of

information systems and IT after taking this course as: 1. Having no knowledge whatsoever 2. Having very little knowledge 3. Just barely knowledgeable 4. Moderately knowledgeable 5. Highly knowledgeable

Technology Dimension

Before the course I would rate my knowledge and understanding about the technology aspects of information systems and IT prior to taking this course as:

1. Having no knowledge whatsoever 2. Having very little knowledge 3. Just barely knowledgeable 4. Moderately knowledgeable 5. Highly knowledgeable

After the course I would rate my knowledge and understanding about the technology aspects of

information systems and IT after taking this course as: 1. Having no knowledge whatsoever 2. Having very little knowledge 3. Just barely knowledgeable 4. Moderately Knowledgeable 5. Highly knowledgeable

Prior Information Systems Background Please list any courses you may have taken in college or high school related to information systems.


200

STATEMENT OF PEER REVIEW INTEGRITY

All papers published in the Journal of Information Systems Education have undergone rigorous peer review. This includes an initial editor screening and double-blind refereeing by three or more expert referees.

Copyright ©2014 by the Education Special Interest Group (EDSIG) of the Association of Information Technology Professionals. Permission to make digital or hard copies of all or part of this journal for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial use. All copies must bear this notice and full citation. Permission from the Editor is required to post to servers, redistribute to lists, or utilize in a for-profit or commercial use. Permission requests should be sent to Dr. Lee Freeman, Editor-in-Chief, Journal of Information Systems Education, 19000 Hubbard Drive, College of Business, University of Michigan-Dearborn, Dearborn, MI 48128. ISSN 1055-3096

Date post:	24-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

IS Course Success in Liberal Arts Institutions – What’s...

Documents