Development and Initial Validation of a Student Assessment Toolkit for Networked Learning in Higher Education Settings

Laura GogiaDecember 2014

The Digital Age

Period in human history in which traditional industry of the industrial revolution has been replaced by an economy based on digital information and processes.

Photo: “Where is the Internet?” Flickr user Lea, https://www.flickr.com/photos/violinha/

The Digital Age is Networked.

People and information sources are linked together in a global network, allowing for unprecedented and disruptive forms of communication.

Photo: “Virtual Marketing, Shirley Chan http://09820006s.blogspot.com/

The Digital Age is Participatory.

Consume but also actively contribute. Affiliation

Formation Creative

Expression Collaborative

Problem Solving Contributing to

Information Circulation

Photo: “Michael Wesch Keynote” by Flickr user Guilia Forsythe https://www.flickr.com/photos/gforsythe/

Networked & Participatory

The new economy and culture have sparked calls for educational reform among a variety of

stakeholders. 21st Century skills New Media skills

Digital skills

Photo: “What’s in my bag?” by Flickr user Brandy Shaul, https://www.flickr.com/photos/zoologist/

Networked Learning

Takes advantage of: The open, distributed

spaces of the web The participatory,

creative nature of digital tools

The power (reach) of networked communication

The power (potency, innovation) of networked thinking

Photo: “Network” by Simon Cockell https://www.flickr.com/photos/sjcockell/

The Problem

Networked Learning emerged from informal learning environments, where

formal assessments of student learning are not common and not necessarily valued.

The Problem

Traditional formalized learning assessments are not consistent with networked learning

pedagogies.Networked Learning Traditional Assessments

Values learning as a process Values knowledge as acquired object

Values collaborative work Value individual work

Values remixing, synthesis Values “original” work

Study Purpose

To create and validate an assessment toolkit for grading student performance in networked learning courses in higher education environments.

Aligned

Agile

Valid

Digital

Study Significance

Networked Learning Is Becoming More Popular In Higher Education.

Higher Education Requires Formalized Student Assessments.

Techniques Or Protocols Must Be Developed That Provide Formalized Assessment Without Compromising Pedagogical Alignment.

What Needs to be Assessed?

Networked Learning (noun):

Connectivist-based learningthat takes place on

distributed, openly networked, &

digital platforms

Networked Learning

Connectivist-based

learningthat takes

place on distributed,

openly networked,

&digital

platforms

Distributed: Consists of more than one platform, supporting different types of activities.

Openly networked: Occurs on the open web, which allows students to interact with a larger community & provides more opportunities for students to make connections between the “spheres of learning.”

Digital: Internet-based

“Connectivist-Based”

Knowledge: a set of connections formed by actions and experience distributed across networks of human and non-human nodes. It is dynamic, changing quickly based on context. It is interpreted by the individuals who are making the connections. It rests in the diversity of opinions.

Learning: the process of making decisions through the connection of information sources.

(Downes, 2006)

“Connectivist-Based”

Core skills include the ability to:▪Choose what to learn & interpret it based on a lens of shifting reality ▪Nurture & maintain personal connections for continuous learning. ▪See connections across fields, ideas, and concepts.

Ability to learn through this process is more critical than current knowledge base.

(Siemens, 2004)

Networked Learning Principles

Developing & Maintaining

Personal Learning Networks

Curating, Critiquing, &

Organizing Data and Data Sources

Connecting Concepts & People Across Spheres of

Learning

Transforming Concepts into a

New Product

Sharing with Personal Learning

Network

How Should It Be Assessed?

Computer-Supported Collaborative Learning Literature (Strijbos, 2011)

Participation

Social

Content Acquisiti

onIndividua

l

Assessment Principles

Participation

 

Individual Extent to which students contribute to the social stream

Transactivity

Extent to which students refer to and build on each other’s contributions

Transliteracy

Extent to which students move across types of digital media

Content Engageme

nt

Sharing Extent to which students share resources or information

Construction

Extent to which students use information to solve problems within the same situation in which the information was presented

Transformation

Extent to which students collaboratively remix or repurpose information to create new knowledge

How are these operationalized?

(Fact: Digital platforms record the actions of its users and store them in ways amenable to analysis and visualization)

**Preliminary Research of Social Media Analytics from a Networked Learning Course provided potential factors for student assessment.

OverviewNetworked Learning Principle

Assessment Principle

Operationalization

Blogs Twitter

Establishing and Maintaining a Personal Learning Network (PLN)

Participation (Individual)

#Posts, Comments 

#TweetsNetwork Degree Centrality

Curating, Critiquing, Organize Data and Data Sources

Content Engagement (Sharing)

Keywords#Links

Keywords#Links

Connecting or coordinating people and concepts over space, time, and spheres of learning a

Participation (Transliteracy) Ratio: Posts, Comments, Tweets

Participation (Transactivity)

#LinksClassmate Mentions

#Retweets, Mentions, Replies, LinksNetwork Betweeness Centrality

Content Engagement (Construction)

KeywordsLinks (Content)

KeywordsLinks (Content)

Transforming data into new products

Content Engagement(Transformation)

Content Content

Sharing new product with PLN

Participation (Individual) #Posts

#TweetsNetwork Degree Centrality

Participation (Transactivity)

Links (Content)Classmate Mentions

 #Mentions, Replies, LinksNetwork Betweeness Centrality

Research Questions

How can networked learning be documented for the purposes of student assessment?

How can social media analytics be used to inform student assessment in networked learning environments?

How does the use of social media analytics relate to a qualitative content analysis of the same data?

How can social media analytics be integrated into a larger student assessment toolkit for higher education faculty?

Research Design

Two Phase Research DesignToolkit Construction

Toolkit Validation

Construction Phase

Exploration of Student Data in Network Learning Environments▪ Descriptive Statistics▪ Quantities of student participation in the form of posts,

comments, retweets, replies, mentions, etc.

▪ Social Network Analysis▪ Characterization of interactions between students within

the community

▪ Content Analysis▪ Identification of key themes within the discourse

▪ Discourse Analysis ▪ Characterization of interactions between keywords, posts

containing keywords, and students using keywords.

Student Data Sources

Secondary Data Analysis #Thoughtvectors MOOC

VCU undergraduate 2014 summer course 95 Student Participants and 30 Open Participants Students were required to blog daily and comment on each

others’ posts – 98% of activity took place on WordPress Over 2309 student posts and 419 open participant posts Twitter discourse was encouraged but with limited student

participation – approximately 4000 tweets total

#ConnectedCourses MOOC Digital Media Literacy Research Hub course on teaching

connected courses with over 250+ active participants Weekly blogging encouraged (over 1880 posts) Active Twitter discourse among students

Tools for Analysis

Application Programming Interface (API)

WordPress Plug-In

Microsoft Excel

KBDeX

Descriptive StatisticsKeyword Searches

NodeXL

Content AnalysisDiscourse Analysis

Social Network Analysis

OverviewNetworked Learning Principle

Assessment Principle

Operationalized (Variables) Measurement Tool

Blog Twitter

Establishing and Maintaining a Personal Learning Network (PLN)

Participation (Individual)

Number: Posts, Comments 

Number: Tweets,People Followed

TAGS-WP/Excel

Curating, Critiquing, Organize Data and Data Sources

Content Engagement (Sharing)

KeywordsNumber: Links

KeywordsNumber: Links

KBDexTAGS-WP/ExcelProduct Assessment b

 

Connecting or coordinating people and concepts over space, time, and spheres of learning a

Participation (Transliteracy)

Ratio: Posts, Comments, Tweets TAGS-WP/Excel

Participation (Transactivity)

Number: LinksClassmate Mentions

Number: Retweets, Mentions, Replies, Links

 TAGS-WP/Excel/NodeXL  

 Content Engagement (Construction) 

KeywordsContent: Links

KeywordsContent: Links

KBDexProduct Assessment b

Transforming data into new products

Content Engagement(Transformation)

Content ContentKBDexProduct Assessment b

Sharing new product with PLN

Participation (Individual)

Number: Posts Number: Tweets TAGS-WP/Excel

Participation (Transactivity)

Content: LinksClassmate Mentions

 MentionsRepliesLinks 

TAGS-WP/Excel

Validation Phase

Construct Validity:▪ Qualitative Content Analysis of post and comment

text▪ How do descriptive statistics, social network analysis, and

discourse analysis metrics relate to a thematic analysis of student posts and comments – with an emphasis on networked learning principles?

Internal Validity:▪ Statistical Analysis▪ Looking for opportunities to assess internal validity through

factor analyses and Chronbach alpha testing▪ Potential correlation with post-course surveys meant to

measure “connectedness”▪ Potential correlation with final course grades

Limitations

This is an exploratory process which will need continued study, replication, and refinement.

Some student data obtained from courses not consistent with formal higher education settings.

Results and usefulness of toolkit are impacted by instructor’s devotion to making networked learning a priority in the learning environment.

Questions

References

Downes, S. (2006). Learning networks and connective knowledge: Discussion paper #92. Instructional Technology Forum. Retrieved from: http://it.coe.uga.edu/itforum/paper92/paper92.html

Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge building discourse explorer: A social network analysis application for knowledge building discourse. Educational Technology Research and Development, 60(5), 903–921. doi: 10.1007/s11423-012-9265-2

Siemens, G. (2004, December 12). Connectivism: A learning theory for the digital age. Retrieved from: http://www.elearnspace.org/Articles/connectivism.htm

Strijbos, J.W.(2011). Assessment of (computer-supported) collaborative learning. IEEE Transactions on Learning Technologies, 4(1), 59-73.