Simple SIS Service(sakora-csv)

Dan McCallum – Unicon, Kim Eke – UNC,

Patty Wolfe - UniconSakai Conference – Denver, Jun 17, 2010

© Copyright Unicon, Inc., 2006. This work is the intellectual property of Unicon, Inc. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of Unicon, Inc. To disseminate otherwise or to republish requires written permission from Unicon, Inc.

1. Concepts

2. sakora-csv at UNC

3. Usage

4. Gotchas

5. Other Options

6. File Formats

Concepts

SSSS Data Flow

SISSakai

DBUpload Tool

CourseManagement

APIs

WWWWWWCSV

CSV

Sakai

lineline

CM Entities

SakaiFS

SakaiFS

CSV

CSV

ScheduledJob

lineline

CSV

CSV

Sakai CM Entities

CM-to-Site Mappings

CM APIs

CM Entities

Provider IDs

Provider IDs

WS SetupTool

Site

GroupProvider

Site Groups/Sections

Users+RolesUsers+Roles

RoleResolvers

<util:map id="org.sakaiproject.coursemanagement.GroupProviderConfiguration"><entry key="siteRoleResolutionOrder">

<list><value>Instructor</value><value>Teaching Assistant</value>

<value>Student</value></list>

</entry><entry key="officialInstructorToSiteRole" value="Instructor"/><entry key="enrollmentStatusToSiteRole">

<map><entry key="enrolled" value="Student"/><entry key="wait" value="Student"/>

</map></entry><entry key="sectionRoleToSiteRole">

<map><entry key="I" value="Instructor"/><entry key="GSI" value="Teaching Assistant"/><entry key="S" value="Student"/>

</map></entry><!– (to be continued…) -->

</util:map>

RoleResolvers

<util:map id="org.sakaiproject.coursemanagement.GroupProviderConfiguration">

<!– (continued…) --> <entry key="courseOfferingRoleToSiteRole">

<map><entry key="CourseAdmin" value="Instructor"/><entry key="I" value="Instructor"/>

</map></entry><entry key="courseSetRoleToSiteRole">

<map><entry key="DeptAdmin" value="Instructor"/>

</map></entry>

</util:map>

sakora-csv at UNC

Sakai Growth

sakora-csv at UNC

• Sakai section awareness killer feature relative

to Bb

• Still feeding files to Bb

• Sites provisioned by request – would like to

remove search/request features in WS Setup

• Coordinator and Tech Support memberships

added at Course Offering level

Usage

Snapshots

• Send all “current” records every time

• (previous file) – (latest file) = deletions

• Applies to:

– sessions.csv

– people.csv

– sectionMembership.csv

– courseMembership.csv

• (Currently) no way to ignore omitted memberships

Upload

curl -F "username=sakora-csv-sub-admin" -F "password=f00b4r" \ -F "sessions=@latest-upload-batch/sessions.csv" \ -F "courseSections=@latest-upload-batch/courseSections.csv" \ -F "courseMembership=@latest-upload-batch/courseMembership.csv" \ -F "courseOfferings=@latest-upload-batch/courseOfferings.csv" \ -F "courses=@latest-upload-batch/courses.csv" \ -F "courseSets=@latest-upload-batch/courseSets.csv" \ -F "people=@latest-upload-batch/people.csv" \ -F "sectionMembership=@latest-upload-batch/sectionMembership.csv" \ -F "runJob=true" \ https://$SAKAI_DOMAIN/sakai-sakora-csv-tool/csv

User Credentials

• Use HTTPS, obviously

• Any super user (site.upd in !admin)

• Batch will always be processed as admin

Directory Polling

• Schedule “SIS CSV Data Loader” job

• Configure (need to do this anyway): batchUploadDir@net.unicon.sakora.api.csv.CsvSyncService

• Currently no way to detect partial uploads

Monitoring

• No UI (patches welcome!)

– Tail catalina.out

– Watch SAKORA_LOG

• Remember processing is async w/r/t upload

• Most logging is at DEBUG level

log.config.count=n

log.config.n=DEBUG.net.unicon.sakora

Monitoring

• {upload.dir}/*-finished.csv – Batch completed w/o fatal error

• {upload.dir}/*-failed.csv – Batch aborted w/ fatal error

• Fatal errors…

– DB, file system exceptions

– Shutdown

– Some file syntax errors

• Batches never retried

Site Membership Syncing

• Pre-2.6 needed SAK-14259

• Not typically needed any longer b/c Site Info

refreshes site membership (SAK-14336)

What About LDAP?

• You’re fine

• Don’t send people.csv if using LDAP for authN

and attributes

• User IDs in people.csv and referencing files

need to be EIDs

– Usually net IDs

Gotchas

Where Did My Terms Go?

• To show up in WS Setup, a term must be:

– “current”

– Within roster.available.weeks.before.term.start

• sessions.csv is the authority for “currentness”

• …even for terms never sent in that file!

I Don’t See My Rosters in Tool X

• Kernel and WS Setup patches

• 2.7

– SAK-18636

• 2.6.2

– SAK-18636

– SAK-17790

– KNL-320

• 2.6.x

– SAK-18636

– KNL-320

I Can’t Remove Instructors!

• Creating user automatically added to new

Site

• Trumps “provided” membership pre-2.7

• SAK-18536

• KNL-403

Instructors Don’t Show up in Roster Tool

• It’s a bug – official instructors aren’t saved

properly

• Actually… it’s the only reason KNL-403 works

with sakora-csv at all

My Section Names Have Commas

• Escape all fields if you can

• Use quotes to escape quotes

EID, Title, Description, Category, parentEID, esEID, coEID“FOO101.SP10.001”,”Intro to “”Foo”””,”T/T, Intro to “”Foo”””,“Lect”,””,””,”FOO101.SP10”“BAR101.SP10.001”,”Intro to “”Bar”””,”M/W/F, Intro to “”Bar”””,“Lect”,””,””,”BAR101.SP10”

My Files are Getting Huge!

• (Currently) no way to ignore omitted

memberships

• Make sure you have indexes in {sakora-csv-src}/sakora-csv-impl/impl/src/sql/

• Can truncate SAKORA_LOG (but not other

SAKORA_* tables)

Sakai is Ignoring the First Line in my Files!

• Set hasHeader@net.unicon.sakora.impl.csv.CsvHandlerBase=false

• sakora-csv doesn’t actually use header rows

as metadata even if hasHeader=true

I Sent an Empty File and Nothing Happened

• E.g…. I just want to delete all records of type

X

• sakora-csv always ignores empty/missing

files.

• Trying to avoid accidental data deletion

• Allows for partial batch uploads

• Sometimes helpful. Sometimes annoying.

How do I Configure X in WS Setup?

• wsetup.skipCourseSectionSelection

• sitemanage.cm.courseset.categories

How do I Configure X in WS Setup?

• titleEditableSiteType

• wsetup.disable.joinable

• roster.available.weeks.before.term.start

How do I Configure X in WS Setup?

• SectionFieldProvider

CourseSet X Doesn’t Show Up In WS SetupDrill-Down Search

• Only top-level CourseSets appear in pull-

down

• Use parentEID field in courseSets.csv reluctantly

Other Options

CM Synchronizer

• A fine option

• Modified versions used at UDel, URI,

probably others

• Undocumented file format

• Code intended as a demo

• Reads entire document into memory

• XML – might simplify your life, might create

strife

SAKORA “Proper”

• IMS LIS reference implementation

• Appealing if you’ve already purchased

PeopleSoft SAIP

• Standardized outcome reporting (but no

Sakai impl yet)

• Event-based

• Less widely deployed than sakora-csv

Others

• Home-grown solutions abound

• Check sakai-dev

– http://bit.ly/c6mEqI

File Formats

File Formats

• Detailed field definitions in Confluence

– http://bit.ly/djyG3o

• Note attached template files

dmccallum@unicon.net

kim_eke@unc.edu

pwolfe@unicon.net

http://www.surveymonkey.com/s/sakai10

http://bit.ly/djyG3o

Questions?