ITANA 2016: API Architecture and Implementation

ITANA API / Governance Working Group

U N I V E R S I T Y O F W AT E R L O O

API ARCHITECTURE AND IMPLEMENTATIONP O W E R I N G S T U D E N T A N D I N S T I T U T I O N A L I N N O VAT I O N

Colin BellDirector Enterprise ArchitectureEA, ISTBMath2008uwat.ca/ist-ea

Zachary SeguinOpen Data API Incoming Dev Lead Client Services, ISTBCS Computer Science2017uwaterloo.ca/open-data

Kartik TalwarOpen Data API Outgoing Dev Lead Client Services, ISTBSc Physics2016uwaterloo.ca/open-data

Nate VexlerOpen Data API Service LeadClient Services, ISTBASc Systems Design Eng.2012uwaterloo.ca/open-data


AGENDA

Background1. Prime Motivators2. Routine Disclosure3. Why Open Data First?4. 5 Star Data5. Open Data License6. History of Open Data

Technical Specifics7. Data Sets8. Data Acquisition9. Technology


AGENDA

Growing1. Governance and Policy2. Private Data (PbD)3. Business Process4. Data Warehousing/ESB/iPaaS5. Future Directions

Questions6. Buy vs. Build?7. How to build today?8. Documentation?9. How do you deal with the ERP

culture clash?


BACKGROUND


ORIGINAL PRIME MOTIVATORS

• Enable Student Development• Enable Homebrew Student Portal


PROTECTING PRIVACY

http://eaves.ca/2013/01/07/the-journal-news-gun-map-open-vs-personal-data/


MAINTAINING TRANSPARENCYPROTECTING PRIVACY

bit.ly/ipc_abd bit.ly/ipc_pbd


ROUTINE DISCLOSURE• Pol icy of Ontar io’s

Information Privacy Commissioner

• 7 pr inciples• bit . ly / ipc_abd

nexus

@Colin: add stuff here


ROUTINE DISCLOSURE1. P r o a c t i v e , n o t R e a c t i v e2. A c c e s s E m b e d d e d i n t o D e s i g n3. O p e n n e s s a n d Tr a n s p a r e n c y =

A c c o u n t a b i l i t y4. F o s t e r s C o l l a b o r a t i o n5. E n h a n c e s E f f i c i e n t G o v e r n m e n t6. M a k e s A c c e s s Tr u l y A c c e s s i b l e7. I n c r e a s e s Q u a l i t y o f I n f o r m a t i o n

nexus

@Colin: add stuff here


WHY OPEN DATA FIRST?

Personally Identifiable Data

ITANA API / Governance Working Groupbit.ly/ipc_abd



Emerging trend in Government (Toronto, Vancouver)

New Vertical (Higher Ed)


Open Data License


OPEN DATA

5stardata.info


IMPROVING OPEN DATA

inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/


WHY IS OPEN DATA IMPORTANT

• Economies of Scale • Value of Data • Cost of Using Data


WHY IS OPEN DATA IMPORTANT

DO MORE W/ LESS

Fall 2009• Nathan lobbies with

others at Student Technology Advisory Committee re Portal/Open Data

March 2010• Presentation at

High Level Computing Committees UCIST/CTSC

Fall 2010• Jeff Verkoeyen’s

uwdata.ca becomes official

Aug 2011• Kartik Talwar

creates api.youwaterloo.ca

Feb 2012• api.uwaterloo.ca

becomes official

Open Data @ uwaterloo Timeline

2012• Nathan Joins IST full time• Open-Data powered

apps:• - Student Portal (internal)

• - uwflow.com (external)

2013• V2 launches• Drupal-Powered Content

Management System is leveraged to deliver Open Data for use in Student Portal

2014• Open Data-powered

Student Portal launches;• mandate to bring

api.uwaterloo.ca deeper into the enterprise

2015• Campus Map Project

(powered by Open Data) is in development

• Student Portal project continues to further the momentum of Open Data

Open Data @ uwaterloo Timeline


IPC CORE DESIGN PRINCIPLES

bit.ly/ipc_abd bit.ly/ipc_pbd

Student PortalAbD/PbD Strategy(Colin Bell)


IMPROVING OPEN DATA

inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/


API: APPLICATION PROGRAMMING INTERFACE


API: APPLICATION PROGRAMMING INTERFACE

CORE AREAS OF SERVICE OF CURRENT API

Open Data API

Student Engagement

(Portal)

Core IT Service (WCMS)

For Students by Students

Emerging Academic

Use

uWaterlooGitHub

github.com/uwaterloo

IRL Community with

DemoCamps

Stay in the loop with our

Mailing List

bit.ly/oduw_ml

http://bit.ly/oduw_ml

Community Created Toolsuwaterloo.xyz


APPLIED EXAMPLE: RESPONDING TO COMMUNITY NEED

GOOSE WATCH

BACK to April 9, 2013


Three Days PRIORAppeal For Data

THE STUDENT SUCCESS OFFICE SWINGS IN ACTION TO PROTECT STUDENTS

Three Days PRIOR Sketch/Prototype


Mid Afternoon the Day BeforeThe endpoint launches

THE UWATERLOO COMMUNITY S W I N G S I N TO A C T I O N TO R E Q U E S T / D E L I V E R D ATA

Two Days PRIORThe reddit Data Request


THE RESPONSE


PROGRESS!

5stardata.info

API

PNG


TECHNICAL SPECIFICS


DATASETS

Food Services

News and Events CO-OP WATpark

New Campus Map Tutors Services Awards


Food ServicesLocations and Hours

Daily menuNutritional Information


CO-OP DataEmployer Info-Sessions

Career Centre Workshops


TutorsFind a Tutor for a courseSee their availability and

get help


WATParkSee live parking lot capacity

Get updates on opening and closing hours


GithubStudents Contributing Data

Students submitting datasets


New Campus Map


CURRENT STATE: DATA ACQUISITION

ScrapersGithub• Crowdsourced• Maintained

CMS (Drupal)• Proxy requests• Pub/sub

Other Proxy Requests

Direct Database

Integrations


FUTURE STATE: DATA ACQUISITION

ScrapersGithub• Crowdsourced• Maintained

CMS (Drupal)• Proxy requests• Pub/sub

Other Proxy Requests

Direct Database

Integrations

WarehousingESB / iPaaS


CURRENT STATE: DATASETS

Food ServicesDB

News/Events/etcPub Sub

CO-OP WATparkProxy

New Campus Map

Curated GH

TutorsProxy

AwardsProxy


TECHNICAL ARCHITECTURE

• Student initiated, no fancy ($) tech involved.

• After a number of iterative improvements, this is our stack.

api.uwaterloo.ca

App Server 1 App Server 2

Incoming Webhooks

Workers

Other Databases

Proxy ServicesAPI DB

App Server

Incoming Webhooks Proxy Services Workers

Github sync, scrapers, updaters

Live requests from other internal APIs

New posts from university websites

API DB Other DBs

App Server

API Request Router (PHP Klein)

Tutoring Services Food Services

FS DB API DB

GET METHOD 1GET METHOD 2GET METHOD 3

Serve from FS DB

Campus Events

GET METHOD 1GET METHOD 2GET METHOD 3

Proxied via internal tutoring API

Tutoring API (other department)

Crons

- sync datasets- run scrapers- cache things

Keep data on file system

GET Serve from DB

POST (webhook) Process and update DB


DOCUMENTATION / ISSUE TRACKING

• Docs: Github w/ Markdown generator• Issue Tracking

Community Issues: GithubService Desk: RTInternal: Gitlab/Jira


GROWING API CAPABILITY


GOVERNANCE AND POLICY

• Administrative Information Governance Committee (AIGC)

Vice Presidents, Senior Management, and Associate Provosts2015 decision to consolidate IM related Policies.Procedures and Guidelines to follow Policy. Draft of Policy being moved through review now.


PRIVATE DATA

• We have few private data APIs on campus. Faculty developed advising system provides student information backbone.

• They are starting to grow. Principle established prioritizing API integration pathways.

• Real uptake depends on Identity and Access Management implementations.

Shibboleth, SAML, OAUTH2 to come 2016/2017.


BUSINESS PROCESS

• Working to change the relationships.Initially we leaned on AbD, many units felt ’bad things’ were happening.Proven that the sky has not fallen.Developing a Steering Committee model to help direct future development merging community needs with institutional goals.Develop SLA with Information Stewards.


DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS• Integrations on campus have been file or DB based.

Times have changed, we are starting to change.• Open Data has provided us with knowledge and

expertise in APIs.• As warehousing improves and we bring either an

ESB or iPaaS solution on campus, our data architecture will start to take shape.


DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS

Data Warehousing

Integration Engine (ESB/iPaaS)

HR

Finance

SIS

Co-Op

…

OR

AbD(Open)

PbD(Priv)

API


FUTURE DIRECTIONS

• Composable Microservices• Create Once, Publish Everywhere (COPE) -

NPR Model

University of Waterloo WCMS Create Once Publish Everyone COPE Strategy(Andrew McAlorum)

News and Events

GROWING AREAS OF SERVICE

Open Data API

Student Engagement

(Portal)

Core IT Service (CMS + more)

For Students by Students

Emerging Academic

Use

Campus Map

Research


THE OPEN DATA ENGINE

(Demand-First)• Exam Schedule• CO-OP workshops• Tutors• Food Services

(Supply-First)• Services• News Events• Awards• Publication


[email protected]@[email protected]@uwaterloo.ca

Buy vs. Build?How to build today?

How do you Document?How to mitigate ERP culture clash?

QUESTIONS?

mailto:[email protected]




Date post:	14-Apr-2017
Category:	Technology
Upload:	colin-bell
View:	120 times
Download:	0 times