Team AR

Team Augmented RealityEOSP Presentation

1Presenters Kanishk Karanawat & Yashasvi VedulaZexi Chen, Tijing Wang

Contents2Project Context Requirements ManagementProcessRisk ManagementProject Management - Planning & TrackingArchitectureQuality ManagementConfiguration ManagementTraining

Project Context3

Client:NextGen:PGH: Non-profit startup Executive Director: Alec RiegerMentor:Bradley Schmerl

4

People Involved4

Team:

Kanishk Karanawat (Chief Architect,Configuration Manager)Tijing Wang (Risk Manager)

Yashasvi Vedula (QA Manager)Zexi Chen (Project Manager)

People Involved5

Client BackgroundCurrently involved in actively organizing night markets and cultural festivals in Pittsburgh.

Wishes to leverage AR technology to spread culture and build community.

6

7

PlacemakARAPPProject Context

Project Context8

Short-term Goals

Deploy Android app on Play Store (End of July)

Must augment 2D images on at least 3 places of interest on the CMU campus

New goal:Showcase Augmented Reality capabilities through proof of concept:

Render AR VideosInteractable AR ContentGeo-location based AR Content

Project Goals9

Requirements Management

Create user stories at the start of every sprint

Get user stories prioritized and approved by the client after the sprint planning meeting

Need client sign-off regarding prototypes, before development can begin

10

Product backlog for prioritization of user storiesAR tasks high priorityOther mobile tasks medium priorityWeb Portal tasks low priority

Initial release doc guides us in managing requirements Final aim to achieve campus toursWork on components/features leading to campus tourRequirements Management11

Process12

Process

Showcase AR FeaturesDeploy to Play Store

Project Scope

Changes#ActionOutcome1.Change process from OUP to SCRUMRegular client demo/feedback, Higher flexibility to re-prioritize tasks. Used old Release plan to make Product Backlog2.Could Haves priority increasedRender video, Interact with AR3.Should Haves priority decreasedSocial Media integration, Web Preview Mode, Mobile Register/Login

13

We still used our old Release plan to make sure we stick to main agenda (campus tours) but based on user feedback allow re-priortization of tasks

Artifacts: Product backlog, Scrum Backlog, Burndown charts, Team velocity, Meeting minutes, Meeting presentation slides

Process: SCRUM planning meeting (Tuesdays), Stand-up meeting (everyday), Sprint retrospective meeting (Monday), Demo (Monday)

Sprints: 2 week sprints Process: Scrum+14

Tailored: Incorporated risk management, quality assurance and architecture into the scrum process

Evolving: in every retrospective meeting we discussed what changes are requiredExample: In early sprints we realized that some members are not logging work in time in JIRA, which will impact project tracking, so we decided to log work before each standup meeting. Made this a routine in our process. Process: Scrum+15

Entry:Finish the last releaseTasks:Discuss with product owner and add new user stories in product backlog, if neededPull user stories from product backlog and add it to sprint backlog in JIRABreak the stories into tasks and conduct code design for the functionalities (class diagram)Review if the detailed code design is consistent with our high level architectureEstimate the time for each task (Planning Poker). . . . . .

Planning process16

Verification:All tasks with estimates added in JIRA by SCRUM masterSCRUM master checks if planned time estimate higher than available development time. If higher, then put the lower priority tasks back into backlog. Inform product owner about the change

Exit:Add tasks to sprint backlogAssign sprint backlog tasks to individualsAssign the related quality assurance tasks to individuals

Planning process17

Development Process: Organization3 Dev Teams:Web Server Team: Responsible for managing REST API services and database (MySQL, S3 Bucket)Android App Team (Client): Responsible for the AR applicationWeb Portal Team (Client): Responsible for artist and moderator functionalities

18

Development Process: OrganizationNamesAndroid AppWeb PortalWeb Services(API + Database + Monitoring)Kanishk XXYashXZexiXXTijingXX

19

Development ProcessProcessAt the start of the sprint, Web Server team will design APIs and expose them via Swagger.

Mobile & web portal team will edit/design prototype, via Proto.io, and get client sign-off

All 3 teams commence coding once design/prototype is confirmed

20

Swagger Portal 21

Development Process

Scrum master checklist:Assign quality assurance tasks (static analysis, inspection, unit testing, etc.)

Task dependency analysis to determine order of certain tasks

Coordinate workload between team members22

ReflectionsSwitching to SCRUM gave us flexibility with requirements. Client was very happy to see his continuous feedback taken into account

23

Risk Management24

Risk Management Process in ScrumIn every daily standup meeting, the team communicate and discuss about the risk.

In every weeks client meeting and team meeting, identify the risks based on the decision making.

In every sprint planning meeting, discuss and allocated tasks related to risk mitigation and assign tasks

In every sprint retrospective meeting, discuss the risk status and complete the detailed risk document. Discuss the status of the assigned task

Maintain a risk track document to monitor the status of the risks. If any risk deadline is a week away, then notify team.25

Risk Tracking

26

ID8ConditionTarget tracking is to be carried out mostly using markers.Date Identified28-Mar-2016Impact1Probability2ConsequenceSome locations may not allow the placing of markers directly on them. The marker sizes required for tracking from a satisfactory distance may be impractical.MitigationExperiment to identify relation between markers size and tracking distanceCommunicate with university authorities to discuss possibility of placing markers around campusExplore geolocation as a strategy for augmentation at POIDeadline5-May-2016 25-July-2016StatusMitigatedUpdates6-Apr-2016: Experimentation on marker sizes concluded13-Apr-2016: Client meets with ex-President of CMU to introduce app.5-July-2016: We have explore the geolocation augmentation way, and demonstrated it satisfactorily to the client. We need to ensure that all the final POIs on CMU campus can be tracked using an appropriate marker.12-July-2016: Ensured and limited four POIs on CMU campus.

Risk Example27

Planning & Tracking28

Strategic Plan: Track using milestone plan

Tactical Plan: Track using burndown chart, velocity chart, and time distribution chart

JIRA used by Scrum Master to assign tasks and manage Product and Sprint backlog

Tracking29

Every tasks progress is logged in JIRA for creating burndown chart

If we cant finish all the task in one sprint, we push tasks into the next sprint based on priority

We use time percentage chart to view whether we are spending too much time in meeting or too little time in quality assurance.

Tracking30

31

Sprint 3 data

32

Velocity33

sprint 10.961039sprint 20.851613sprint 30.898204sprint 40.967532

Initial Time Distribution Plan5h = meeting [2h (iteration plan) + 1h team + 1h client + 1h mentor]7h = Common time coding1h = Documentation5h = Individual Role + QA6h = Individual coding & UnitOverall55% : Work with team 45% : Work individually

34

Overall Time Distribution Chart35

ReflectionsWe initially used to plot burndown chart upon 100% task completionGave team wrong impression about effort & statusInstead we started plotting on basis of progress. This improved our tracking

36

Architecture37

Quality Attributes - ExtensibilityScenarios & MeasureAbility to add new AR content, new POIs, new Tours No disruption/downtime of any of the existing services or introducing defects in system Design Decision3-Tier Architecture API-Driven Design

TradeoffsSystem complexity increasedIncreased latency (to fetch meta-deta)

OutcomeChanges reflected in system in real-time (mobile app, web portal)

Reusability promoted (APIs reused by clients)

Allows us to scale in future (DB level) and use any DB storage solution and strategy

App re-deployment not required

38

Scalability also covered as we dnt want any disruption when lot of new content is added

Quality Attributes - ModifiabilityDesign DecisionDecided to use Wikitude Javascript SDK instead of Native SDK (Java)AR files stored in data tier (S3 bucket) TradeoffsIncreased code complexity (manage Android Java code as well as HTML/CSS/JS code)

Difficult to test using automation tools

JS SDK allows us to use HTML, JS, CSS code for AR functionalities instead of Java

No need to re-compile/build or deploy code

Changes reflected in system in real-time Scenarios & MeasuresAR component should be modifiable - ability to add/edit AR screens and functionality dynamicallyShould not require re-deployment Outcome

39

Stateless Vert.x Process - Scalability PromotedProcess Monitor - Availability PromotedDynamic View -Web40

Quality Attribute VerificationAIM: Both the software solution and hardware are able to meet the QA requirements. Evaluate T2.small instance (evaluating machine for PROD)Scenario 1: Time taken from the initiation of an action to the response being received must not exceed 5 seconds, when 100 concurrent users are accessing the system. (Average Load)

Scenario 2: Despite having 100 concurrent requests, the time taken to download a graphics file of the largest size (HD quality image) must be kept under 5 seconds. (Peak Load)

41

Quality Attribute VerificationAIM: Both the software solution and hardware are able to meet the QA requirements. Evaluate T2.small instance (evaluating machine for PROD)Scenario 3: 100 users simultaneously reading/uploading data to server & response time should be less than 5 seconds. (Mixed Load)Scenario 4: 100 users simultaneously uploading 10MB content. System should not crash and should handle these requests. (Stress test)42

QA Scenario 3 - Mixed Load43#UsersRPSLatency (sec)1.100551.72.200593.23.300783.7

Scenario 3: Mixed Load - 100 users simultaneously reading/uploading data to server & response time should be less than 5 seconds.Action: POST: /pois/{poiId}/ar/{arId}/uploadContent/{contentType GET requests : POI, AR, Campus 1 MB data upload size. Results: CPU utilization: 9% Memory utilization: 17% (300 users)

Locust Report - Scenario 3 for 100 users 44

Scenario 4: Stress test - 100 users simultaneously uploading 10MB content. System should not crash and should handle these requests.Action: 10 MB data uploaded for 5 minutesResults: CPU Utilization 44% and Memory Utilization 60% Helped found memory related bug in code (Vert.x memory settings different from JVM XmX settings)

45#UsersRPSLatency (sec)Comments1.251.115.830% CPU 30% Memory2.501.23240% CPU 32% Memory3,7515230% CPU 56% Memory 4.10013644% CPU 60% Memory

QA Scenario 4 - Stress Test

Quality Management46

Process

47

Mostly concnerned with maintainbility index and make sure Javadoc comments, hard-coding magic numbers is avoidedManaged 3 diff source codes Android, 2 Web component

In java did manual testing instead of automated testing

We defined unit test as testing one feature/characteristic and & integration as testing multiple together

QA manager periodically generated reports using Stan4j tool

User acceptance testing, where we demonstrated our product at end of sprint and went through the aceptance criteria of the user stories.

Released code in a separate branch

Static AnalysisIntegrated the CheckStyle plugin into the IDEs (IntelliJ for both Android and Java)Modified the default ruleset to exclude some rules. For eg. lines beyond 80 charactersWe had been using Stan4J to generate LCOM4 and cyclomatic complexity metricsIn end used SonarQube to verify out metrics. Mainly concerned with Maintainability index.

48

Metrics Android App49

Rating A, Tech Debt Ratio 4%, 5k LOC 1200 Comments Line

Integration Testing - Android

Junit unit testing only runs on JVM and not on Android VM. Therefore, difficult to test Android Activity Lifecycle.The close coupling of AR, UI, and location-aware functionality made unit testing difficult on Android. Java and JS code also heavily dependentUsed instrumentation testing. Runs on Android phone on Android VM. Hooks to control Android Activity LifecycleWe used Robotium on Android to write automated scripts to test the application after each new module.Behaves in a manner similr to a user with a set of instructions. 50

After researching we found out that junit will only test

Java & JVM code, App location aware and Ar component.

Integration Testing Web Services

For the web services part, we extend JUnit to call the actual web services during integration testing (as opposed to mocking web requests in unit testing)51

Planned coverage:Line - 100% Branch - 80%

Actual coverage:Line - 83% Branch - 35%

Test Cases: 61

We used all-pairs combinatorial testing to test our mobile application in subsequent phasesUsed the classification tree approach to generate test cases

System Testing52

System Testing53Consider various characteristics while designing the classification tree:GeofenceWifi/Net conditionsLighting conditionAR Content typePhone OrientationUsers walking speed ..

ReflectionsAs UI and functionality get more intertwined, it gets more difficult to test separate units. In such cases, traditional methods and metrics become hard to follow. We had difficult time differentiating between unit tests & integration tests and had to define our own terminologies.

We initially wanted to automate all our tests, but later realized that for location and AR use cases, manual testing may be a more thorough alternative.54

Configuration Management55

Bitbucket Repository: Code version control (2 repos: Android code, Web server & Web portal Code)Jenkins Continuous Integration: Build Changes & Deploy (Archive Successful Builds, Auto-Polling every minute)AWS Dev Env: Restart process for latest changes to reflectRelease branch: created for configuration management & deployments

Flow

CommitAuto-PullBuildDeployProcess RestartedBranching56

Jenkins archive helped us in of our demos as once one of the developers didnt follow the QA process and pushed changes for demo. When the new deployment broke, we could quickly roll back to previous version to proceed with demo.

Branching Strategymasterfeaturerelease

v0.1v0.2v0.3v 1.0

BranchPull & IntegrateMergeBranchPush Bug Fixes57

Training58

TrainingTrainings for Android, Wikitude SDK (Android), Web Development (Bootstrap, JQuery), Vert.x Web Server & BitbucketETVX process followed:Entry: Environment setup/Software installation on local machinesTask: Training Tasks mapped with learning outcomesValidation: Demonstration of training tasksExit: Check-in training source code59

Training60Training designed to include common functionalities used in projects (Derived from list of requirements to be implemented)

Different set of trainings for developers with Beginner and Intermediate experiences

Different trainings for different developers (based on assigned system)

Structured process helped everyone to have common understanding & establish terminology. Used common working hours for training. Helped beginners to learn from more experienced developers. Saved us time in configuration issues & training task issuesLocal system setup helped us start development immediately after training Still need for individual custom trainings for personal learningReflection61

THANK YOUQ/A62

Supplementary Slides63

Top 3 RisksRisk 3 : A number of successful AR SDKs, such as Metaio and ARPA, have been discontinued with no prior warning to developers. Thus, there is some concern that the same may happen to the SDK picked by the team.

Risk 18: The marker tracking is dependant on the lighting conditions as well as the distance.

Risk 16: Communication is delayed sometimes when some issue is experienced, while working on a task

64

Add technical risk- AR tracker detection from particular distance/angle

ID17ConditionClient will be unavailable from last week of July onwards till end of projectDate Identified28-Jun-2016Impact1Probability4ConsequenceSince client is non-technical and will be out of town, it would be difficult to have the handover process as well as receive final sign-off.MitigationInitiate the handover process earlyStart user acceptance testing earlyDevise handover plan and give handover when client is available in town, and handover easier part through phone/mail communicationDeadline20-July-2016StatusMitigated

Risk Example - 265

Handover plan included: User manuals: Android App, Web Portal, AWS Portal, Technical Repo (containing architecture docs, source code), Web API Documentation

6. Make user stories for this release7. Make prototype in proto.io for these stories8. Have client approve the prototype9. If not approved, change the prototype accordingly and get approved10. Break the stories into tasks and conduct code design for the functions (class diagram)11. Review if the detailed code design is consistent with our high level architecture12. Estimate the time for each task13. Put the tasks in the product backlog using JIRA

Backup - Planning process continued66

API Dependency Issue - BackupAPI Dependency issue:How to coordinate effectively between REST API team and Client team (mobile and web)?Communicate and have common understanding about data modelUnderstanding regarding JSON data returnedCommon understanding between error handlingCommon understanding between HTTP methods availableInitially ran into delays due to dependencies

67

API Dependency68Solution: Use SwaggerHelps design, test, document APIsWeb UI to explore APIs. Provides documentation. Run APIsDocumentation can be used by future teams as well to understand existing APIshttp://placmakarapi.cf:8081/

69

Process Backup - Swagger Portal 2

Process Backup - Scrum+ Example : Stand up meeting15 minutes daily stand-upEntry:4 team members are attending the meeting. Tasks:Discuss daily progress, including What each team member did yesterday and what he is working on todayWhat are the problems and difficultiesWhat are the concerns (risks). Log work in JIRA

70

Process Backup - Scrum+ Example: Stand up meetingVerification:All the steps above are followed.JIRA is updated

Exit:Finish the meetingScrum master updates Burndown chart

71

Process Backup - Retrospective ProcessEntry:Finish the last sprint. Finish the sprint review (sprint demo) meeting.Tasks:Discuss last retrospective decisions: whether they have been met.Discuss what the team should: Start doing, Stop doing,Continue doing. Vote on the matters:Extra tasks:Calculate velocity ;Tracking: collecting individual feedback on quality assurance tasks as metrics. Collecting feedback on the above discussion as metrics. JIRA update

Verification:All the steps above are followed.

Exit:The list of decisions are documented.72

Backup -Architectural Verification - Mobile App

73

Backup - WBS

74

Burndown Full Semester75

Code Readability & Maintainability increaseCode Re-usability increasesStatic View - Mobile Backup 76

Instance Monitor - Availability PromotedBackup: Physical View77

Backup: Dynamic Mobile View78

79

Sprint Backlog

Product Backog80

Total 26 user stories

Architecture Backup - Test EnvironmentTest Environment:Machine for generating requests C4.2x large (8 cores 15GB RAM)Locust used for load testing.Tested T2.small instance (evaluating machine for PROD)Delay between 2 consecutive requests : minimum : 5ms & max: 50ms

81

Architecture Backup - QA Scenario 1 - Average LoadCPU utilization: 7% Memory utilization: 7% (300 users)GET requests : POI, AR, Campus#UsersRPSLatency (ms)1.1002234262.2002059363.3002151359

82

Architecture Backup - QA Scenario 2 - Peak LoadScenario 2: Despite having 100 concurrent requests, the time taken to download a graphics file of the largest size (HD quality image) must be kept under 5 seconds. Action: Download from S3 10 MB payload

83#UsersRPSLatency (seconds)1.100253.72.200207.83.30019.310.7

Scenario 2: Despite having 100 concurrent requests, the time taken to download a graphics file of the largest size (HD quality image) must be kept under 5 seconds. Action: 3 MB Payload

84#UsersRPSLatency (seconds)1.100701.22.200822.33.300803.5

Total Cost Per Month = $28.04

Final Infrastructure Cost85#ComponentTypeQuantityCostComments1.AWS EC2 Instancet2.small1$19.04For hosting the web server, application servers and database server (and build server for continuous integration, if required)2.Elastic Block StorageSSD Storage1$0.50For permanent data storage (software installations, source code, database data)3.S3 Bucket-1$1.50For storage of multimedia content4.CloudWatch/ Auto Scaling -1$6.5For real-time monitoring, alerting, availability5.Route 53-1$0.50For DNS Mapping

Finalized T2.Small instance for deployment, based on QA Verification

Backup - CheckStyle

86

Backup - Coverage

87

Static Analysis Stan4J88

QA Backup - Defect Metrics89DateModuleDefectsKLOCDefects/KLOCComments23-Jun-2016Android App53.71.35Defects logged in Jira14-Jul-2016Web Services61.9k3.16Unit testing defects14-Jul-2016Web Services21.9k1.05After removing unit test defects20-Jul-2016Android App34.80.625Defects logged in Jira24-Jul-2016Android App05.10Defects resolved1-Aug-2016Android App75.11.37Combinatorial testing started2-Aug-2016Web Services00Resolved all defects found

Static Analysis Web Services90

Alll REST APIs documented using Swagger: http://placmakarapi.cf:8081/

Bug List Sample91

Classification Tree92

Only for critical modules

Inspection checklist provides inspectors with guidelines

Inspection document to keep track of defects encountered during inspection

Review/Inspect93

Inspection Sample94#ModuleFileTypeLineDefectSolutionWeb Server/ app_serverorg.ngpgh.ar.webserver.model.PoiModelVariable name4Variable name poidID is incorrect based on API contractRe-name to poiID2Web Server/ app_serverorg.ngpgh.ar.webserver.controller.MainRouterAPI Implementation 260GET /pois/{poiID}/ar(Function getArListByPoiID)Does not implement query parameter size3Web Server/ app_serverorg.ngpgh.ar.webserver.dao.ArDaoCoding Style82ArrayList is created and then converted to Array which is returned. Extra code written for this operation.Return ArrayList instead and remove array conversion code.4Web Server/ app_serverorg.ngpgh.ar.webserver.controller.MainRouterRefactor292GET /pois/:poiId/ar/retrieveByFieldimplementation has duplicate code from API - GET /pois/:poiId/ar/Remove duplication to improve readability5Web Server/ app_serverorg.ngpgh.ar.webserver.model.UserModelAPI Implementation 6Variable name userID is missing from POJO, based on API contractAdd field userID in POJO and also in corresponding DAO6Web Server/web_portallogin.htmlMissing fieldField zip code is missing in registration page as requested by clientAdd this field in html page as well as in database

User Acceptance Testing

Carried out in every demo meeting with the client

We ask the client to use a feature implemented for each user story

His feedback is recorded and any recommended changes are added as enhancement tasks for the next sprint95

User Story With Acceptance Criteria Example96User Story 11: As a CMU admin, I want to render interactable AR content, so that the students can gain more information about Campus locations/events

UAT-11a: User must be able to render at least one button over the CMU map on Forbes and Morewood. The user must be able to click this button, which would then invoke a side drawer. This side drawer would display information on the specific location corresponding to the button clicked.

User Story 12: As a mobile app user, I want to see all the nearby POIs in a map view, so that I can easily visualize all the nearby POIs.

UAT-12a : The user must be able to see a Google Map with exactly 5 markers corresponding to the 5 POIs. On clicking the marker on the map, an info-window must be displayed to the user with the title and description of the POI.UAT-12b : The user must be able to click the info-window to be directed towards the page where the details of the specific POI selected are displayed.

TrainingSample NameAndroid Training 1 - Time Estimate 8 hoursEntryDownload Android Studio (https://developer.android.com/studio/index.html)Read DevPlanEstimationNextSem2.0 document to get better understanding of the various systems and modules to be developed.Tasks(For beginners)Create a hello world Android App (2 hours) Outcome: Learn usage of basic UI components, activity management, intent calls. https://developer.android.com/training/basics/firstapp/index.html OR (For intermediate users) Interactive Story app (3 hours) Outcome: Advanced app which demonstrates usage of MVC pattern, and more control widgets, as well as activity management. https://teamtreehouse.com/library/build-an-interactive-story-app(Advanced) Weather App (4 hours) Outcome: Learn integration with 3rd party API, JSON Parsing https://teamtreehouse.com/library/build-a-weather-appGPS location updates (1 hour) https://developer.android.com/training/location/receive-location-updates.htmlStorage (2 hours) https://developer.android.com/training/basics/data-storage/databases.htmlInter-app interaction (1 hour) https://developer.android.com/training/basics/intents/sending.htmlValidationFirst 2 tasks are mandatory for all the developers. Validation will include demonstration of the Android App deployed on the developers smartphone/emulator.Depending on which developer is responsible for which module, rest of the developers will need to finish their respective trainings and demonstrate to rest of team to show their understanding of the concepts.ExitAll the developers finish their respective trainings.At end of training developers will need to push their respective code to GIT repository.

97

Training Backup - Design ExampleAndroid Training Objective:Prioritized list of items, from most important concepts we are interested in learning to least important:

UI Components and Event handling (List view, map view, tabs, controls widgets [buttons, drop-down, spinner, etc.] derived from prototypes) Activity Management - Activity, Services, Intent callsIntegrate with external APIs Sensor Interaction (GPS)Data Persistence (Local storage and MySQL storage)Interact with other Apps (Invoke other mobile apps)98

Training Backup - Experience & Area of Work

99