Date post: | 14-Jan-2016 |
Category: |
Documents |
Upload: | emil-berry |
View: | 216 times |
Download: | 0 times |
December2002, GarchingALMA Computing IDR
ALMA AIPS++ Audit
Steven T. Myers (NRAO)
2ALMA Computing IDR – December 2002, Garching
Process
• SSR Requirements– Offline Data Processing Requirements (SW-18)– Not package specific, any package must fulfill req.
• ALMA AIPS++ Audit– AIPS++ is baseline plan for ALMA– First step is to audit AIPS++ against SW-18 requirements– If too many unmet, may want to rethink baseline plan– Most important is where AIPS++ will be in 2007– Includes performance benchmarking and user testing
• Progress monitoring– “delta” progress must be monitored– Cycle timescales TBD, e.g. 12-18 months? At milestones?
3ALMA Computing IDR – December 2002, Garching
Process (2)
• Feedback to Requirements Process– Requirements may need to be modified or refined– Take input from ALMA project, or ASAC etc.
• Input to AIPS++ development planning– Identify milestones (e.g. ALMA Level 2 and 3)– Costing and delivery for unfulfilled Priority 1 & 2 items– Iterate with ALMA Computing and SSR
• Eventually move toward acceptance testing– Carried out by SSR?– Official procedure?
4ALMA Computing IDR – December 2002, Garching
Current Status
• ALMA SSR for “Offline Data Processing” SW-18• Completed and reviewed Apr 2002• e2e SSR May 2001, revised Nov 2002
• Audit • Started Jul 2002, drafts Sep 2002 & Dec 2002• Based mostly on documentation, only minor testing• Ready for SSR comment, passed to AIPS++ project
• Next• Performance benchmarking, testing (need leader)• Revise priorities, add timescales (e.g. based on milestones)• Lead up to next audit in 2003 or 2004
5ALMA Computing IDR – December 2002, Garching
Grading (1)
• First pass (2002)– Audit state of AIPS++ as of September 2002 in meeting
ALMA requirements (SW-18)– Identify if functionality is present, based on documentation
and auditor inventory of package– Some testing (e.g. on existing images or data)– Fold in AIPS++/IRAM tests where possible (e.g. in iramcalibrater module), appeared in builds at end of audit
• Procedure– Identify AIPS++ tools, functions, and documents relevant to
each requirement– Grade based on functionality, usability, and/or
documentation (depending on specific requirement)
6ALMA Computing IDR – December 2002, Garching
Grading (2)
• Priorities (from SSR SW-18 Requirements)– 1 = Critical (all Priority 1 features should be present)– 2 = Important (90% of Priority 2 items should be fulfilled)– 3 = Desirable (enhancements and future development)
• Grades (from audit)– A = acceptable– A/E = acceptable, but enhancements desired– I = inadequate– N = not available– U = unable to grade (e.g. ALMA definitions needed)
7ALMA Computing IDR – December 2002, Garching
Grading (3)
• Severity (for I and N grades)– low – medium– high
• Grading procedure– 3 principal auditors (Myers, Viallefond, Morita)– Plus deputy auditors (Brogan, Coulais, Caillat)– Input from others (Lucas, Glendenning, Cornwell, Brouw)– Myers audited all req., tried to have overlap on most others– In cases with disparate grades (~14% of req.), Myers drafted
unification
8ALMA Computing IDR – December 2002, Garching
Results – Chart Guide
Work to be done by ALMA
These should be 0 (in ~2007)
These should be <10% of the total
Explanatory – not results!
9ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (489) 52% / 9% / 33% /
6%
–Priority 1 (293)60% / 7% / 29% / 4%
–Priority 2 (135)49% / 12% / 32% / 7%
–Priority 3 (61)23% / 13% / 54% / 10%
Results – Overall
10ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (23)52% / 4% / 17% /
26%
–Priority 1 (12)42% / 8% / 33% / 17%
–Priority 2 (9)67% / 0% / 11% / 22%
–Priority 3 (2)50% / 0% / 50% / 0%
Results – 1 General
11ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (53)58% / 11% / 26% /
4%
–Priority 1 (26)65% / 4% / 27% / 4%
–Priority 2 (22)55% / 23% / 18% / 5%
–Priority 3 (5)40% / 0% / 60% / 0%
Results – 2 Interface
12ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (127)81% / 1% / 16% /
1%
–Priority 1 (91)86% / 1% / 12% / 1%
–Priority 2 (24)88% / 0% / 4% / 8%
–Priority 3 (12)33% / 0% / 67% / 0%
Results – 3 Data Handling
13ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (76)22% / 4% / 63% /
11%
–Priority 1 (51)27% / 4% / 63% / 6%
–Priority 2 (15)20% / 0% / 67% / 13%
–Priority 3 (10) 0% / 10% / 60% / 30%
Results – 4 Calibration & Editing
14ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (38)39% / 24% / 29% /
8%
–Priority 1 (24)38% / 29% / 21% / 13%
–Priority 2 (13)46% / 15% / 38% / 0%
–Priority 3 (1) 0% / 0% / 100% / 0%
Results – 5 Imaging
15ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (91)56% / 19% / 24% /
1%
–Priority 1 (45)73% / 9% / 16% / 2%
–Priority 2 (29)45% / 28% / 28% / 0%
–Priority 3 (17)29% / 29% / 41% / 0%
Results – 6 Data Analysis
16ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (56)32% / 13% / 46% /
9%
–Priority 1 (36)39% / 11% / 50% / 0%
–Priority 2 (11)27% / 9% / 45% / 0%
–Priority 3 (9)11% / 22% / 33% / 33%
Results – 7 Visualization
17ALMA Computing IDR – December 2002, Garching
•Breakdown A : A/E : I/N : U
–All (25)32% / 0% / 64% /
4%
–Priority 1 (8)63% / 0% / 38% / 0%
–Priority 2 (12)17% / 0% / 75% / 8%
–Priority 3 (5)20% / 0% / 80% / 0%
Results – 8 Special Features
18ALMA Computing IDR – December 2002, Garching
Audit Summary
• Requirements and Sub(sub)requirements “equal”:– 61% of all req., and 67% of Priority 1 grade A or A/E– High severity defects (I or N) 6% of all or 9% of Priority 1– 29% of Priority 1 requirements graded I or N – target – 60% of all requirements classified Priority 1 – should fix this!
• Problem Areas:– Calibration & Editing (63% I or N for Priority 1)– Visualization (50% I or N for Priority 1)– Imaging (needs ALMA input and algorithm development)– Interface (performance and look-and-feel deemed
inadequate)
• Cost to complete (Kumar) ~ 26 FTE?
19ALMA Computing IDR – December 2002, Garching
Next – Benchmarking
• Goals:– Quantify AIPS++ performance on ALMA sized
representative datasets– Compare with other packages– Locate problem areas in package– Basis for assay and regression testing
• Test Datasets– Representative of ALMA data (e.g. size)– Real and simulated data– Should cover major modes– SSR: define needed sets as soon as possible
20ALMA Computing IDR – December 2002, Garching
Procedure
• Identify test datasets– SSR defined or provided (e.g. IRAM, BIMA, simulated)– AIPS++ provided (e.g. simulated)
• Build and run scripts– AIPS++ provide scripts (Rusk, Jan 2003)– SSR involvement (new SSR hire?)– IRAM PdB Phase II and III?
• Migrate into assay module(s)– Build into alma package– Use benchmarking tools
• Compare versus other packages– SSR led, with AIPS++ input– Must compare “apples” with “apples”
21ALMA Computing IDR – December 2002, Garching
Outcome
• Identify problem areas – Determine cause of problem
• Augmentation or change of technology required
• Algorithm issue
• Size of problem issue (e.g. pure flops)
• Effectiveness of this step depends on how carefully the benchmarking was done!
– Profiling of code– Fix where necessary (cost, fit into development plan)– Priorities for development
• Build benchmarking process into auditing and development cycles
22ALMA Computing IDR – December 2002, Garching
Example – Hot Topics
• Imaging performance– Comparisons versus MIRIAD, GILDAS
• Gridding procedure (e.g. frequency independent)
– Comparison with AIPS• Joint Stokes deconvolution
• Interface performance and presentation– Interface speed
• Event rate, a glish issue? Change technology?
– GUI look and feel• Development issues (need GUI expert? User desires?)
• Technology choice (e.g. is Python our savior?)
• Measurement sets and fillers
23ALMA Computing IDR – December 2002, Garching
Upcoming Deadlines
• AIPS++ Technical Review– Will address many of these tough questions– Tentatively scheduled for late Jan 2003 or Feb 2003
• In time for PDR
– Need to have some VLA benchmarks in place for this• Will be significant AIPS++ and NAUG work in this area
– Would like to have some first ALMA benchmarks also• ALMA is AIPS++ top customer! (Rusk begin Jan 2003)
• Timescales– Need to have some info by PDR for March/April 2003– Will have Technical Review in hand, could scale from this if
necessary– Who will work on this? This will determine delivery date…
24ALMA Computing IDR – December 2002, Garching
AIPS++ Reorganization
• New NRAO Director – Fred Lo– Critical reviews of all NRAO projects
• New roles in project– Joe McMullin (Project Manager)– Steve Myers (Project Scientist)– Kumar Golap (deputy Project Manager)– George Moellenbrock (Operations Manager)\
• ALMA Subsystem interim leads– Tim Cornwell (Pipeline) Lindsey Davis– Kumar Golap (Offline)
• Upcoming reviews– DM Review (late Jan 2003)– Technical Review (late Jan 2003 or Feb 2003)
25ALMA Computing IDR – December 2002, Garching
Other AIPS++ Developments
• More User input into AIPS++– NRAO AIPS++ User Group (NAUG)
• Auditing and testing, subsystem scientists
– VLA Audit, testing and benchmarking (2003 Q1)– NRAO-wide requirements, audit (EVLA,GBT)
• Based on ALMA with some changes
– Viewer focus group (May 2002)– User Interface focus group (Jan 2003)
• More ALMA input into AIPS++– ALMA is high-profile customer for AIPS++!– ALMA has substantial influence on AIPS++ development
• e.g. through Project Scientist
• Through SSR and subsystem requirements