Abstract. Performance of systems engineering tasks requires appropriate skills and methods often surmised from marketing and advertisement pieces. What is the correlation between the ‘required’ skill set and the individual’s experience? We observe that skills seem to vest with time and that the systems engineer likewise becomes more proficient in those traits that improve project performance. Can these proficiencies be tied to time or years of experience? We apply a time-based exponential learning curve to data collected from three sources: classified job advertisements, surveys of practicing systems engineers, and pertinent literature. From a triangulation methodology we investigate the relationship between the number of years of experience and the breadth and depth of knowledge expected in the workplace. We then relate the contribution to work made by systems engineers to the number of their years of their experience. For the processes of requirement’s analysis and project management skills, systems engineers with 15 years of experience are twice as effective as those with 10 years of experience (with a 3.6 year error).
Introduction
Who should be hired to fill a job that requires systems engineering skills? Should the individual have 3-5, 8-10, or 15+ years experience? As Systems Engineers progress through their career they acquire and improve the skills and traits required by engineering solutions. Hiring practices, job assignments, acquisition of skills, and improvements in proficiencies are seemingly correlated to this progression. The authors outline an employment lifecycle for Systems Engineers which reflect characteristics that can occur within phases of their work, e.g., by years of experience, job title, annual salary (Cosgrove 2006). As defined by the essential skills and proficiencies found in the workplace, the systems engineer can be seen to ‘progress’ from a noviciate to a senior-level.
Acquiring skills can be thought of as an economic transaction, whereby an employer pays an employee to learn. There can be two direct consequences of that learning: (1) the employer and the employee benefit (internalities), and (2) the customer benefits
(externalities). Since the customer is not a direct party to the learning, but yet benefits, an externality is created. The learning of Systems Engineering skills can create internalities and externalities that can then be defined in terms of a profession and a discipline that apply holism to understand and solve problems. Both internalities and externalities need to be observed and measured. We posit that the maturation of a Systems Engineers seems to be the consequence of a minimum of training, education, internalization and reflection, experience, and awareness. But, since the artifacts and their proportional contributions are as yet undetermined, this research frames an approach to identifying the artifacts that seem to play a role in determining a maturity curve for systems engineers.
If the end state for a project is to satisfy stakeholder requirements within the allocated time and budget, then there would seem to be a set of traits that must be acquired or innately bred. While the data used in this research was particularly focused on Systems Engineering, we view our methodology, analysis, and measures as broadly applicable to disparate occupations and various avocations.
We applied a qualitative methodology by (i.e., surveying practicing Systems Engineers, referencing current literature for defining their traits, and reviewing recruitment advertisements placed on the internet, and in magazines and newspapers. Data gathering was determined to be complete once a noticeable pattern evolved in each of the three data sets. This was consanguineous to performing a meta-analysis, less the presence of any statistical comparison (de Souza 2008). The check for independence between the data sets was non-exhaustive, but the coupling was assumed to be weak for the purposes of this research. Future work will test this assumption. Two variables were analyzed – years of experience and annual income. Scaling parameters were determined and plotted on a fuzzy logic scale based on triangulation comparisons between the three sets of data (Silverman 2005). Triangulation is a sampling strategy used to validate data (Denzin and Lincoln 1994). It is the focus of triangulation to determine the efficacy of a data source with regards to the dependencies of the intended consequences. Specifically, we applied the triangulation method of Frank, Frampton, and Di Carlo, 2007 that advocates establishing categories of repeated elements. Through corroboration of these elements by multiple occurrences, the data collection can be declared “steady state” and then terminated. The three sources were analyzed to determine these categories of common elements. The error of the effect size normally associated with most meta-analyses was omitted, as no statistical analysis of the data was required.
The Fuzzy Logic model defined by Zedah in 1965, was applied to the learning curve to compare the traits of System Engineers and correlated to their years of experience, then later with comparison to annual salaries. The fuzzy logic scale offered a convenient and substantiated method of describing the patterns of data. A fuzzy set is class of objects with a continuum of grades with varying degrees of membership (Pedrycz and Gomide 2007). By combining the categories, grades (scaling factors), and associations (membership), a series of fuzzy logic scales were developed and used to accommodate and plot the three data sets. Over seven hundred traits were gathered and correlated from the three data sources. A Fuzzy Logic Scale was created to summarize the traits desired by employers. The scale was designed in accordance with the number of years of
experience and was scaled from 1 to 10, with 1 being the least experienced and 10 being the most experienced.
The fuzzy scale was designed to represent high numbers of correlations of traits between data sets with a high marker and a low marker for small numbers of correlations. Traits were binned according to membership relations on a linearly increasing scale for years of experience. Figures 1, 2, and 3 summarize the data sets for surveys, classified advertisements, and literature review, respectively. An attempt was made to eliminate potential bias in either the logic of the fuzzy scale or in the binning of data by comparing the traits obtained from the literature review and the survey.
A key observation of the amalgamated data sets was the grouping of certain traits at various levels of systems engineering experience in the maturity cycle. To further analyze this pattern, learning curves were plotted and the relationship determined between the power and coefficient of the curves relative to a zero starting point. This zero-point normalization assumes no prior knowledge of the traits as so distinguished. We note two limitations in this paper and presentation is the restriction to literature only gathered in the past forty years. The early years of writings on Systems Engineering are quite productive and competent with requisite skills and proficiencies. A broader, more comprehensive perspective can be found in de Souza’s Master’s thesis (de Souza 2008). Secondly, the data was collected only to include up to 15-years experience. For some of us, such a limited scope denies the true worth of longevity and maturity.
Figure 1. Fuzzy Scale of Amalgamated List (Survey)
Figure 2. Fuzzy Scale of Amalgamated List (Classified Ads)
The requirements of a Systems Engineer as printed in the classified ads were chosen to allow the data to be ‘naturally occurring’ (Silverman, 2005) without interventions or special setup of interviews or specific environments. However, it was not assumed the originator of the advertisement was well versed and could articulate and present their needs and requirements. Accordingly, there is error in crafting the desired Systems Engineer requirements due to differences in understanding between the hiring manager and the actual needs, the number of words or appropriate symbols that can be compressed into decipherable text, in addition to the restrictions (e.g., competitive disclosures) that need be placed on what can be stated about the traits desired. These ads were confined to those available on the Internet. Forty classified ads were used.
Consequently, the newly hired Systems Engineer may possibly not have the requisite full range of desired traits.
Figure 3. Fuzzy Scale of Amalgamated List (Literature Review)
To review and analyze the literature a Meta-Visualizing Tool was developed (Figure 4) to show the relationship and relevance between the major sources of literature and the traits of systems engineers.
Table 1. Areas of Concern for Classified Ads
Area of Concern Details Implications
1 Need of Organization The skills of the potential hires may be misunderstood
Inaccurate requirements specified in ads
2 Interpretation of Needs / Requirements
The requirements statements may not cover the needs
Inaccurate requirements specified in ads
3 Limiting Requirements
Limited ad space results in only the basic requirements
Inaccurate requirements specified in ads
4 Ability matching to Classified Ads
The applicants may not match their abilities to the real needs
May not be the most suitable candidate hired
Figure 4. Meta-Visualizing Tool for Literature Review
Hall 1962
Thus a compilation of articles and books that covered the traits of successful Systems Engineers spanned 40 years. The ability to think systemically was seen essential (Baird 1971); Frank Zwikae, and Boasson 1997); Frank 2000); Augustine 2000); Frank and Elata 2005) and Davidz 2005). Further insights resulted from investigations into the roles of Systems Engineer (Sheard 1996), a review of specialized Systems Engineers, in Space Systems (Moore 2000), and in comparison studies with Systems and IT Architects (Frank, Frampton and Di Carlo 2007). A triangular comparison (Figure 5) was performed on traits gleaned from reviewing published literature, those displayed in online classified ads, and responses from surveys completed by practicing Systems Engineers.
Attention then focused on acquisition stages for to acquire these traits. In the process of understanding the development of systems engineering skills, the essential elements of systems engineering were compiled. We chronicled the range and attributes of systems engineering over forty years beginning in the 1960s (Figures 6a (1960s to 1980s), 6b (1990s), and 6c (2000s), as the discipline matured and explored its boundaries.
Class of Systems
Compatibility
Current Engineering
Development Planning
Development Studies
Exploratory Planning
Integrated Approach
Integration Process
Iterative Process
Optimization through balancing objectives
Program Planning
Recognition of System Objectives
System Approach
System Considerations
System Studies
Testing
Application of Engineering Efforts
Application of Scientific Efforts
Better Design of man-organized and man-made systems
Comprehending Complex Systems
Integration Process
Interdisciplinary Approach
Maintenance Analysis
Operation Analysis
Scientific Methods
System Functional Analysis
System Reliability Analysis
Systems Approach
Task Analysis
Use of an Iterative Process
Analysis
Broad Technical Plan
Business
Designing of Systems
Formulate Economic Objectives
Formulates Operational Performance
Modeling
Research
Simulation
Years
Des
crip
tive
Phra
ses
of
Syst
ems
Engi
neer
ing
1960s 1970s 1980s
Figure 6(a). Attributes of Systems Engineering (1960s to 1980s)
Figure 6(b). Attributes of Systems Engineering (1990s)
Academic Discipline
Analysis
Analytical
Balance of all System elements
Balanced set of system people
Balanced set of system product
Collaborative Approach
Combination of methods & tools thru use of suitable methodological process
Comprehensive Approach
Cost Assessment
Creation of Alternatives
Creation of Systems
Decision Making
Designing of Systems
Development of information
Document Requirements in Specifications
Encompassing the entire technical effort
Planning
Policy Making
Post Implementation Assessment
Process Solutions
Production of Systems
Professional Discipline
Qualitative formulation
Quantification of System Goals
Quantification of Systems
Quantitative formulation
Requirements Satisfaction Process
Resource Deployment
Resource Management
Risk Assessment
Robust Approach
Satisfaction of Needs
Satisfy Customer Needs
Scientific effort
Years
Desc
riptiv
e Phr
ases
of
Syst
ems E
ngin
eerin
g
1990s
Evolve and verify
For Large Scale and Scope
Formulation
Formulation, Analysis & Interpretation of proposed
Identification of System Goals
Information and Knowledge Organization & Management
Integration of the '-ilities'
Integration Process
Intellectual Discipline
Interdisciplinary Approach
Interpretation
Iterative Process
Life-cycle View
Maintenance of Systems
Management of System Configuration
Management Process
Management Technology
Needs Satisfaction
Solving complex System Problems
System Analysis
System Definition
System Management Procedure
System Performance Parameters Identification
System Synthesis
Technical Effort related to Life cycle of product / system
Technical Process
Testing
Transform Operational Need to System design
Translation to Work Breakdown structures
Verification of design
Verification Process
Engineering effort
Evaluation
Years
Desc
riptiv
e Phr
ases
of
Syst
ems E
ngin
eerin
g
2000s
Balance of all System elements
Complex Systems
Conformance to Schedule
Cost Effectiveness Methods
Creating and Executing an Interdisciplinary Process
Customer and Stakeholders consideration
Design Synthesis
Engineering Discipline
Engineering of Complex Systems
Evaluation
Functional Analysis
Initial definition of system
Interdisciplinary Approach
Iterative Process
Life-cycle View
Needs Satisfaction
Non-Sequential Process
Non-Traditional Engineering discipline
Parallel Process
Requirements Satisfaction Process
Resource Allocation
Satisfaction of Needs
System Analysis
System as a Whole
System Considerations
System Life Cycle Consideration
Systems Thinking
Team Approach
Testing
To Guide
Top- Down Approach
Total Operation
Verification Process
Figure 6(c). Attributes of Systems Engineering (2000s)
Methodology. Data Gathering involved a series of processes as depicted in Figure 8. The sources of the data, i.e., literature review, surveys and classified ads, were assumed to be representative of the Systems Engineering community. To determine the accuracy and precision of the data, collection, and analysis for the survey, errors were hypothesized, listed, and managed. Some of the inherent survey errors were gleaned from Demming (1994) in Denzin (1989), and then adapted to improve triangulation methodology.
The survey of practicing Systems Engineers allowed the thesis to be initiated in two ways. The surveys allowed the gathering of the traits that make a good Systems Engineer that in turn facilitated an initial categorization from which to acquire traits through classified ads and the literature review. The methodology stipulated that the data acquired be in its natural state of existence, i.e., without intervention or coercion from the surveyors, the surveying instrument, or the enterprise and its framework of operations and activities. The natural state of existence refers to the absence of possible biasness, i.e. skewedness in survey questions, propaganda in classified ads, etc. Hence, acquiring data from the classified ads and from the literature review was deemed to be in the natural state, as is it taken that there are no ‘interference’ in the contained information.
Three sampling strategies were considered: Theoretical Sampling,1 Illustrative Sampling,2 and Triangulation. The aim was to match the sampling strategy with the style and ilk of the Systems 1 Selecting groups or categories to study on the basis of their relevance to the research question and theoretical position
(Mason 2006)
Engineering community. The objective was to identify and match traits that were ideally and pragmatically representative of both reality and expectations.
Triangulation (Denzin and Lincoln 1994) was determined to best determine whether a specific source of data would be robust to changes in methods and local and individual sensitivities to the field. Hence, the combination of multiple methods, empirical materials, perspective and observers in a single study is best understood as a strategy that adds rigor, breadth, and depth to any investigation (Denzin and Lincoln 1994); (Flick 1992). Triangulation can be understood from five definitions – four offered by Denzin in 1978: Data,3 Investigator,4 Theory5 and Methodological6 Triangulation; and a fifth (Janesick 1994): Interdisciplinary7 Triangulation. Figure 8 illustrates triangulation.
Analysis of Data. A‘modified’ meta-analysis methodology was used to analyze the data. Data mining dealt with structured databases of facts (Hearst 2003), while the text mining uncovered trends and patterns in literature and classified advertisements. A ‘modified’ meta-analysis methodology was used to analyze the data. Data mining dealt with structured databases of facts (Hearst 2003), while the text mining uncovered trends and patterns in literature and classified advertisements.
Figure 8. Data Gathering Process
Selection of Sources
Selection of Sources Sampling StrategySampling Strategy Gathering DataGathering Data
• Identification of Sources• Limitations of Sources• Usage of Sources
2 Relationship between sampled contexts and phenomenon and the population of interest is illustrative/ evocative in nature.
This approach seeks only to provide a ‘flavor’ to the population (Mason 2006) 3 Use of a variety of data sources in a study (Denzin and Lincoln 1994; Janesick 1994) 4 Use of several different researchers or evaluators (Denzin and Lincoln 1994; Janesick 1994) 5 Use of multiple perspectives to interpret a single set of data (Denzin and Lincoln, 1994; Janesick 1994) 6 Use of multiple methods to study a single problem (Denzin and Lincoln 1994; Janesick 1994) 7 Consideration of other disciplines to study a single problem (Denzin and Lincoln 1994; Janesick 1994)
Fuzzy Logic. An amalgamation of over 700 traits was completed along with the counts for each trait. A Fuzzy Logic Scale was created to summarize the traits desired by employers. The scale was designed in accordance with the number of years of experience and was scaled from 1 to 10, with 1 being the least experienced and 10 being the most experienced. The original idea behind fuzzy logic, as advanced by Zedah in the mid-1960s, was to initiate and propagate the transition from traditional mathematical modeling in engineering to a new, much more qualitative, ‘rough’ modeling using fuzzy sets and fuzzy methods (Bandemer and Gottwald 1995).
Employers desire all Systems Engineers to possess good oral and written communications skills, coupled with a team player spirit. For values of 3+ on the fuzzy scale of experience, Interpersonal Skills, Analytical Skills, Systems Thinking are sought.
To eliminate bias in the fuzzy logic experiential scale built for classified ads, an amalgamated list of traits obtained from analyzing literature and surveyed was compared to the fuzzy logic experiential scale for classified ads (Figure 9). The top 3-traits listed in Figure 9 are satisfying requirements, cost management, and project management. Following the retention percentages by FitzGerald (2005), we assume that the experts (those with above 15 years of experience) are able to retain 75% of the essential traits listed for a successful project. Experts seem to acquire traits from involvement in many projects. These experts have learned their trade, and are able to contribute all of the 75% that they have retained. Based on the listed traits, initial contributions of a novice Systems Engineer appear to be 15%.
1Low
10High
5Mid
1 year >15 years5-7 years2 - 3 years 10 years
3Mid-Low
7Mid-High
Matrix relationship understanding
Subject Matter Expert in an Engineering / Technical discipline / Trade Studies / Human Systems Integration
Manage Teams / Organization Skills / Standards Familiarization
Oral / Written Communication Skills / Team Player
Product Development Process / Coaching Skills
Quality Management
Technical Skills / Attention to Details
Total Systems Consideration / Production
Verification and validation / Complex Systems Handling / Interdisciplinary Knowledge / Project Management Skills / Specific Domain Knowledge
Coordinating Skills
Modeling / Interface Management
Systems Thinking / Testing Skills
Figure 9. Fuzzy Logic of Traits Correlated with Years of Experience (Classified Ads)
Raccoon (1996) covered fifteen forms of the learning equation. As different sets of traits appear to be acquired over the years of experience of performing Systems Engineering, we adopted the time-phased approach to plot the acquisition of traits. Hence, the exponential learning curve is
adopted. Merging retention (FitzGerald 2005) with essential traits for success, then coupling the scales to which traits are tagged (Figure 9) shows the following relationship (Figure 10).
Figure 10. Relationship between a Systems Engineer Contributions and Experience
y = 0.1171e0.1238x
y = 0.006e0.3219x
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0 2 4 6 8 10 12 14
Con
trib
utio
n to
Pro
ject
Year of Experience
Learning Curve of Traits
Requirements Analysis and Project management Skills
Learning Curve
Cost Management Learning Curve
Conclusion This research applied sociological research methodologies to define a maturity curve for Systems Engineers. By developing a fuzzy scale from which to compare data garnered from literature, recruitment advertisements, and surveys of practicing Systems Engineers, an exponential relationship was applied to contributions to project success relative to the number of years of experience. The results indicate that skills in requirements analysis for Systems Engineers with 15-years of experience are three-fold better than Systems Engineers with 6-years of experience. A summary of traits over the 40-year time frame (1960s to 2000s) is shown in Figure 11. The growth in the number of traits over time is due in part to the acknowledgement that the purview of Systems Engineering has increased, as well as to continued refinement of the work and required skills.
Trai
ts o
f a S
yste
ms
Engi
neer
1960s 1970s 1990s 2000s
Holistic View of Things
System Thinking
Objective
Leader
Team Player
Multi Disciplined
Holistic View of Things
System Thinking
Objective
Leader
Team Player
Multi Disciplined
Good Technical Planning
Technology Awareness
Interdisciplinary Knowledge
Systems Interaction
System Planning
Analysis
Simulation
Systems Engineering
Process
Analysis of Alternatives
Good Engineering Judgment
Requirements Analysis
Good Technical Planning
Technology Awareness
Interdisciplinary Knowledge
Systems Interaction
System Planning
Analysis
Simulation
Systems Engineering
Process
Analysis of Alternatives
Good Engineering Judgment
Requirements Analysis
Requirements Owner
Design (Systems)
Analysis (Systems)
Verification and validation
Logistics/Ops Engineer
Technical Glue for Projects
Customer Interface
Technical Manager
Information Manager
Process Engineer
Coordinator
Requirements Owner
Design (Systems)
Analysis (Systems)
Verification and validation
Logistics/Ops Engineer
Technical Glue for Projects
Customer Interface
Technical Manager
Information Manager
Process Engineer
Coordinator
Abstract Thinking
Analysis of Alternatives
Architectural design
Asking the right questions
Change Manager
Complex Systems Understanding
Concept Design
Concept generation
Controlling
Creative
Critical Thinking
Decision Making Skills
Design (Logical)
Design (Physical)
Design (Systems)
Design Process
Functional analysis
Generalist
Holistic View of Things
Integration
Interdisciplinary Knowledge
Interface Management
Interpersonal Skills
Abstract Thinking
Analysis of Alternatives
Architectural design
Asking the right questions
Change Manager
Complex Systems Understanding
Concept Design
Concept generation
Controlling
Creative
Critical Thinking
Decision Making Skills
Design (Logical)
Design (Physical)
Design (Systems)
Design Process
Functional analysis
Generalist
Holistic View of Things
Integration
Interdisciplinary Knowledge
Interface Management
Interpersonal Skills
Knowing when to Stop
Leader
Learning By Experience
Maintaining Design Integrity
Management Skills
Mentoring Skills
Modeling
Monitoring
Multi Disciplined
Negotiating Skills
Objective
Optimization
Oral Communication
Skills
Organizational Capability
Pattern Recognition
Planning
Knowing when to Stop
Leader
Learning By Experience
Maintaining Design Integrity
Management Skills
Mentoring Skills
Modeling
Monitoring
Multi Disciplined
Negotiating Skills
Objective
Optimization
Oral Communication
Skills
Organizational Capability
Pattern Recognition
Planning
Problem Solving Skills
Project Management
Quality Assurance
Requirements Analysis
Resource Allocation
Risk Analysis
Risk Management
Simulation
Software Engineering
Solutioning
System Architecture
Systems concepts
Systems Engineering Process
Systems Synergy
Systems Thinking
Team Leader
Team Player
Technical Knowledge
Testing
Trade Studies
Understanding of Relationship
Verification and validation
Written Communication Skills
Problem Solving Skills
Project Management
Quality Assurance
Requirements Analysis
Resource Allocation
Risk Analysis
Risk Management
Simulation
Software Engineering
Solutioning
System Architecture
Systems concepts
Systems Engineering Process
Systems Synergy
Systems Thinking
Team Leader
Team Player
Technical Knowledge
Testing
Trade Studies
Understanding of Relationship
Verification and validation
Written Communication Skills
Figure 11. Summary of Traits by Year of Publication (Literature Review)
When combined, the three data sets differentiate the skills expected from various levels of experience. Figure 12 illustrates the traits consistent with expectations from online classified advertisements. The experimental fuzzy logic scale was adapted from Moore’s (2000) definitions of Fat-Short and Thin-Tall Systems to describe the different types of Systems Engineers. Originally, the term was used by Mead and Conway (1979) to describe designers. Fat-Short is a person with a specialized set of technical skills who cannot easily integrate concepts from multiple disciplines. Thin-Tall was a person with a broad technical skill who can easily integrate concepts from multiple disciplines.
TallTallShortShort
ThinThinFatFat
1Low
10High
5Mid
1 year >15 years5-7 years2 - 3 years 10 years
3Mid-Low
7Mid-High
Matrix relationship understanding
Subject Matter Expert in an Engineering / Technical discipline / Trade Studies / Human Systems Integration
Manage Teams / Organization Skills / Standards Familiarization
Oral / Written Communication Skills / Team Player
Product Development Process / Coaching Skills
Quality Management
Technical Skills / Attention to Details
Total Systems Consideration / Production
Verification and validation / Complex Systems Handling / Interdisciplinary Knowledge / Project Management Skills / Specific Domain Knowledge
Coordinating Skills
Modeling / Interface Management
Systems Thinking / Testing Skills
Figure 12. Fuzzy Logic of Traits to Years of Experience (Classified Ads)
References Handbook of qualitative research1994. eds. Norman K. Denzin, Yvonna S. Lincoln. California, USA: Sage Publications Inc.
Andress, Frank J. 1954. The learning curve as a production tool. Harvard Business Review 32, (1) (01//Jan/Feb54): 87-97.
Arnold, E. P. 2006. "Defining, Finding, and Hiring REAL Systems Engineers", proceedings of 16th Annual Symposium of the International Council on Systems Engineering, INCOSE, Orlando FL.
Augustine, Norman R. October 2000. The engineering of systems engineers. IEEE Aerospace & Electronic Systems Magazine (Jubilee Issue).
Baird, Jack A. 1971. How can systems engineers do systems engineering?. Aerospace and electronic systems, IEEE transactions on. Vol. AES-7.
Bandemer, Hans, and Siegfried Gottwald. 1995. Fuzzy sets, fuzzy logic, fuzzy methods with applications. New York: Chichester ; J. Wiley.
Barber, D. E. 1998. An overview of the systems engineering capability model EIA/IS 731. Digital avionics systems conference, 1998. proceedings., 17th DASC. AIAA/IEEE/SAE. Vol. 1.
Bar-Yam, Y. 2003. When systems engineering fails-toward complex systems engineering. Systems, man and cybernetics, 2003. IEEE international conference on. Vol. 2.
Boarder, J. C. 1995. The system engineering process. Engineering management conference, 1995. 'global engineering management: Emerging trends in the Asia Pacific'., proceedings of 1995 IEEE annual international.
Booton, Richard C., and Simon Ramo. 1984. The development of systems engineering. Aerospace and electronic systems, IEEE transactions on. Vol. AES-20.
Brill, James H. 1998. Systems engineering - A retrospective view. Systems Engineering 1, (4): 258-66.
Chen, Pin, and Jennie Clothier. 2003. Advancing systems engineering for systems of systems challenges. Systems Engineering 6, (3).
Chiang, H. Ren. 2003. Expanding the role of systems engineers in the age of escalating complexity. Systems Engineering 6, (2): 116.
Covey, Stephen R. 15 September 1990. The seven habits of highly effective people. 1st edition ed.Free Press.
Dahmann, J. S., and K. J. Baldwin. 2008. Understanding the current state of US defense systems of systems and the implications for systems engineering. Systems conference, 2008 2nd annual IEEE.
Davidz, Heidi L. 2005. Accelerating the development of senior systems engineers - PhD dissertation (MIT).
de Souza, Roy. 2008. Maturity Curve of Systems Engineering, Thesis, Naval Postgraduate School, Monterey.
Deming, W. Edwards. 1944. On errors in surveys. American Sociological Review 9, (4) (08): 359-69.
Denzin, Norman K. 1989. The research act: A theoretical introduction to sociological methods. 3rd Edition ed. New Jersey: Prentice-Hall.
———. 1978. The research act: A theoretical introduction to sociological methods. 2nd Edition ed.McGraw-Hill.
Emes, Michael, Alan Smith, and Douglas Cowper. 2005. Confronting an identity crisis: How to brand systems engineering. Systems Engineering 8, (2): 164-86.
Feldman, Ronen, and James Sanger. December 11, 2006. The text mining handbook: Advanced approaches in analyzing unstructured dataCambridge University Press.
FitzGerald, Tara. 2005. Learning by doing. Business Mexico 15, (7) (07): 32-.
Frank, Moti. 2000. Engineering systems thinking & systems thinking. Systems Engineering 3, (3).
Frank, Moti, Tony Di Carlo, and Keith Frampton. 2007. Characteristics of successful systems engineers, systems architects and IT architects.
Frank, Moti, and David Elata. 2005. Developing the capacity for engineering systems thinking CEST of freshman engineering students. Systems Engineering 8, (2): 187-95.
Frank, Moti, Ofer Zwikael, and Michael Boasson. 2007. Jobs requiring a capacity for engineering systems thinking (CEST): Selection using an interest inventory. Project Management Journal 38, (3) (09): 36-44.
Garfinkel, Harold. 1984. Studies in Ethonomethodology. T.J. Press. Ltd. Padstow
Gelbwaks, N. L. 1967. AFSCM 375-5 as a methodology for system engineering. Systems Science and Cybernetics, IEEE Transactions on 3, (1): 6-10.
Goncalves, D. 2008. Developing systems engineers. Management of engineering & technology, 2008. PICMET 2008. portland international conference on.
Hall, A.D. 1962. A methodology for systems engineering. Princeton, N.J.: D. Van Nostrand Co.
Hart, Christopher. 1998. Doing a literature review: Releasing the social science imagination. 1 edition (March 1, 1999) ed.Sage Publications Ltd.
Hearst, Marti. What is text mining? 2003 [cited 6 August 2008]. Available from http://people.ischool.berkeley.edu/~hearst/text-mining.html
Janesick, Valerie J. 1994. The dance of qualitative research design. 209-219 Sage Publications.
Jansma, P. A. “Trisha”, and Ross M. Jones. 2006. Advancing the practice of systems engineering at JPL. IEEEAC Version 6, paper #1031.
Kaplan, David (Editor). 2004. The sage handbook of quantitative methodology for the social sciences. Thousand Oaks, Ca: Sage, c2004.: Sage Publications, Inc.
Kasser, Joe. 1995. Applying total quality management to systems engineering. Norwood, MA, USA: Artech House.
———. Eight deadly defects in systems engineering and how to fix them. Paper presented at Proceedings of the 17th International Symposium of the INCOSE, San Diego, CA.
———. The hitchins-kasser-massie (HKM) framework for systems engineering. San Diego, CA.
Kasser, Joe, and Angus Massie. 20 Jul 2001. A framework for a systems engineering body of knowledge. Paper presented at 11th International Symposium of the INCOSE, Melbourne, Aust.
Kasser, Joe, and K. Palmer. Reducing and managing complexity by changing the boundaries of the system. Paper presented at The Conference on Systems Engineering Research, Hoboken NJ,.
Kossiakoff, Alexander, and William N. Sweet. 2003. Systems engineering - principles and practice. Hoboken, New Jersey: John Wiley and Sons, Inc.
Langford, Gary O., Raymond Franck, Tom Huynh, and Ira Lewis. 2008. Gap analysis: Rethinking the conceptual foundations. Monterey, California, Naval Postgraduate School.
Lewkowicz, P. E. 1988. Effective systems engineering for very large systems: An overview. Aerospace applications conference, 1988. Digest, 1988 IEEE.
Mead, Carver, and Lynn Conway. 1979. Introduction to VLSI systems. New York, NY: Addison-Wesley Publishing Company.
Mintz, R. B. 1994. Systems engineering fundamentals. Northcon/94 conference record.
Moore, Robert C. 2000. Characteristics of a successful space system engineer. Aerospace and Electronic Systems Magazine, IEEE 15, (3): 21-7.
Naval Center for Cost Analysis (NCCA). Naval center for cost analysis (NCCA). 2008 [cited 01 November 2008]. Available from http://www.ncca.navy.mil/services/inflation.cfm
Pedrycz, Witold, and Fernando Gomide. 2007. Fuzzy systems engineering: Toward human-centric computing. Hoboken, New Jersey and simultaneously in Canada: John Wiley & Sons Inc.
Raccoon, L. B. S. 1996. A learning curve primer for software engineers. ACM SIGSOFT Software Engineering Notes 21, (1 (January)) 77-86.
Rittel, H. W., and M. M. Webber. 1973 Number. Dilemmas in a general theory of planning. Policy Sciences Vol 4, 155-69.
Sheard, Sarah A. 1996. Twelve systems engineering roles. Proceedings of the INCOSE Sixth Annual International Symposium (Boston, Massachusetts, USA).
Silverman, David. 2005. Doing qualitative research: A practical handbook. Second Edition ed. London: Sage Publications Ltd.
Towill, Denis R. 1990. Forecasting learning curves. International Journal of Forecasting 6, (1) (03): 25-38.
———. 1985. Management systems applications of learning curves and progress functions. Engineering Costs & Production Economics 9, (4): 369-83.
Vidale, R. F. 1970. University programs in systems engineering. Systems science and cybernetics, IEEE transactions on. Vol. 6.
Williams, Christine, and Derro, Mary-Ellen. NASA systems engineering behavior study, Oct 2008 [cited 01 December 2008]. Available http://www.nasa.gov/news/reports/NASA_SE_Behavior_Study.html
Zadeh, L. A. 1973. Outline of a new approach to the analysis of complex systems and decision processes. IEEE Trans. Systems, Man and Cybernetics 3, 28.
———. 1965. Fuzzy sets. Information and Control 8, (3) 338.
Biographies Roy de Souza graduated with a Masters of Science in Systems Engineering from the Naval Postgraduate School, Monterey, California. He received his bachelors in Mechanical Engineering from the National University of Singapore. He is currently a Major in the Singapore Armed Forces. INCOSE member. Gary Langford is a lecturer in the Systems Engineering Department at the Naval Postgraduate School in Monterey, California. His research interests include the theory of systems engineering and its application to commercial and military competitiveness. Mr. Langford founded and ran five corporations – one NASDAQ listed. He was a NASA Ames Fellow. He has an A.B. in astronomy from UC Berkeley, and an M.S. in physics, Cal State Hayward. INCOSE member.
