1 – What We Found
Accelerate/Measuring Performance
- Measure capabilities, not maturity
- Measure outcomes, not outputs
- What is a good software performance metric?
- Should be global – teams not pitted against each other
- Outcomes, not outputs – reward things that have been delivered, not hours put in
- Examples:
- Delivery lead time
- Deployment frequency
- Time to restore service
- Change fail rate
- Delivery lead time is a better metric than overall lead time because the ideation/design portion of lead time can be fuzzy (“fuzzy front end”, when do we start the clock?)
- Delivery lead time is generally measured as time between committing code and code running in production
- Deployment frequency is the reciprocal of batch size (in a Lean context)
- Mean time to recover is a better modern metric that time between failures (failures are inevitable in large, complex systems)
Measuring and Changing Culture
- Strategic software ⇒ build it yourself
- Non-strategic software ⇒ use a SaaS provider
- Westrum’s typology of organisational culture:
- Pathological (power-oriented)
- Bureaucratic (rule-oriented)
- Generative (performance-oriented)
- Quality and timeliness of flow of information are the main distinguishing characteristic between the different cultures
Technical Practices
- Five principles of continuous delivery:
- Build quality in
- Work in small batches
- Computers perform repetitive tasks, humans solve problems
- Relentlessly pursue continuous improvement
- Everyone is responsible
- The foundations required to begin continuous delivery:
- Comprehensive configuration management
- Continuous integration
- Continuous testing
- “Improvements in CD brought payoffs in the way work felt” – CD promotes sustainable development by reducing deployment pain and burnout
- CD reduces unplanned/re-work which means higher quality software
- Version controlling system and application code was more positively correlated with software delivery performance than version controlling application code
- Having automated test suites that are maintained by developers (rather than QA or outsourced) is positively correlated with software delivery performance
- Trunk-based development or short-lived (less than a day) positively correlated with software delivery performance
Architecture
- Type of system (greenfield, system of engagement, app, etc.) is not important – characteristics of architecture are important (to software delivery performance)
- Being loosely coupled correlates positively with software delivery performance
- Testability (the ability to test most changes without deploying to an integrated environment) is important
- Deployability (the ability to deploy a service independently of other services) is important
- Bounded contexts and API-driven architectures are strategies to achieve loose coupling
- Productivity (per developer) can be increased as developers are added to teams if the culture, architecture, continuous delivery strategy and leadership are right
- Developers should choose their own tools (the right tools for the job) but debugging methods and inter-service communication should be standardised
- Customer satisfaction with regards to internal tooling is as important as customer satisfaction with regards to externally-facing products
Integrating InfoSec into the Delivery Lifecycle
- DevOps also applies to InfoSec, testing and product management
- In a continuous delivery environment, releases are too frequent to allow security teams to approve every release so it makes more sense so involve security early in the development process (shift left)
Management Practices for Software
- Lean practices applied to software development:
- Limiting work in progress (WIP)
- Creating and maintaining visual displays of key metrics for leaders and engineers
- Using data from application performance and infrastructure monitoring to make business decisions regularly
- Limiting WIP only increases delivery performance if done in conjunction with other lean practices
Product Development
- Adoption of lean product development, i.e.,
- Working in small batches
- Making flow of work visible
- Gathering and implementing customer feedback
- Team experimentation predicted good software delivery performance and vice-versa (a virtuous cycle)
- Ability to take an experimental approach to product management is highly correlated with the technical practices that contribute to continuous delivery
Making Work Sustainable
- Developers or testers not knowing how software is deployed is indicative of deployment pain – barriers that isolate developers from the release process are bad because they help to make developers unaware of the downstream consequences of their changes
- Deployment pain can be avoided by:
- Building systems that can easily be deployed in multiple environments
- Ensure that the state (excluding production data) of production environments can be replicated (from code) easily
- Build intelligence into the application and platform so that the deployment process can be as simple as possible
- Causes of burnout:
- Work overload
- Lack of control
- Insufficient rewards
- Breakdown of community
- Absence of fairness
- Value conflicts
- Fixing the environment (rather than the person) has a higher likelihood of short- and long-term success
- Ways to avoid burnout:
- Fostering a learning – rather than blaming – culture
- Communicating a strong sense of purpose
- Investing in employee development
- Asking for and removing blockers
- Giving developers time, space and resources to experiment and learn
Employee Satisfaction, Identity and Engagement
- The extent to which people identified with their organisation predicted a generative, performance-oriented culture and also predicted organisational performance
Leaders and Managers
- Characteristics of a transformational leader:
- Vision
- Inspirational communication
- Intellectual stimulation
- Supportive leadership
- Personal recognition
- Servant leadership is focusing on followers’ development and performance
- Transformational leadership is getting followers to identify with the organisation and engage in support of organisational objectives
- Things managers can do to facilitate and encourage organisational transformation:
- Establish a dedicated training budget
- Encourage staff to attend technical conferences
- Encourage teams to organise “yak days” for addressing technical debt
- Organise regular internal DevOps mini-conferences
- Hold blameless post-mortems
- Ensure team can choose their own tools
2 — The Research
The Science Behind this Book
- Primary research involves collecting new data
- Secondary research involves working off (and being bounded by) existing data
- A Likert-type scale is one that includes the phrases “strongly agree”, etc. Each option corresponds to a numerical value, so it is an example of quantitive research.
- Types of quantitative analysis (in order of increasing complexity):
- Descriptive
- Exploratory
- Inferential predictive
- Predictive
- Causal
- Mechanistic
- Exploratory analysis gives us correlation
- Pearson correlation measures the strength of a linear relationship between two variables:
- -1 for exact inverse correlation
- 0 for no correlation
- 1 for exact correlation
- Inferential predictive analysis involves deciding a theory and testing that theory using the data to test/proves the theory
- Predictive analysis is used for forecasting
- Causal analysis generally requires randomised studies, e.g. A/B testing
- Mechanistic analysis is rarely seen in business and involves predicting and observing exact changes of state, e.g. an atom losing an electron
- Clustering analysis involves classifying data points into groups by certain characteristics and grouping together
Introduction to Psychometrics
- Latent construct is a way of measuring something that can’t be measured directly, e.g. with organisational culture, ask respondents to choose between three options each for cooperation, responsibilities, encouragement, attitudinal to failure, attitude to novelty, etc.
- Latent constructs make it more difficult for a single bad data source or measure to skew results
- Validity tests check for:
- Discriminant validity: latent constructs that are not supposed to affect an overall measure, are not affecting an overall measure
- Convergent validity: latent constructs that are supposed to to affect an overall measure, are affecting an overall measure
- Internal consistency tests check that questions/statements are read and interpreted similarly by those who take the survey
Why Use a Survey?
- Surveys are difficult to corrupt (in a directed and. coordinated way) by bad actors, whereas system data can be doctored by one person
- Quick way to collect data
- Well-written questions/statements avoids ambiguity/conflation (e.g. of metric names) that might exist when aggregating system data from many disparate sources
The Data for the Project
- Snowball sampling is adding more and more respondents to the sample population using referrals. This is good because it means that new respondents are likely to be similar to the initial sample. Useful for measuring difficult-to-measure populations.