< Previous6-2 Safety Management Manual (SMM) g) identify connections or “correlations” between or among various factors; h) test assumptions; and i) develop predictive modelling capabilities. 6.1.7 Organizations should include a range of appropriate information sources in their safety analysis, not just “safety data”. Examples of useful additions to the data set include: weather, terrain, traffic, demographics, geography, etc. Having access to and exploiting a broader range of data sources will ensure analysts and safety decision makers are aware of the bigger picture, within which the safety decisions are made. 6.1.8 States, in particular, should be especially interested in information which identifies safety trends and hazards that cut across the aviation system. 6.2 TYPES OF ANALYSIS Analysis of safety data and safety information also allows decision makers to compare information to other groups (i.e. a control or comparison group) to help draw conclusions from the safety data. Common approaches include descriptive analysis (describing), inferential analysis (inferring) and predictive analysis (predicting), as illustrated in Figure 14. Figure 14. Common statistical analysis types 6.2.1 Descriptive analysis 6.2.1.1 Descriptive statistics are used to describe or summarize data in ways that are meaningful and useful. They help describe, show or summarize data in ways so patterns can emerge from the data and help to clearly define case studies, opportunities and challenges. Descriptive techniques provide information about the data; however, they do not allow users to make conclusions beyond the analysed data or to reach conclusions regarding any hypotheses about the data. They are a way to describe the data. 6.2.1.2 Descriptive statistics are helpful because if we simply presented the raw data, particularly in large quantities, it would be hard to visualize what the data is showing us. Descriptive statistics enables users to present and see the data in a more meaningful way, allowing simpler interpretation of the data. Tools such as tables and matrices, graphs and charts and even maps are examples of tools used for summarizing data. Descriptive statistics include measures of central tendency such as mean (average), median and mode, as well as measures of variability such as range, quartiles, minimum and maximum, frequency distributions, variance and standard deviation (SD). Chapter 6. Safety Analysis 6-3 These summaries may either be the initial basis for describing the data as part of a more extensive statistical analysis or they may be sufficient in and of themselves for a particular investigation. 6.2.2 Inferential analysis Inferential (or inductive) statistics aim to use the data to learn about the larger population the sample of data represents. It is not always convenient or possible to examine each item of an entire population and to have access to a whole population. Inferential statistics are techniques that allow users to available data to make generalizations, inferences and conclusions about the population from which the samples were taken to describe trends. These include methods for estimating parameters, testing of statistical hypotheses, comparing the average performance of two groups on the same measure to identify differences or similarities, and identifying possible correlations and relationships among variables. 6.2.3 Predictive analysis Other types of analyses include probability or predictive analyses that extract information from historical and current data and use it to predict trends and behaviour patterns. The patterns found in the data help identify emerging risks and opportunities. Often the unknown event of interest is in the future, but predictive analysis can be applied to any type of unknown in the past, present or future. The core of predictive analysis relies on capturing relationships between variables from past occurrences and exploiting them to predict the unknown outcome. Some systems allow users to model different scenarios of risks or opportunities with different outcomes. This enables decision makers to assess the decisions they can make in the face of different unknown circumstances and to evaluate how they can effectively allocate limited resources to areas where the highest risks or best opportunities exist. 6.2.4 Combined analysis 6.2.4.1 Various types of statistical analyses are interconnected and often conducted together. For example, an inferential technique may be the main tool used to draw conclusions regarding a set of data, but descriptive statistics are also usually used and presented. Also, outputs of inferential statistics are often used as the basis for predictive analysis. 6.2.4.2 Analytical techniques can be applied to safety analysis in order to: a) identify the causes and contributing factors related to hazards and elements which are detrimental to the continuous improvement of aviation safety; b) examine areas for improvement and increase in the effectiveness of safety controls; and c) support ongoing monitoring of safety performance and trends. 6-4 Safety Management Manual (SMM) 6.3 REPORTING OF ANALYSIS RESULTS 6.3.1 Results of safety data analysis can highlight areas of high safety risk and assist decision makers and managers to: a) take immediate corrective actions; b) implement safety risk-based surveillance; c) define or refine safety policy or safety objectives; d) define or refine SPIs; e) define or refine SPTs; f) set SPI triggers; g) promote safety; and h) conduct further safety risk assessment. Figure 15. D3M integration with safety management Chapter 6. Safety Analysis 6-5 6.3.2 The results of a safety analysis should be made available to aviation safety stakeholders in a way that can be easily understood. The results should be presented with the audience in mind, such as organizational decision makers, external service providers, CAAs and other States. Safety analysis results may be presented several ways, the following are some examples: • Imminent safety alerts: for the transmittal to other States or service providers of safety hazards with potential outcomes that could be catastrophic which require immediate actions. • Safety analysis reports: usually present quantitative and qualitative information with a clear description of the degree and source of the uncertainty involved in the analysis findings. These reports may also include relevant safety recommendations. • Safety conferences: for States and service providers to share safety information and safety analysis results that can promote collaborative initiatives. 6.3.3 It is helpful to translate recommendations into action plans, decisions and priorities that decision makers in the organization need to consider and, if possible, to outline who needs to do what about the analysis results and by when. 6.3.4 Visualization tools such as charts, graphs, images and dashboards are simple yet effective means of presenting results of data analysis. Several examples of visual data analysis reports can be found on ICAO’s integrated Safety Trend Analysis and Reporting System (iSTARS) at https://icao.int/safety/iSTARS. 6.3.5 Safety dashboards 6.3.5.1 The safety performance of the organization should be demonstrable and should clearly indicate to all interested parties that safety is being managed effectively. One approach to demonstrating this is through a “safety dashboard”, which is a visual representation that enables senior executives, managers, and safety professionals a quick and easy way to view the organization’s safety performance. 6.3.5.2 In addition to a real time display of the organization’s SPIs and SPTs, dashboards may also include information relating to category, cause and severity of specific hazards. Ideally, the information presented on the dashboard can be customized to display the information required to support the decision-making at varying levels of the organization. The use of triggers is useful for providing basic visuals to highlight if there are any issues to be addressed for a specific indicator. Analysts and decision makers will want the ability to configure the dashboard to display their top indicators as well as a feature which allows them to delve deeper into the metrics. 6.3.5.3 Collecting and analysing the data required for effective management and decision-making is an ongoing process. The results of data analysis may reveal that more and better data need to be collected and analysed in support of the actions and decisions that the organization needs to take. Figure 15 shows how reporting of analysis results may determine further requirements for data to be collected. 6.4 SAFETY INFORMATION SHARING AND EXCHANGE Safety can be further improved when safety information is shared or exchanged. It ensures a consistent, data-driven and transparent response to safety concerns at the global, State and organizational levels. Sharing of safety information refers to giving, while exchange refers to giving and receiving in return. 6.4.1 Sharing within the State 6.4.1.1 States should promote the establishment of safety information sharing or exchange networks among users of the aviation system, and facilitate the sharing and exchange of safety information, unless their 6-6 Safety Management Manual (SMM) national law provides otherwise. Safety promotion guidance for States and service providers is provided in Chapters 8 and 9, respectively. 6.4.1.2 The level of protection and conditions on which safety information will be shared or exchanged between State authorities and service providers need to be consistent with national laws. Further information on the protection of safety data and safety information can be found in Chapter 7. 6.4.2 Sharing between States States should share safety information with other States as soon as possible if, in the analysis of the information contained in its SDCPS, safety matters that may be of interest to another State are identified. States are also encouraged to share safety information within their RASG. Prior to the sharing of safety information, States should ensure that the level of protection and conditions on which safety information will be shared is in line with Annex 19, Appendix 3. Detailed guidance is available in Chapter 7. 6.5 DATA-DRIVEN DECISION-MAKING 6.5.1 The primary purpose of safety analysis and safety reporting is to present a picture of the safety situation to decision makers which will empower them to make decisions based on the data presented. This is known as data-driven decision-making (also referred to as DDDM or D3M), a process-driven approach to decision-making. 6.5.2 Many aviation occurrences have resulted, at least in part, from poor management decisions, which can result in wasted money, labour and resources. The goal of safety decision-makers is, in the short term, to minimize poor outcomes and achieve effective results, and in the long term, contribute to the achievement of the organization’s safety objectives. 6.5.3 Good decision-making is not easy. Decisions are often made without being able to consider all the relevant factors. Decision-makers are also subject to bias that, whether consciously or not, affects decisions made. 6.5.4 The intent of D3M is not necessarily to make the “perfect” or ideal decision, but rather to make a good decision that achieves the short-term objective (about which the actual decision is being made) and works towards satisfying the longer-term objective (improved organizational safety performance). Good decisions meet the following criteria: a) Transparent: the aviation community should know all the factors that influence a decision, including the process used to arrive at the decision. b) Accountable: the decision-maker “owns” the decision and the associated outcomes. Clarity and transparency also bring about accountability – it’s not easy to hide behind a decision where roles and responsibilities are defined in detail and where expectations associated with the new decision are clearly outlined. c) Fair and objective: the decision-maker is not influenced by considerations that are not relevant (e.g. monetary gain or personal relationships). d) Justifiable and defensible: the decision can be shown to be reasonable given the inputs to the decision and the process followed. e) Reproducible: given the same information that was available to the decision-maker, and using the same process, another person would arrive at the same decision. f) Executable: the decision is clear enough and that clarity minimizes uncertainty. Chapter 6. Safety Analysis 6-7 g) Pragmatic: humans are creatures of emotion, which means eliminating emotion from a decision isn't feasible. However, what can be eliminated are self-serving emotional biases. A healthy question to ask in the face of difficult decisions is: whom does the decision serve? 6.5.5 Advantages of data-driven decision-making 6.5.5.1 D3M enables decision makers to focus on desired safety outcomes which align with the safety policy and objectives, and address various aspects related to change management, safety risk assessments, etc. D3M can assist with decisions related with: a) changes that can be expected in statutory and regulatory requirements, emerging technologies or resources which may affect the organization; b) potential changes in the needs and expectations of the aviation community and interested parties; c) various priorities that need to be established and managed (e.g. strategic, operational, resources); d) new skills, competencies, tools and even change management processes that may be needed to implement new decision(s); e) risks that need to be assessed, managed or minimized; f) existing services, products and processes that currently provide the most value for interested parties; and g) evolving demands for new services, products and processes. 6.5.5.2 A structured approach such as D3M drives decision makers to decisions that are aligned with what the safety data is indicating. This requires trust in the safety performance management framework, if there is confidence in the SDCPS, there will be trust in any decisions derived from them. 6.5.6 Common challenges with data-driven decision-making 6.5.6.1 Implementing processes for data collection and analysis takes time and money, as well as expertise and skills that may not be readily available to the organization. The appropriate amount of time and resources vested into the decision-making process needs to be carefully considered. Factors to consider include the amount of money involved in the decision, the extent of the influence of the decision and the decision’s safety permanence. If the organization does not understand what is involved, then the D3M process may become a source of frustration for safety decision makers, causing them to undermine or abandon the process. Like SSP, SMS, D3M and safety performance management requires a commitment to build and sustain the structures and skills necessary to maximize the opportunities presented by D3M. 6.5.6.2 It is harder to build trust in data than it is to trust an expert’s input and opinion. Adopting the D3M approach requires a shift in the culture and mind-set of the organization where decisions are based upon reliable SPIs and the results of other safety data analysis. 6-8 Safety Management Manual (SMM) 6.5.6.3 In some cases the decision-making process may become bogged down in an attempt to find the “best possible” solution, also known as “analysis paralysis”. Strategies that can be used to avoid this include: a) setting a deadline; b) having a well-defined scope and objective; and c) not aiming for a “perfect” decision or solution the first time, but rather coming up with a “suitable” and “practical” decision and improving further decisions. 6.5.7 Data-driven decision-making process 6.5.7.1 The D3M process can be a critical tool that increases the value and effectiveness of the SSP and SMS. Effective safety management depends on making defendable and informed decisions. In turn, effective D3M relies on clearly defined safety data and information requirements, standards, collection methods, data management, analysis and sharing, all of which are components of a D3M process. Figure 16 illustrates shows the D3M process. Figure 16. Data-driven decision-making phases Step 1 – Defining the problem or objective 6.5.7.2 The first step in planning and establishing the D3M process is to define the problem that needs to be solved or the safety objective that needs to be achieved. What is the question that needs to be answered? What decision do the safety decision makers need to take? How will it align with the more strategic organizational objectives? In the process of defining the problem statement, decision makers should ask themselves the following questions: • Does the collection and analysis of data support and relate to the organization’s safety objectives or safety targets? • Is the required data available? Or can it be obtained in a reasonable manner? • Is it practical and feasible to collect and analyse the data? • Are the required resources (people, equipment, software, funds) available? Chapter 6. Safety Analysis 6-9 6.5.7.3 In the safety management context, the main problem statements within the organization are related to evaluating and selecting safety priorities – in alignment with the safety objectives – and establishing measures for safety risk mitigation. Step 2 – Access to data to support the decision-making 6.5.7.4 The next step is to identify what data is needed to answer the problem (taking into account the provisions on information protection). No data is any more valuable than other data. Focus should be on whether the available data is appropriate to help answer and resolve the problem. If the data required is available, proceed to step 4. If the right data is not available, the organization will need to collect, store, analyse and present new safety data and safety information in meaningful ways. Step 3 – Request data to support the decision-making 6.5.7.5 If the data isn’t already available, the organization needs to find ways of collecting it. This may mean establishing another SPI and perhaps aligned SPTs. Establishing additional indicators can come at a cost. Once the cost is known, the organization should estimate if the benefits outweigh those costs. The focus should primarily be on identifying, monitoring and measuring safety data that is needed to make effective data-driven safety decisions. If the costs outweigh the benefits, consider alternative data resources and/or indicators. 6.5.7.6 In the planning phase of the D3M process, the organization needs to define what it wants to achieve by establishing the SPTs and SPIs, and analysing the data. Why does the organization need to address the identified problem? What is a reasonable target? And how and where will safety decision makers use the results of data collection and analysis? Having a clear understanding of why the organization needs to collect, analyse, share and exchange safety data and information is fundamental for any SDCPS. 6.5.7.7 The following elements combine to enable an organization to identify trends, make informed decisions, evaluate the safety performance in relation to defined objectives, assess risks or fulfil its requirements: a) safety performance management - as the safety data and safety information governance framework; b) SDCPS - as the safety data collection and processing functionality; and c) D3M as a dependable decision-making process. Step 4 – Interpret results of data analysis and make data-driven decision 6.5.7.8 The data gathered needs to be presented to the decision makers at the right time and in meaningful ways. The appropriateness and size of the data sets, the sophistication of the analytics and the skills of the data analysts will only be effective if the data is presented when needed and in formats that make it easy to for decision makers to comprehend. The insights gained from the data should inform decision-making, and ultimately, improve safety performance. 6.5.7.9 There are many decision-making models available. Using an agreed and standardized approach will maximize consistency and effectiveness of the organization’s data-driven decisions, most include the following steps: a) assemble a team/group with the necessary skills and experience (e.g. safety action group (SAG)); b) clearly define the safety problem or objective and the context; c) review the organization’s SPTs and safety objectives to ensure continued alignment; d) review and interpret the safety data to understand what it is indicating; 6-10 Safety Management Manual (SMM) e) consider and analyse the viable alternatives; f) consider the risk of feasible actions (or inactions); g) gain consensus among the decision-making group; h) commit to the data-driven decision and act on the decisions (turning data into action); and i) monitor and evaluate the outcomes. Step 5 – Communicate the decision 6.5.7.10 For the safety decision to be effective, it needs to be communicated to stakeholders, these include: a) staff required to enact the necessary actions; b) person who reported the situation (if required); c) all personnel, to ensure they are kept informed of safety improvements (safety promotion, States refer to chapter 8; service providers refer to chapter 9); and d) organizational knowledge managers to ensure the safety decision is incorporated into the learning of the organization. 6.5.7.11 For more information on safety communications, refer to Section 8.6 for States and Section 9.6 for service providers. ______________________ 7-1 Chapter 7 PROTECTION OF SAFETY DATA, SAFETY INFORMATION AND RELATED SOURCES 7.1 OBJECTIVES AND CONTENT 7.1.1 This chapter describes the basic principles governing the protection of safety data and safety information captured by or derived from safety reporting systems, as well as the sources of such data and information.1 It also provides guidance and advice on the implementation of these principles by State aviation regulatory authorities, service providers, legislators, lawyers, prosecutors, judicial officers and other competent authorities with responsibility for making decisions about the use and protection of safety data, safety information and their related sources. This chapter may be of use to any other persons seeking access to or the disclosure of safety data or safety information. 7.1.2 The chapter includes the following topics: a) fundamental principles; b) scope and level of protection; c) principles of protection; d) principles of exception; e) public disclosure; f) protection of recorded data; and g) safety information exchange and sharing. 7.2 FUNDAMENTAL PRINCIPLES 7.2.1 The objective of protecting safety data, safety information and their related sources is to ensure their continued availability, with a view to using it for maintaining or improving aviation safety, while encouraging individuals and organizations to report safety data and safety information. In this context, the importance of implementing protections cannot be overstated. The protections are not intended to relieve sources of their safety related obligations or interfere with the proper administration of justice. 7.2.2 Aviation safety is not the sole responsibility of States or service providers. It is a shared responsibility to which all stakeholders should contribute by, among other things, providing relevant data and information through safety reports. 7.2.3 While data and information can come from various sources, reporting of safety data and safety information by individuals and organizations in the aviation system is fundamental to safety management. Effective safety reporting systems help to ensure that people are and remain willing to report their errors and experiences, so that States and service providers have access to relevant data and information that is necessary to address existing and potential safety deficiencies and hazards. This assurance is provided by creating an environment in which people 1 As per Annex 19, sources of safety data and safety information include both individuals and organizations. Next >