< Previous4-10 Safety Management Manual (SMM) • Encouraging desired behaviour. SPTs can change behaviours and contribute to desired outcomes. This is especially relevant if achievement of the target is linked to organizational rewards; such as management remuneration. SPTs should foster positive organizational and individual behaviours that deliberately result in defensible decisions and safety performance improvement. It is equally important to consider the potential unintended behaviours when selecting SPIs and SPTs. • Choosing valuable measures. It is imperative that useful SPIs are selected, not only ones which are easy to measure. It should be up to the organization to decide what the most useful safety parameters are; those that guide the organization to improve decision-making, safety performance management, and achievement of its safety objectives. • Achieving SPTs. This is a particularly important consideration, and linked to the desired safety behaviours. Achieving the agreed SPTs is not always indicative of safety performance improvement. The organization should distinguish between just meeting SPTs and actual, demonstrable organizational safety performance improvement. It is imperative that the organization consider the context within which the target was achieved, rather than looking at an SPT in isolation. Recognition for overall improvement in safety performance, rather than an individual SPT achievement, will foster desirable organizational behaviours and encourage exchange of safety information that lies at the heart of both SRM and safety assurance. This could also enhance the relationship between the State and the service provider and their willingness to share safety data and ideas. Caveats on setting SPTs 4.3.3.7 It is not always necessary or appropriate to define SPTs as there may be some SPIs that it is better to monitor for trends rather than determine a target. Safety reporting is an example where having a target could either discourage people not to report (if the target is not to exceed a number) or to report trivial matters to meet a target (if the target is to reach a certain number). There may also be SPIs that it is better to define a direction of travel to target continuous safety performance improvement (i.e. to reduce the amount of events) rather than defining an absolute target as these may be difficult to determine. The following should also be considered in deciding appropriate SPTs: a) Drive undesirable behaviours; if managers or organizations are too focussed on achievement of the numbers as an indicator of success they may not achieve the intended improvement in safety performance. b) Operational targets; too much focus on achieving operational targets (such as: on-time departures, reduction in overhead costs, etc.) without a balance of SPTs can lead to “achieving the operational targets” while not necessarily improving safety performance. c) Focus on quantity rather than quality; this can encourage personnel or departments to meet the target but in doing so deliver a poor product or service. d) Cap innovation; although not intended, once a target is met this can lead to a relaxation and that no further improvements are needed and complacency can set in. e) Organizational conflict; targets can create conflict between departments and organizations as they argue over who is responsible rather than focusing on trying to work together. 4.3.4 Getting safety performance measurement right involves deciding how best to measure the achievement of the safety objectives. This will vary from State to State and from service provider to service provider. Organizations should take the time to develop their strategic awareness of what it is that drives safety improvement for their safety objectives. Chapter 4. Safety Performance Management 4-11 4.3.5 SPIs and SPTs can be used in different ways to demonstrate safety performance. It is crucial that organizations tailor, select and apply various measurement tools and approaches depending on their specific circumstances and the nature of what is being measured. For instance, in some cases, organizations could adopt SPIs that all have specific associated SPTs. In another situation, it may be preferable to focus on achieving a positive trend in the SPIs, without specific target values. The package of selected performance metrics will usually employ a combination of these approaches. 4.4 MONITORING SAFETY PERFORMANCE 4.4.1 Once an organization has identified the targets based on the SPIs they believe will deliver the planned outcome, they need to ensure the stakeholders follow through by assigning clear responsibility for delivery. Defining SPTs for each aviation authority, sector and service provider supports the achievement of the ALoSP for the State by assigning clear accountability. 4.4.2 Mechanisms for monitoring and measuring the organization’s safety performance should be established to identify what changes may be needed if the progress made isn't as expected and reinforce the commitment of the organization in meeting the safety objectives. 4.4.3 Baseline safety performance Understanding how the organization plans to progress towards its safety objectives requires that they know where they are, in relation to safety. Once the organization’s safety performance structure (safety objectives, indicators, targets, triggers) has been established and is functioning, it is possible to learn their baseline safety performance through a period of monitoring. Baseline safety performance is the safety performance at the commencement of the safety performance measurement process. The datum point from which progress can be measured. The case of the example used in figures 9 and 10, the baseline safety performance, for that particular safety objective was “100 runway excursions per million movements during the year (2018)”. From this solid basis, accurate and meaningful indications and targets can be recorded. 4.4.4 Refinement of SPIs and SPTs 4.4.4.1 The SPIs and associated SPTs will need to be reviewed to determine if they are providing the information needed to track the progress being made toward the safety objectives and to ensure that the targets are realistic and achievable. 4.4.4.2 Safety performance management is an ongoing activity. Safety risks and/or availability of data change over time. Initial SPIs may be developed using limited resources of safety information. Later, more reporting channels may be established, more safety data may be available and the organizations safety analysis capabilities will likely mature. It may be appropriate for organizations to develop simple (broader) SPIs initially. As they gather more data and safety management capability, they can consider refining the scope of SPIs and SPTs to better align with the desired safety objectives. Small non-complex organizations may elect to refine their SPIs and SPTs and/or select generic (but specific) indicators which apply to most aviation systems. Some examples of generic indicators would be: a) events including structural damage to equipment; b) events indicating circumstances that an accident nearly occurred; c) events in which operational personnel or members of the aviation community were fatally or seriously injured; 4-12 Safety Management Manual (SMM) d) events in which operational personnel became incapacitated or unable to perform their duties safely; e) rate of voluntary occurrence reports; and f) rate of mandatory occurrence reports. 4.4.4.3 Larger more complex organizations may elect to institute a broader and/or deeper range of SPIs and SPTs and to integrate generic indicators such as those listed above with activity-specific ones. A large airport, for example, providing services to major airlines and situated under complex airspace might consider combining some of the generic SPIs with deeper-scope SPIs representing specific aspects of their operation. The monitoring of these may require greater effort but will likely produce superior safety results. There is a clear correlation between the relative complexity of SPIs and SPTs and the scale and complexity of the State or service providers’ operations. This relative complexity should be reflected in the indicator and target set. Those responsible for establishing safety performance management should be conscious of this. 4.4.4.4 The set of SPIs and SPTs selected by an organization should be periodically reviewed to ensure their continued meaningfulness as indications of organizational safety performance. Some reasons to continue, discontinue or change SPIs and SPTs include: a) SPIs continually report the same value (such as zero per cent or 100 per cent), these SPIs are unlikely to provide meaningful input to senior management decision-making; b) duplicate SPIs, SPIs that measure something similar to one another and provide the same assurances; c) the SPT for an SPI implemented to measure the introduction of a programme or targeted improvement has been met; d) another safety concern becomes a higher priority to monitor and measure; e) to gain a better understanding of a particular safety concern by narrowing the specifics of an SPI (i.e. reduce the “noise” to clarify the “signal”); and f) safety objectives have changed to ensure the indicators remain relevant. 4.4.5 Safety triggers 4.4.5.1 A brief perspective on the notions of triggers is relevant to assist in their eventual role within the context of the management of safety performance by an organization. 4.4.5.2 A trigger is an established level or criteria value that serves to trigger (start) an evaluation, decision, adjustment or remedial action related to the particular indicator. One method for setting out-of-limits trigger criteria for SPTs is the use of the population standard deviation (STDEVP) principle. This method derives the standard deviation (SD) value based on the preceding historical data points of a given safety indicator. The SD value plus the average (mean) value of the historical data set forms the basic trigger value for the next monitoring period. The SD principle (a basic statistical function) sets the trigger level criteria based on actual historical performance of the given indicator (data set), including its volatility (data point fluctuations). A more volatile historical data set will usually result in a higher (more generous) trigger level value for the next monitoring period. Triggers provide early warnings which enable decision makers to make informed safety decisions, and thus improve safety performance. An example of trigger levels based on standard deviations (SDs) is provided at Figure 11 below. In this example, data-driven decisions and safety mitigation actions may need to be taken when the trend goes beyond +1SD or +2SD from the mean of the preceding period. Often the trigger levels (in this case +1SD, +2SD or beyond +2SD) will align with decision management levels and urgency of action. Chapter 4. Safety Performance Management 4-13 4.4.5.3 Once SPTs and trigger settings (if used) have been defined, their associated SPI may be tracked for their respective performance status. A consolidated summary of the overall SPT and trigger performance outcome of the complete SPIs package may also be compiled and/or aggregated for a given monitoring period. Qualitative values (satisfactory/unsatisfactory) may be assigned for each SPT achievement and each trigger level not breached. Alternatively, numeric values (points) may be used to provide a quantitative measurement of the overall performance of the SPIs package. 4.4.5.4 It should be noted that trigger values serve to trigger (start) an evaluation, decision, adjustment or remedial action related to the particular indicator. An SPI being triggered is not necessarily catastrophic or an indication of failure. It is merely a sign that the activity has moved beyond the predetermined limit. The trigger aims to attract the attention of decision makers who are now in a position to take remedial action, or not, depending on the circumstances. 4.4.6 Caveat on triggers 4.4.6.1 There are challenges defining reliable trigger levels. Triggers and their associated levels work best when there is ample safety data and safety data management capabilities. This can impose an additional workload on the organization. The notion of trigger was designed and best suited to SRM of purely technical systems (e.g. aircraft engine monitoring). In this case, large amounts of quantitative data lend itself to accurate triggers and trigger level definitions. The notion of triggers is arguably less relevant to SRM of socio-technical systems. Socio-technical systems are systems where people actively interact with the processes and technologies to achieve the system’s service delivery or production objectives. Both SSP and SMS are socio-technical systems. The less reliable and meaningful triggers used in socio-technical systems is due to the limitations of reliable measures when humans are involved. 4.4.6.2 A more flexible approach is therefore needed for the triggers to be meaningful. Annex 19 does not require that States or service providers define trigger levels for each SPI. However, there are benefits for Figure 11. Example of representation of safety triggers (alert) levels 4-14 Safety Management Manual (SMM) organizations where their data for an SPI is very specific, there are enough data points and the data is sufficiently trustworthy. Figure 12. Example of setting safety triggers 4.4.6.3 The illustration above is an extension of the previous example, “fifty per cent reduction in runway excursions by 2022”. In this scenario, it is now the year 2020. The organization has been collecting safety data (SPI – “No runway excursions/million movement/yr”) and working with stakeholders to reduce the instances. The SPT for 2019 (<78 runway excursions/million movement in year) was achieved. However, the SPI shows that, not only was the SPT for 2020 (<64 runway excursions/million movement in year) not achieved, the number of excursions has exceeded the trigger in two consecutive reporting periods. The decision makers have been alerted to the deterioration in safety performance and are in a position to make decisions based on the data to take further action(s). Their data-driven decisions will aim to drive the safety performance back to within the acceptable zone, and on track to achieve their safety objective. 4.4.7 Identifying actions required 4.4.7.1 Arguably the most important outcome of establishing a safety performance management structure is the presentation of information to the organization’s decision makers so they can make decisions based on current, reliable safety data and safety information. The aim should always be to make decisions in accordance with the safety policy and towards the safety objectives. 4.4.7.2 In relation to safety performance management, data-driven decision-making, is about making effective, well-informed decisions based on the results of monitored and measured SPIs, or other reports and analysis of safety data and safety information. Using valid and relevant safety data combined with information that provides context supports the organization in making decisions that align with its safety objectives and targets. Contextual information may also include other stakeholder priorities, known deficiencies in the data, and other complementary data to evaluate the pros, cons, opportunities, limitations and risks associated with the decision. Having the information readily available and easy to interpret, helps to mitigate bias, influence and human error in the decision-making process. 4.4.7.3 Data-driven decision-making also supports the evaluation of decisions made in the past to support any realignment with the safety objectives. More guidance about data-driven decision-making is provided in Chapter 6. Chapter 4. Safety Performance Management 4-15 4.5 UPDATE OF SAFETY OBJECTIVES Safety performance management is not intended to be “set and forget”. Safety performance management is dynamic and central to the functioning of every State and every service provider, and should be reviewed and updated: a) routinely, in accordance with the periodic cycle established and agreed by the high-level safety committee; b) based on inputs from safety analyses (see Chapter 6 for details); and c) in response to major changes in the operation, top risks or environment. ______________________ 5-1 Chapter 5 SAFETY DATA COLLECTION AND PROCESSING SYSTEMS 5.1 INTRODUCTION 5.1.1 The distinction between safety data and safety information is made in the definitions found in Annex 19. Safety data is what is initially reported or recorded as the result of an observation or measurement. It is transformed to safety information when it is processed, organized, integrated or analysed in a given context to make it useful for management of safety. Safety information may continue to be processed in different ways to extract different meanings. 5.1.2 The effective management of safety is highly dependent on the effectiveness of safety data collection, analysis and overall management capabilities. Having a solid foundation of safety data and safety information is fundamental for safety management, since it is the basis for data-driven decision-making. Reliable safety data and safety information is needed to identify trends, make decisions and evaluate safety performance in relation to safety targets and safety objectives, and to assess risk. 5.1.3 Annex 19 requires that service providers develop and maintain a formal process to collect, record, act on and generate feedback on hazards in their activities, based on a combination of reactive and proactive methods of safety data collection. 5.1.4 Similarly, Chapter 8 of Annex 13 — Aircraft Accident and Incident Investigation requires States to establish and maintain an accident and incident database to facilitate the effective analysis of information on actual or potential safety deficiencies, and to determine any preventive actions required. 5.1.5 Annex 19 requires States to establish safety data collection and processing systems (SDCPS) to capture, store, aggregate, and enable the analysis of safety data and safety information to support their safety performance management activities. SDCPS is a generic term used to refer to processing and reporting systems, databases and schemes for exchange of safety information and recorded information. The term “safety database” may refer to a single or multiple database(s). State authorities with responsibilities for the implementation of the SSP should have access to the SDCPS to support their safety responsibilities. 5.1.6 Service providers are also required to develop and maintain the means to verify their safety performance with reference to their SPIs and SPTs, in support of their safety objectives by means of SDCPS. They may be based on reactive and proactive methods of safety data and safety information collection. 5.1.7 The guidance in this chapter is equally valid for States and service providers to assure that the safety data and safety information collected will enable effective and valid decision-making. 5.1.8 Organizations should ensure that it has personnel qualified to collect and store the safety data, and the competencies needed to process the safety data. This usually requires individuals with strong information technology skills as well as knowledge of data requirements, data standardization, data collection and storage, data governance and the ability to understand potential queries that may be needed for analysis. Additionally, the organization should ensure that each SDCPS has a designated custodian to apply the protection to safety data, safety information and related sources in accordance with Appendix 3 to Annex 19. Chapter 7 contains further details. 5-2 Safety Management Manual (SMM) 5.2 SAFETY DATA AND SAFETY INFORMATION COLLECTION 5.2.1 Objectives at different levels of the aviation system 5.2.1.1 ICAO has been introducing provisions across Annexes, Procedures for Air Navigation Services (PANS) and documents since the 1970s requiring States to establish reporting systems to collect safety data and safety information. Most of these provisions relate to sector-specific safety reporting systems; with the exception of Annex 13, which focuses specifically on the reporting of accidents and serious incidents. The provisions for mandatory and voluntary safety reporting systems found in Annex 19 originated in Annex 13. 5.2.1.2 Many service providers have collected a wealth of safety data and safety information, including mandatory and voluntary safety reporting systems as well as automated data capture systems. This safety data and safety information allows service providers to identify hazards and supports safety performance management activities at the service provider level. There are many benefits to sharing safety information, least of not which is the identification of hazards that are beyond the view of a single service provider. Information on the sharing and exchange of safety information can be found in Chapter 6. 5.2.1.3 Annex 19 requires States to establish SDCPS to capture, store, aggregate and enable the analysis of safety data and safety information to support the identification of hazards which cut across the aviation system. Arrangements may also be put in place for a third party to collect, store and analyse the safety data and safety information on behalf of the State. Information on the protection of safety data and safety information can be found in Chapter 7. 5.2.1.4 Furthermore, safety data and safety information need to be collected, stored and analysed at the regional level through the regional aviation safety groups (RASGs) to facilitate the identification of hazards that transcend State borders and to promote collaborative efforts to mitigate safety risks. 5.2.2 Determining what to collect 5.2.2.1 Each organization needs to determine what safety data and safety information it needs to collect to support the safety performance management process and make safety decisions. Safety data and safety information requirements can be determined using a top-down and/or a bottom-up approach. The chosen approach can be influenced by different considerations, such as national and local conditions and priorities, or the need to provide the data to support the monitoring of the SPIs. 5.2.2.2 Identifying and collecting the safety data should be aligned with the organization’s need to effectively manage safety. In some cases, the SRM process will highlight the need for additional safety data to better assess the impact (the level of probability and severity) and determine the associated risks. Equally, the safety performance management process may highlight a need for additional information for a more comprehensive understanding of a particular safety issue or to facilitate the establishment or refinement of SPIs. 5.2.2.3 Possible bias needs to be taken into account when collecting and using safety data and safety information. For example, the language used in voluntary reports can sometimes be emotive or aimed at achieving the objectives of an individual, which may not necessarily be in the best interests of the whole organization. In these cases, the information should be used judiciously. 5.2.2.4 States and service providers should consider taking an integrated approach to the collection of safety data that come from different resources, both internal and external. Integration allows organizations to get a more accurate view of their safety risks and the organization’s achievement of its safety objectives. It is worth noting that safety data and safety information that initially seems to be unrelated, may later turn out to be critical for identifying safety issues and supporting data-driven decision-making. 5.2.2.5 It is advisable to streamline the amount of safety data and safety information by identifying what specifically supports the effective management of safety within their organization. The safety data and safety Chapter 5. Safety Data Collection and Processing Systems 5-3 information collected should support the reliable measure of the system performance and the assessment of known risks, as well as the identification of emerging risks, within the scope of the organization’s activities. The safety data and safety information required will be influenced by the size and complexity of the organization’s activities. 5.2.2.6 Figure 13 provides examples of typical safety data and safety information, which in many cases are already available. Coordination amongst departments or divisions is necessary to streamline efforts for reporting and collecting safety data to avoid duplication. Figure 13. Typical safety data and safety information resources Next >