ICAO USOAP and Safety Performance
From SKYbrary Wiki
Critical Elements of a State’s Safety Oversight System
As an initial step to determine the most effective safety strategy, ICAO has been conducting studies aiming to correlate Universal Safety Oversight Programme audit results to accident rates. USOAP assesses eight critical elements related to the establishment and implementation of a State’s safety oversight system. The audit programme is a rich data source, with more than 144 USOAP audits conducted and over 146,000 records currently archived. This data has its limitations, but it is important to note that ICAO continues its analysis efforts in order make best possible use of it for the benefit of aviation safety.
State USOAP Results as a Potential Safety Indicator
The measurement used to determine USOAP results is the State’s ‘Effective Implementation’ (EI) scores, or alternatively, the Lack of Effective Implementation score. The lack of effective implementation is a measure of the State's safety oversight capability and is defined for each critical element (CE) of the State’s safety oversight system. The LEI for each CE is calculated by taking the percentage of the total number of not satisfactory PQs over the total number of applicable PQs (linked to the CE). The overall LEI is the average of the 8 LEIs for each CE and is the one published in the audit reports. The most favourable USOAP results are those having the lowest LEI percentage. Given the focus put on reducing the variance in regional accident rates, each State was categorized within the ICAO region it belongs to. No clear pattern could be identified as there are States having favourable, moderate or unfavourable USOAP results in every region.
ICAO has developed dedicated tools that make use of the output of safety audits performed in accordance with the adopted USOAP CMA and allow for monitoring of States' progress in the implementation of the ICAO SARPS in the following 8 areas:
- Primary Aviation Legislation and associated civil aviation regulations
- Civil aviation organizational structure
- Personnel licencing activities
- Aircraft Operations
- Airworthiness of civil aircraft
- Air Navigation Services
- Accident and serious incident investigations.
The USOAP interactive viewer enables the viewer to see the audit ‘Effective Implementation’ (EI) scores for the eight USOAP areas per country. Multiple countries may be selected at once to compare their respective performance. Users may also compare a country or countries against the current global average in each category.
The Datatable accessible on the iSTARS API data service page provides for viewing of the exact EI scores of a State, the scores assigned to the CEs of a State’s safety oversight system and the overall score per state.
Aggregate results at global and regional level can also be found in the ICAO USOAP Results Report: 2013 - 2016.
The access to the audit data used to produce the scores is only available to authorised users. .
The Evolution of a Performance Based System
The findings of the USOAP results’ analysis demonstrate that establishing strong safety oversight systems is a necessary first step to ensure the successful transition to improved safety management. Therefore, safety strategies must have the ability to consider the varying maturity levels of State’s safety oversight systems. States that have not yet implemented the eight critical elements of a safety oversight system effectively must first resolve these deficiencies and develop a sound foundation upon which to build their State Safety Programmes. Only those States having mature safety oversight systems will be able to realize the benefits associated with safety management principles, and achieve further reductions in their accident rates.
Representative Protocol Questions
ICAO safety experts analysed further the identified relationship between USOAP results and safety performance based on accident rates. The in-depth analysis led to the identification of a representative sub-set of protocol questions (RPQs), notably 114 out of 975 in total, which maintained the high correlation factor between the LEI percentage and the accident rate. This is an important finding as it will allow States to better focus their resources and safety improvement efforts and realize the desired reduction of accident rates in a timely and more efficient manner.