It is crucial to validate driving behaviour against current road rules to improve Autonomous Vehicle (AV) safety. Validating the AV behaviour is challenging due to the way Queensland Overtaking Road Rules are written, as it includes vague terms and exceptions. This research introduces a Defeasible Deontic Logic (DDL) based framework to validate AV driving behaviour against current overtaking road rules. The overtaking maneuver is the case study to illustrate the usefulness of the proposed system. The evaluation shows the effectiveness of the approach.
The advancement of technology took human civilization to a platform where life has become more
accessible, and human error in day-to-day life has become minimal. However, despite the constructive
and promising impact, this advancement of technology has some negative impacts also. For example,
all we know is that vehicle is advantageous for society, but at the same time, it is also known that road
crash is one of the major concerns of global public health due to the epidemic growth of road fatalities.
More than 3,7002 people die daily from road crashes. From 2013-20173, in Queensland, Australia, the
average death due to high speed was 58 per year, while the nu
An Autonomous Vehicle (AV) can be introduced to make the driving decision according to road rules to reduce road fatalities and injuries [2]. As AVs are designed and programmed to follow traffic rules [3], therefore, it is suggested that AVs would be the immediate solution to traffic violations [4]. AV is not permitted to violate traffic rules as it is operated by software where the traffic rules are programmed [5].
However, shifting from human-driven to AV may be difficult since it is uncertain how AV will integrate into the current regulatory system. There is no separate and comprehensive regulatory framework for AVs [6]. Leenes and Lucivero [3] mentioned that the current road rule model of the AV might be incomplete for some road scenarios. For example, there are vague (open texture) expressions in Queensland (QLD) overtaking road rules4, such as “approaching traffic”, “clear view”, “safely overtake”, etc. Following these vague rules is almost impossible for AV without proper additional interpretation [7]. Also, an AV will not be able to follow the rules related to exceptions [7].
In this research, the challenges and issues described above will be discussed and then solved by proposing a novel validation system. This work aims to design and build a system that: • Develop a methodology to formalize road rules into a machine-computable format. • Perform validation of AV behaviour against road rules.
To achieve these aims, an automatic validation system is proposed to conduct a multidimensional analysis of AV behaviour information, AV environment information, and road rules. This validation system validates AV behaviour against current QLD overtaking road rules. It also determines which road rules need additional interpretation for an AV to follow them correctly. Thus, AV safety in the current transport system can be improved [8]. Therefore, the objectives of this research are: • Design and develop an automatic validation system to validate the AV behaviour against current QLD overtaking road rules.
• Evaluate the proposed validation system’s performance.
To develop this system, this research aims to answer the fundamental question; “How to validate Autonomous Vehicle’s behaviour against current road rules?” The following research questions are addressed based on the operations level and evaluation of the validation system to answer this question. • RQ1: How to formalize road rules to handle and resolve rule vagueness, exceptions, and potential conflicts in rules for AV? • RQ2: How to model AV driving information (behaviour & environment) to enable AV to comply with the formalized road rules? • RQ3: How to align the formalized road rules and model AV information for the validation? • RQ4: How to evaluate the proposed validation system’s (RQ1, RQ2, & RQ3) effectiveness (performance)?
In this research, an automatic validation system is designed and developed to compensate for the identified research gap in the operations level (RQ1 & RQ2 & RQ3) and evaluation (RQ4) of the validation system performance. The QLD Overtaking Road Rule is considered as the case study to evaluate the proposed validation system. Overtaking road rules is considered as it is one of the most challenging road rules, which has several complicated and variety of conditions of rules. It is one of the significant, salient causes of road crashes [9]. This maneuver is considered one of road automation’s most required driving actions [10], as it keeps vehicle velocity consistent with the necessary road speed.
As a “disruptive technology,” AV has the potential to provide advantages to a wide range of people. However, there are some untapped issues about AV which need to be resolved first before wide public deployment. Because current road rules are created for humans, and there are several issues, such as vagueness and exceptions, that can create problems for an AV to follow them correctly. By validating AV behaviour regarding current road rules, it can be possible to determine which road rules need additional interpretation for an AV to follow them properly. Thus we can ensure AV safety in the current transport system. Some important research related to validating AV driving action regarding current road rules is depicted here.
A rule formalization method is presented in [11] to identify which vehicle is liable for the collision. In this process, first, a subset of Vienna Road rules was formalized in the Higher-Order-Logic (HOL). Second, a black box recorded the behaviour of the AV. Then, a formal compliance checking of road rules was performed regarding AV behaviour. Later, the author extended the work and tried to solve the rule vagueness issue using the safe distance theorem introduced in 2017 [7].
Rulebook, a formalism methodology for UK & Singapore rules, is introduced in [12] to make the self-driving vehicle behaviour compatible with the current traffic regulations. The author experimented with this rulebook on three specific traffic scenarios: unavoidable collision, lane change near intersection and clearance and lane-keeping based on 15 rules. However, the rulebook was domainspecific, and for a different nation, the priorities of the rules needed to be changed.
Esterle, et al. [13] presented a legal analysis (machine interpretability) of German road rules for AV to follow correctly. Regulations were codified and formalized using LTL to describe the temporal behaviour of vehicles for dual carriageway (two-lane roads). The effectiveness of this rule formalization was evaluated in a public dataset (‘INTERACTION’).
A rule formalization method is introduced in [14] to keep the autonomous vehicle accountable. This method worked in three major steps. First, implicit redundancy was eliminated from the legal text. Next, explicitly sorted out the AV and the user’s responsibility and then logically broke the rules into “predicate precursors”. The methodology used the overtaking rules as a case study. Although the theoretical case study experiment was impressive, however, there was no experimental proof.
A double-level model checking approach is proposed in [15] to capture the correct behaviour of an AV concerning traffic laws and to minimize changes to present road junction rules in the UK Highway traffic rules, including road entities and traffic signs. The model experimented on three different scenarios of the road junction. However, a complete traffic environment was not considered while assessing AV behaviour.
A limited number of studies worked on validating AV driving action regarding traffic rules. However, none of these studies addressed the combination of rule vagueness and rule exceptions, which are significant variant features of road rules and might create problems in AV driving action validation. In comparison to these works, in terms of handling and resolving rule vagueness and rule exceptions, this research proposed a Defeasible Deontic Logic (DDL)-based validation system that can effectively address and resolve these issues. DDL is used to formalize road rules to facilitate automated reasoning. DDL is used to effectively handle rule vagueness and exceptions. This methodology performs automated reasoning between formalized road rules and ontology-based AV driving information to make the validation.
The architecture of the validation system is shown in Figure 1. The proposed system consists of three modules: Rule Formalization (RQ1), Ontology Modelling (RQ2), and Reasoning (RQ3). A brief description of each module is given below.
This module aims to formalize QLD overtaking road rules into a machine-computable (M/C) format and thus solve issues: rule vagueness and exceptions in rules. Defeasible Deontic Logic (DDL) is used as a formal foundation of this formalization methodology [16]. This methodology works in four steps: define atoms, norms identification, if-then structure identification, and rule formalization.
Atoms are defined based on terms of rules. A term is a variable or an individual constant in the sentence. This work deals with those variables and constants that refer to subject (s), predicate (p), property (pr), object (o), and qualifier (q) in the rule sentence. Throughout the empirical study of the QLD overtaking road rules, atoms are defined semantically in terms of five patterns. These patterns are sufficient to cover the cases of road rules considered for this current research. These patterns and corresponding atom examples are given below. Pattern 1: Subject-Predicate-Object; Queensland Road Rule: Division 3 rule 151: 1 (a) ⎯ driver must not overtake a vehicle unless⎯ Defined Atom: driver_Overtake_vehicle Pattern 2: Subject-Predicate-Qualifier-Object; Queensland Road Rule: Division 3 rule 140: b⎯ the driver can safely overtake the vehicle⎯ Defined Atom: driver_CanSafelyOvertake_vehicle Pattern 3: Subject-Property; Queensland Road Rule: Division 2, Rule 15⎯ A vehicle includes a bicyle ⎯ Defined Atom: vehicle_Isabicycle.
Pattern 4: Subject-Predicate-Object-Object; Queensland Road Rule: Division 3 rule 151A-1⎯ motorbike between two adjacent lines ⎯ Defined Atom: motorbike_InBetween_adjacentLine1_adjacentLine2.
Pattern 5: Subject-Qualifier-Predicate-Object; Queensland Road Rule: part 19 rule 305⎯ A police vehicle is allowed not to display the light ⎯ Defined Atom: police_vehicle_allowed_NotToDisplayTheLight.
Deontic modality (norms) are conditions in the traffic rule that perform particular actions. They specify conditional terms and concepts of rules. Each norm is represented by one or more rules that may be constitutive or prescriptive. The constitutive rules define terms specific to legal documents. The prescriptive rules prescribe the mode of behaviour using deontic modalities: obligation, permission, and prohibition.
Road rules specify the actions of the subject. They consist of deontic modalities and conditions that control the subject’s behaviour. A rule comprises if (antecedent or premise) and then (consequent or conclusion). If the premise becomes true, then the consequent part of the rule triggers. Therefore, the if-then structure is identified from rules using atoms and deontic modality (norms).
After defining and identifying atoms, deontic modality and if-then structures for rules, the expressions are converted into a DDL-based logical format (M/C: Machine-Computable). DDL is an extension of Defeasible Logic (DL) with Deontic Operators and the compensatory obligation operators introduced [17].
Defeasible Logic (DL) is a non-monotonic, sceptical approach that does not support a contradictory conclusion. It aims to resolve the conflicts between knowledge. For example, suppose there is information that has support to conclude A, but also there is information that does not support A and prevents it from concluding A. If A’s support has priority over ¬ A, then it might be possible to conclude A. No conclusion can be made in such scenarios unless the rules are prioritised. The superiority relation (≫) used the priority set among the rules, where one rule may override the other rule’s conclusion.
In addition to defeasibility, traffic rules engage with deontic concepts. This research considered the Obligation [O], Prohibition [F] and Permission [P] deontic operators to encode traffic rules. The deontic operators are modal operators. A modal operator applies to a proposition to create a new proposition where the modal operator qualifies the “truth” of the proposition to which the operator is applied. For instance, a proposition from Queensland Overtaking Traffic Rule 142: driver_OvertakeToTheLeftOf_vehicle means that the “driver is overtaking the front vehicle from its left side”. Now, distinguish this proposition based on the above deontic operators: ⎯ Overtake Left: this is a factual statement that is true if the vehicle overtakes to a vehicle’s left and false otherwise (¬ OvertakeLeft is true). ⎯ [O]OvertakeLeft: this is a deontic statement meaning that the vehicle must overtake to the left of the vehicle. The statement is true if the obligation to overtake is in force in a particular case. ⎯ [F]OvertakeLeft: this is a deontic statement meaning that the vehicle is prohibited from overtaking the vehicle on the left-hand side. The statement is true if the prohibition to overtake is in force in a particular case. ⎯ [P]OvertakeLeftt: this is a deontic statement meaning that the vehicle has permission to overtake the left of the vehicle. The statement can be evaluated as true if the permission to overtake is in force in a particular case.
The deontic modalities modelled the normative effects (O, P, F). The standard deontic logic relationships between these deontic modalities have been used.
[F] Overtake ≡ [O] ¬ Overtake [O] Overtake ≡ [F] ¬ Overtake [P] Overtake ≡ ¬ [O] ¬ Overtake Now, a complete example of road rule formalization using DDL is shown in Table 2.
A driver must not overtake a vehicle unless— (a) the driver has a clear view of any approaching traffic; and (b) the driver can safely overtake the vehicle.
Atom driver_HasClearViewOf_approachingTraffic Atom driver_CanSafelyOvertake_vehicle Atom driver_Overtake_vehicle
r140: => [F] driver_Overtake_vehicle r140exc: driver_HasClearViewOf_approachingTraffic & driver_CanSafelyOvertake_vehicle => [P] driver_Overtake_vehicle r140exc >> r140
This module aims to introduce AV information’s ontology (knowledge base) design. The ontology design is required to represent AV information in machine-computable (M/C) format. So that AV information could be computed with formalized road rules.
An important characteristic of ontology is that it represents knowledge in the machine-computable (M/C) format as RDF (Resource Description Framework) data. RDF is designed as a conceptual statement to give a clear specification for modelling data [18]. M/C knowledge representation in RDF can bridge the gap between AV perception and knowledge processing. An ontology (M/C Knowledge base) could effectively represent road maps and driving behaviour, which is helpful for AV’s knowledge processing [19, 20]. Therefore, AV information (behaviour and environment) ontologies (MC knowledge base) are created in this work.
Sophisticated knowledge bases (ontologies) of AV information (behaviour and environment) are created to validate AV driving actions against road rules. The ontologies are used to provide the input (or case description) to the formalized road rules. The formalized rules used the ontology terms and definitions to provide the input for reasoning about the legal requirements for the AV in the situation identified by the data available to the AV. These ontologies describe concepts and relationships, including temporal representations between vehicle driving behaviour and the driving environment (road speed, marking, etc.). Also, these ontologies can be reused and easily extended by adding other concepts based on requirements. Protégé5 is used to build these ontologies. AV information is collected and processed from the CARRS-Q advanced driving simulator. Road information was collected from QLD Transport & Main Roads websites [21].
Two ontologies: AV behaviour and AV environment ontology, are shown in Figure 1. An example of AV information in the ontology is shown in Figure 2. These ontologies have 63 classes and 96 properties (object properties and data properties). The AV behaviour ontology is created using all AV behaviour information (i.e., speed, direction, lane number, etc.). The environment ontology is created using road information (i.e., road marking, road type, etc.) and AV surroundings information (i.e., weather, other vehicles, etc.). An example of ontology representation is shown in Figure 3.
In both ontologies, the “time” class is common (see Figure 3). This class kept track of every timestamp’s AV information. One driving maneuver may have hundreds of timestamps. For example, the case study example is shown in section 3.3, Figure 3, which is an 11.1s driving maneuver. This maneuver has 222 timestamps. The classes of the ontology (behaviour and environment) are 5 https://protege.stanford.edu/ automatically populated for each timestamp. A particular/full range of timestamps is validated based on the validation requirement. Because, sometimes, while assessing maneuver, all timestamps may not be necessary to assess. From the evaluation of certain timestamps, the validation result can be made.
This section introduces the reasoning to make the validation of the AV overtaking driving action (ontology knowledge base of AV information) against the QLD overtaking road rules (formalized road rules). The input of this reasoning engine is atoms (from formalized road rules), formalized road rules, and knowledge bases (ontologies).
Based on the validation requirement, the Road Rules Formalization module provided the required atoms for this module. Then a SPARQL_Query_Algorithm is developed to trigger predefined SPARQL queries for each atom to fetch the related driving information from the ontology and thus validate the atom whether it is true/false for the specific AV action. SPARQL is an efficient query technique for accessing an ontology. These queries are made based on the empirical study of the QLD overtaking road rules. In total, 694 queries are defined for QLD overtaking road rules.
For example, to verify the atom driver_Of_bicyle (atom of QLD overtaking road rules 141), the SPARQL Query is triggered, as shown below. The answer to the query shows that it is AV (Automated Vehicle). Therefore, it can be concluded that this atom is not true as the atom is about a bicycle.
Query 1: What type of vehicle it is? (AV_Behaviour)
prefix ab:<http :// www.semanticweb.org/bhuiyanh/ontologies /2019/8/ untitled−ontology −50#> SELECT ?Vehicle ?Type WHERE { ab:time_1 ab:driving ?Vehicle. ?Vehicle ab:is_a ?Type. } Query_Result: AV (Autonomous Vehicle)
Based on the atom, the number of queries varies. True atoms are determined for AV behaviour through the query result analysis. In query result analysis, most queries such as speed, acceleration, curvilinear coordinates, etc., are computationally computed to determine some characteristics (i.e., safe distance, approaching vehicle, etc.) of an atom. Based on these characteristics outcome (true/false) and other query results, the atom is determined whether true or false. In this research, true atom identification is the mechanism to resolve vague term issues in road rules. For the full details, see [22].
After determining the true atoms, a DDL compatible reasoner, Turnip6 is used to do the reasoning. Formalized rules and true facts (atoms) are sent in this Turnip and thus accomplish the mapping and reasoning (validation). 6 Turnip is a tool offering a run-time implementation of DDL. It is a tool that accepts facts, strict rules, defeasible rules, defeaters, superiority relation, and modality of DL. It supports (legal) reasoning with incomplete and inconsistent information. (https://turnipbox.netlify.com/)
For example, if we imagine that an AV intends to overtake a vehicle from its right side (see Figure 5). As it is a right overtaking, the AV must abide by QLD overtaking road rules 145, 144, 142, and 140 (see Table 3). In this right overtaking, for any situation (any specific timestamp), suppose Turnip receives the true atoms and formalized rules like Table 4 (due to the space limitation, few information is shown here). From Table 4, it is seen that, in Table 4 (a), AV is prohibited ([F]) to overtake as AV behaviour does not comply with traffic rules due to not having a clear view of approaching traffic. In this reasoning, the prohibition rule overrides the permission rule. This overriding happens through the superiority relation (≫) between rules, see [22]. However, if AV has a clear view of approaching traffic, then AV behaviour would comply with traffic rules (Table 4 (b)), which means AV has permission ([P]) to overtake. This is how the proposed automated reasoning happens and thus validates AV behaviour regarding current QLD overtaking road rules.
Atom driver_IsOvertaking_vehicle Atom vehicle_IsStationary ………………………..………………………..………………………..……… ………………………..………………………..………………………..……… Atom driver_OvertakeToTheRightOf_vehicle Atom driver_Overtake_vehicle
r145_a_1: driver_IsOvertakingByCrossingADividingLine_anotherDriver & driver_isDrivingOn_twoWayRoad & ~driver_HasPassed_anotherDriver => [F] anotherDriver_IncreaseTheSpeed …………………………………………………………………………………… r144_b_2_2:driver_IsOvertaking_vehicle & vehicle_IsTravellingOn_lineOfTraffic & ~driver_IsAtSufficientDistancePastToAvoidObstructingThePathOf_vehicle => [F] driver_ReturnTo_markedLane ………………………..………………………..………………………..……….. r142_1: => [P] driver_OvertakeToTheRightOf_vehicle ………………………..………………………..………………………..……….. r140: => [F] driver_Overtake_vehicle Facts (True atoms) anotherDriver_IsDrivingOn_markedLane vehicle_IsOn_centreOfRoad ………………………………………… ………………………………………… driver_HasClearViewOf_approachingTraffic driver_OvertakeToTheRightOf_vehicle
r142_1_a_i >> r142_1 ……………………… r140exc >> r140 Facts (True atoms) anotherDriver_IsDrivingOn_markedLane vehicle_IsOn_centreOfRoad ………………………………………… ………………………………………… ………………………………………… driver_OvertakeToTheRightOf_vehicle
• The green circle indicates legal action (Permission [P]) of AV in a specific timestamp • The red rectangular indicates illegal action (Prohibition [F]) of AV in a specific timestamp
This section describes how the proposed validation system works through a case study. A right overtaking case scenario (Figure 5) is made in the CARRS-Q Advanced Driving Simulator7. It is an 11.1s right overtaking maneuver. The simulator provides information every 0.05 seconds. This information includes all vehicles, roads, traffic information, etc. In this overtaking, the yellow vehicle (AV) intends to overtake the blue (TV-1) vehicle. As it is a right overtaking, therefore, according to the QLD overtaking road rules, the AV must follow the rules 145, 144, 142 and 140 (see Table 3). To perform this maneuver, the AV must deal with several vague terms, exceptions, and rule norms, such as “clear view of approaching vehicle”, “safe to do so”, “must not overtake a vehicle to the left unless”, etc.
The proposed validation system checks the AV driving behaviour for every timestamp (0.05s) to validate this overtaking (Figure 4). This maneuver is completed in 222 (11.1s) timestamps. After validating all timestamps, if there is no unlawful (prohibited driving action) action, this maneuver will be considered a legal overtaking. However, the overtaking will be regarded as illegal if there is any unlawful driving action for any timestamp.
Figure 4 shows the validation trajectory of this overtaking maneuver performed by AV. Due to the space limitation, only 50 (90 – 140) timestamps (4.5s – 7s) validation is shown in Figure 4 rather than showing 222 timestamps. In this trajectory graph (Figure 4), it is seen that, while AV overtaking TV-1, AV behaviour is illegal in the 118th (5.9s) timestamp (red rectangular). In this timestamp, AV did not consider the approaching vehicle, although AV should consider the approaching vehicle and keep its safe distance. In this maneuver for this specific timestamp, AV has no permission (prohibition ⎯[F]) to overtake. As a result, although in the whole maneuver, for all timestamps, AV has permission ([P]) (indicated by the green circle), except for one timestamp; therefore, the complete maneuver is considered illegal driving action.
A large-scale experiment is carried out to assess the proposed validation system. Forty cases of overtaking maneuvers are evaluated based on eight realistic Queensland overtaking traffic scenarios (Figure 6). The CARRS-Q (Centre for Accident Research and Road Safety ⎯ Queensland University of Technology) Advanced Driving Simulator is used to design experiment scenarios. Every case is a specific overtaking maneuver. 7 https://research.qut.edu.au/carrsq/engage/research-infrastructure/
Five different overtaking maneuvers are designed for each scenario. Two of these maneuvers are examples of explicit legal and illegal driving actions. The other three are borderline maneuvers, which may not be directly classified as traffic violations. One of the main reasons to make these three different types of maneuvers is that traffic rules contain vague terms (e.g., safe distance, approaching vehicle, clear view, etc.) requiring judgment by the drivers. Clearly, AVs need a deterministic and algorithmic approach. The determination if atoms corresponded to vague terms is delegated to the ontology and query method, where the queries implemented state-of-the-art techniques from traffic research. For example, determine whether the distance between two vehicles is safe. The parameters for the borderline situations are placed near the calculated threshold, whilst the values for the clear cases are considerably away. For instance, if a safe distance of 10 metres is determined, then values of 9 or 11 metres would be borderline, whereas 1 or 20 metres would be for clear cases.
Figure 7 shows an 11.1s snippet of experiment data (AV information). The CARRS-Q Advanced Driving Simulator is used to create this data. This simulator can provide the data under controlled and repeatable conditions, allowing for more useful and meaningful analysis. For every 0.05s, AV behaviour (speed, acceleration, position, lane number, going right, going left, heading, etc.) and the AV environment (road speed, intersection, marked lane, continuous line, broken centerline, weather, other vehicle behaviours, etc.) information are provided by the simulator. The simulator is scripted based on the ontology schema to fetch this information.
The proposed validation system validated AV behaviour for each maneuver, whether it is legal/illegal, according to the computable traffic rules. If the validation is illegal, then it is recorded as an illegal action of the AV. Every maneuver is considered as several driving actions. Each driving action is recorded as 0.05s (timestamp) of data. AV behaviour compliance with road rules is validated for each maneuver for every timestamp. After validating all experiment timestamps, the legal result is determined. For a maneuver (experiment), the whole maneuver is considered legal if the driving action is permitted (Permission ⎯ [P]) in all timestamps. However, among all timestamps, if any driving action is prohibited (unlawful action ⎯ [F]) for any timestamp, then the maneuver is considered an illegal maneuver.
Thirty-two participants of all ages having a valid Australian driving licence evaluated the abovementioned 40 overtaking maneuvers. Among these 32 participants, there were 8 driver trainers and 24 general drivers. Participants were considered irrespective of gender. Driver trainers participated as domain experts in this experiment as they should have a greater understanding of legal vehicle passing maneuvers in Queensland. There were three different categories of general participants which were:
experienced drivers (+10 years of driving experience), the average experienced driver (5-10 years of driving experience) and the inexperienced driver (under five years of driving experience).
Domain experts and general participants validated the legality of driving maneuvers from a set of video footage presented by a researcher on a computer screen. These maneuvers are simulation videos. Researchers used a desktop to show videos. For each maneuver, three types of videos were shown on one screen so that participants could watch all possible driving actions of the vehicles. Participants watched every maneuver video twice and evaluated whether the maneuver was legal or illegal according to their knowledge of QLD road rules. If their assessment of any maneuver was illegal, then they provided their reasons in writing.
Every domain expert evaluated 20 maneuvers in two stages. In the first stage, they evaluated 10 maneuvers, followed by a 10 min break and then another 10 maneuver validations were completed in stage 2. Every general participant evaluated 10 maneuver videos. Maneuvers were allocated to the participant based on randomly chosen scenarios. There were eight video scenarios, with every scenario including five overtaking videos. Thus, there were 40 videos in total.
To recognize their participation in this research, participants were compensated (150$ and 25$ gift cards for domain experts and general participants, respectively) for their voluntary involvement.
The validation system performance (effectiveness) is determined based on how many participants agreed with the system’s evaluation. The evaluation is conducted in two aspects:
Figure 8 shows the performance of the proposed validation system. In clear overtaking maneuver cases, on average, there was 84% legal/illegal and 86% reason identification agreement between participants and the system. In borderline overtaking maneuver cases, participant average agreement rates with the system’s legal/illegal validation and reason identification are almost identical, which is 59%. The borderline cases are designed to test the human perception of the maneuvers with a very close threshold between legal and illegal in terms of a maneuver. According to the 50% outcome is truly indicative that the borderline cases are really borderline. Based on these agreement rates of clear cases and borderline cases, it can be stated that the proposed validation system is a promising approach to assessing AV behaviour.
Performance of the proposed validation system Border line cases
Clear Cases 59,26% 59,38%
86,35% 84,85% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Reason Validation
Legal/Illegal Validation
The dissimilarity between the system and participants validation of the maneuver is shown in Table 5. 40 overtaking maneuvers were validated 400 times by 32 participants, where 24 general participants validated 240 times and 8 domain experts validated 160 times. Table 5 shows that in total 40 maneuvers, there are 122 dissimilarities between participant and system evaluation regarding legal/illegal validation. The majority of these dissimilarities are seen in borderline overtaking maneuver cases, which is 77.87% (95 out of 122).
A similar pattern is noticed in terms of reason identification (Table 5). The major dissimilarity (35 out of 45) between the system and participants validation happened in the borderline overtaking maneuvers. For a reason identification, 25 illegal maneuvers were validated 156 times by 32 participants, whereas 24 general participants validated these maneuvers 88 times and 8 domain experts validated 68 times.
Participants Validation
General Participants
Domain
Experts 96 144 40 48 400 156 64 96 27 41
After examining participant validation, it is clear that certain participants sometimes fail to consider important factors and judge unlawful maneuvers as legitimate. Also, some participants overstate judgements and misjudge the legality of some legal maneuvers. Additionally, it is noted that participant’s lack of awareness of road rules is the reason behind these misjudgements. For instance, although left overtaking is lawful in Queensland, some participants, including domain experts, remarked that a particular overtaking maneuver is illegal as it involves left overtaking. The empirical observation demonstrates that these discrepancies are caused by a variety of participant factors, including a participant’s improper knowledge of traffic regulations, misunderstanding of legal or illegal driving action of vehicle, etc.
Such disparities in participant validation could not have occurred if it had been possible to select suitable volunteers after assessing their understanding of traffic regulations and safe driving. However, finding participants is a difficult process. Through the interview, it is also harder to find the expected participants. Also interviewing process is time-consuming and expensive. Furthermore, evaluating video material is a difficult process. It might be challenging to see everything precisely from video footage at times. When someone sees three perspectives at once, it gets more complicated. The limitations on how many times each overtaking maneuver’s video may be seen could potentially affect the evaluation.
This research developed an automatic system to validate Autonomous Vehicle (AV) behaviour against current QLD Overtaking Road Rules. Through this validation system, this research aims to check whether AVs comply with existing QLD overtaking road rules. It also determines which rules need additional interpretation in terms of the information available by an AV; so that AV can follow them correctly. Thus, this validation system can improve AV safety for the current transport system. Defeasible Deontic Logic (DDL) is used to formal specification (machine-computable) of current Queensland Overtaking Road Rules. This logic also facilitates automated validation by reasoning between formalized road rules and AV information (ontology knowledge base). DDL performs effectively to handle rule vagueness and exceptions. A data-driven experiment is conducted to evaluate this proposed validation system. The evaluation shows the effectiveness of this validation system. Therefore, it can be stated that this validation system will be helpful in improving Autonomous Vehicle safety for the current transport system.