Preliminary analyses show that in order to pursue those challenges, the future European aerial combat systems will need to be equipped with an innovative cockpit offering the pilot breakthrough display and interaction capabilities. In this context, it seems clear that new products (head-down, eyes-out, interface modalities, virtual assistant…) have to be developed.
Hence, this topic addresses the rise in maturity, with the objective to reach TRL 4, supported by demonstrations, of technological and technical solutions necessary for future enhanced products.
The proposals may consider existing manned and unmanned air platforms and future ones under development, including training aircraft in a long term perspective or as quick-win.
Against the background of the design of new generation air combat platforms in Europe, or upgrades of those today in service, the following themes have to be considered:
- Adaptive human system collaboration: adaptive collaborative HMI for operations in a distributed environment with multiplatform assets and the definition of novel design and interaction principles for managing automated/autonomous aircraft functions and cooperating with System-of-Systems team mates, including adaptive interfaces;
- Visualisation: both visualisation products and advanced pilot information presentation capabilities;
- Crew monitoring system: systems and techniques able to support and assist pilots, and in general human operators in performing the flight and mission control in a more demanding operational environment;
- Interaction modalities: the need for innovative HMI technologies including e.g. wearable, visionics, haptics, vocal command, virtual operator assistant, Augmented Reality, 3D holography and implementation concepts;
Emulation of the pilot interactions with its environment might be addressed when needed in a transversal way within the studied areas.
All of those themes will be able to rely, at different levels, on different technology building blocks exploiting the opportunity to use advanced research techniques such as artificial intelligence, machine learning or others that can enable more advanced capabilities for the overall mission performance. This topic is therefore transversal. However, as a complement, it may be interesting to study the theme of “Decision-making system” whose objective would be to prioritize, order and present, whatever the situation during the mission, the most relevant information to the pilot with an objective of efficiency and safety.
Whatever the theme considered, quick wins must be identified, evaluated and tested so as to prepare their implementation on current or upcoming systems.
The proposals must cover the following activities as referred in article 10.3 of the EDF Regulation, not excluding upstream and downstream activities eligible for development actions if deemed useful to reach the objectives:
- Studies, such as feasibility studies to explore the feasibility of new or improved technologies, products, processes, services and solutions, related to advanced air combat cockpit HMI functions and related technologies enabling effective multi-role and networked operations including MUM-T in a highly contested environment.
In particular, the targeted activities are:
- Operational and training use-cases definition to elaborate specifications (performances, safety objectives…). For that purpose, workshops will be implemented with participating Member States and associated countries ministry of defence representatives, including end-users, to establish high-level operational requirements and relevant scenarios.
- For each of the abovementioned themes, relevant technology identification, analysis, including quick wins opportunities identification and analysis leading to the demonstration and development activities dedicated to these opportunities to be performed in a shorter timeframe in order to enable quick implementation process.
- Technologies evaluations and demonstrations. These activities will help on to select the relevant technologies to upgrade and to demonstrate further the efficiently of these upgraded technologies through operational scenarios.
- Relevant technologies maturation and development.
In order to maximize inter-domain synergies and take advantage of distributed expertise, proposals must address the abovementioned themes according to their following respective description, notwithstanding others fields of interest leading to identify new technologies to be explored, preliminary developed and demonstrated:
Theme N°1: Adaptive human system collaboration
This theme addresses the definition of a new paradigm of human-machine teaming in future collaborative and connected air warfare. New generation military aircraft participating in collaborative air combat will require a human system interface that enhances the awareness of the tactical situation and allows an ergonomic cooperation between crews and machines for a safe flight and high performance in cooperation with both manned and unmanned assets. This will cover human-machine/human-human/machine-machine Teaming, but will not include functional algorithms.
Legacy human-machine interfaces lack the necessary flexibility and adaptability to meet the demands of future combat systems. To avoid compromising the effectiveness of human operators in the future, applied research is required to address topics such as:
- HMI principles for cross platform mission management considering (human- machine/human-human/machine-machine teaming, not including the specific functional algorithms;
- Adaptive HMI mechanisms e.g. based on crew management system (CMS) data and in accordance with the specific operational context.
This will then lead to main characteristics of the Human Machine Interface as follows:
- Strengthened and adaptive cooperation between all systems, either manned or unmanned, involved in an operation;
- Human supervised delegation of tasks to more and more autonomous systems;
- “Real co-pilot” like assistance to provide the crew with system proposals and to adapt interfaces.
Theme N°2: Visualisation
This theme addresses both visualization and advanced pilot information presentation capabilities, including 3D presentation, and other novel presentations that could be implemented in the next generation of aircraft, through:
- Augmented reality, large area displays (free form, multi touch, auto-stereoscopy), 3D holography and implementation concept;
- Helmet mounted display (HMD) solutions crucial for the next generation cockpit. Technological solutions exploration should be carried out for increasing technical characteristics in terms of presentation field and functional capabilities (integrated night vision, primary flight display function, and support for CMS sensors, target designation and view through the cockpit. It will also have to take into account the control of its inertia characteristics (mass and centre of gravity of the HMD carried by the pilot’s head).
Therefore, the following areas could be investigated:
- Digital integrated night vision;
- HMD wireless link;
- Enhanced synthetic vision system (including live virtual constructive visual integration).
Under the scope of this theme, there will be demonstrations with physical, digital mock-up and/or simulations on the basis of operational “use-cases”. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration also based on the initial user requirements.
Theme N°3: Crew monitoring system
This theme concerns the real-time monitoring of the physiological and cognitive states of the crew. The elements of interest, or deleterious capacities, to be monitored are, for example, operator incapacity (G-LoC61, hypoxia, spatial disorientation…), hypovigilance, attentional tunnelling, mental workload, stress and situational awareness. These aspects are crucial in particular for the future air combat systems where the operational environment and the way of operate are significantly more complex than the current ones. Crew monitoring system can be applied to operational embedded systems as well as to training systems (embedded or on ground).
In order to more specifically mature the CMS models the following areas could be investigated:
- Mental workload;
- Ability to collaborate;
- Situational Awareness;
The validation of CMS models is a crucial point in the CMS chain’s rise to maturity also based on a pilot behavioural knowledge base (PBKB) that needs to be contextualised in accordance to the diversity of human, missions and tasks, including through AI and ML-based techniques.
Under the scope of this theme, there will be demonstrations with physical, digital mock-up and/or simulations on the basis of operational use-cases. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration.
Theme N°4: Interaction modalities
This theme addresses both the modalities of interaction as well as their combination in the field of e.g. wearable, visionics, haptics, vocal command, touch, gesture, etc.
In terms of means, it takes into account:
- Audio, in terms of input/outputs: voice command, natural language processing, in a very constrained environment such as that of a fighter, voice synthesis and advanced audio functions such as 3D Sound for example;
- Eye: the eye tracker which is used as a CMS sensor is here dedicated to interaction. Coupled with another modality such as voice, it is a vector of efficiency for target designation in an eyes-in or eyes-out use;
- Touch: a particular objective will be to study multi-touch (up to 5 fingers) technologies to interact with the displays;
- Gesture controls;
- Haptic/Tactile display of information;
- Handwriting recognition and more generally the ability to interact naturally with a “white board”.
Multimodality would provide greater security, resilience and accuracy by removing ambiguity about the operator’s intentions.
Under the scope of this theme, there will be demonstrations with physical/digital mock-up and/or simulations on the basis of operational use-cases. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration.