AI-Assisted Guidance and Documentation System for Crashed Vehicle Handling:

As a response to the zero-emission targets, the number of electric & alternative fuel vehicles is increasing drastically. Due to the lack of experience with new cars and on-site information, firefighters are having difficulty responding to accidents involving these new vehicles. It becomes even harder as there are no widely accepted procedures and many fundamental structural changes in the new vehicles. First responders and firefighters need to adapt themselves, thus they need guidance and help from the industry to be able to respond to this change properly. Inspired by electrification, FUSE aims to provide guidance for all types of the vehicles.

FUSE is an AI-assisted guidance and documentation system and a device designed to be used by firefighters for crashed vehicles.

With the help of sensors and cameras, FUSE creates a 3D scan of the incident and compares it with the 3D models of vehicles to generate real-time recommendations in augmented reality form. There are 4 main features of the handheld device:

- Connection to the vehicle's battery management system (BMS) and the minimum set of data (MSD)

- 3D scanning

- Thermal imaging

- Laser cross projection

While the first 3 are used by AI to create the recommendations and make calculations, the laser is used to project important location information. Let me give an example, when an EV is on fire, one of the most recommended ways to extinguish it is by flooding the battery with water by puncturing it from a precise location. The AI would know and show the operator where to aim and the operator helps other firefighters by projecting it onto a real-world position. On the other hand, the command center has a web version of the interface, and every piece of information collected and shown can also be accessed for a better situation assessment.

Essentially, the concept is completely reforming the current rescue sheets into a real-time interactive guide. This is a unique solution that uses the capabilities of radar & lidar sensors with a regular and infrared camera together with a smart brain. Furthermore, the information collected can be used to train the AI and firefighters for future cases.

What It Does

FUSE is an AI-assisted handheld device and a system designed for firefighters to be used for car accidents. FUSE has a set of sensors and cameras together with flashlights and a laser cross generator. It's not only packed with sensors but a powerful AI which has the access to a database of 3D models and rescue information of the vehicles together with real-world knowledge. It compares 3D data of vehicles with real-world scans and generates recommendations in augmented reality form.

The closest archetype of this concept is the handheld infrared cameras used by firefighters for measuring the surface temperature, however, it can't be compared with this concept as the main purpose is completely different.

Essentially, the concept is reforming the rescue sheets into a real-time interactive guide. This is a unique solution that uses the capabilities of radar & lidar sensors with a regular & infrared camera with a smart brain. The AI combines all these real-world data that is scanned with the information in its database. (3d models of the vehicles, rescue sheet information manufacturers obligated to provide & real-world knowledge). There are AIs used in a similar way such as AI used in autonomous vehicles or for face recognition systems. The FUSE system barrows this approach and applies it to an emerging area that needs a new and up-to-date solution. It also uses the battery management system available in the vehicles to access important data.

How It Works

With the help of a set of sensors and cameras, FUSE creates a 3D scan of the incident and compares it with the 3D models of the vehicles to generate real-time recommendations in augmented reality (AR) form. There are 4 main features of the handheld device:

- Connection to the vehicle's battery management system (BMS) and the minimum set of data (MSD)

- 3D scan

- Thermal imaging

- Laser cross projecting

While the first 3 are used by the AI to create the recommendations and make calculations, the laser is used to project important location information. For instance, when an EV is on fire, one of the most recommended ways to extinguish it is by flooding the battery with water by puncturing it from a precise location. The AI would know and show the operator where to aim, thus they can help other firefighters by projecting it onto a real-world position.

The command center has a web version of the interface so every piece of information collected and shown can also be accessed by them.

Pain Points

There are some new problems to solve when it comes to vehicle accidents, especially accidents involving electric and alternative fuel vehicles. Some of the main pain points this concept aims to address are shown below:

1. Access to the Vehicle and Structural Info

It is crucial for the first responders to know the location of the airbags, intersection points, battery, high voltage cables, reinforced panels, etc. for any response including extrication as well as how to discharge the system. However, these are only available as written documents and that is not always the best way to reach the information in time of an emergency response.

2. Information Transfer to Firefighters

Firefighters are informed about the incident via e-Call and MSD (for the location and transmission system of the vehicle) followed by e-mails for rescue sheets. (They might also have the printed version) Detailed information about the vehicles is not easy to access.

3. Correct Estimation of Battery Status

Currently, firefighters need to observe all the indications about the battery such as physical deformation, leakage, fire, smoke generation; hissing or cracking sounds, thermal status, etc. which are easy to miss and not the strongest indicators. Any wrong assessment can have fatal consequences. Currently, assessing the condition of the battery correctly is a big challenge for firefighters.

4. Directing Water to a Precise Location

When there's a fire or a thermal runaway in the battery, the water needs to be directed to specific locations such as the air channels. In order to access these areas, the battery might need to be punctured from the best location possible, puncturing a wrong area can also cause a short circuit or even initiate a thermal runaway. If this is carried out incorrectly, it could lead to fatal consequences.

The Handheld Device

Placement

The unique form factor of FUSE allows it to be placed in different positions. While it can be placed straight, it can also be placed inclined at a 50° angle to allow for easy discussions during the incident, or sideways. There's also a bumper part in the front that absorbs impacts to protect the more sensitive sensor area.

Securing

The user of this tool is a firefighter in direct contact with a command center, thus they need their hands to be free to answer the calls and many things they do at an accident scene.
To secure the device easily, FUSE has stainless steel hooks placed inside the body tightly, allowing for a secure connection for a strap to be locked.

Tripod Connection Area

FUSE can be mounted to a tripod to observe an ongoing situation easier as it could take some time to be able to respond to some incidents that take longer.

Battery

For a secure battery connection, there's a locking mechanism underneath and the wide opening allows for easy and quick access to this latch.

Display

The screen of FUSE is designed with a matte display, and it can be tilted up to 40° for various applications and better ergonomics. When tilted at a certain angle, it locks in and stays on that angle till the user changes it again. That can also be handy to prevent having glares at the screen on a sunny day.

The display was designed indent with a frame of soft material that absorbs impacts and protects the display from taking damage easily.

Since the main purpose of the interface is to give guidance, it requires much information to be displayed. The moderately large screen is designed in an optimum size to offer great visibility while displaying this information and keeping the device compact.

Physical Interface

The keypad is designed as a single piece of rubber material to provide a waterproof solution with affordable manufacturing costs and low maintenance. The transparent icons are backlit, and they are highlighted with a stronger light when active.

FUSE is designed with physical buttons for seamless interaction in harsh use cases and variable weather conditions, and it is designed perfectly aligned with the digital interface.

Front Sensor Area

The components are carefully selected from the reference applications in different products since this is a new type of device. FUSE uses available and accessible components in the market. In the front, FUSE has a set of sensors and cameras together with flashlights and a laser cross generator.
The more sensitive area is designed indent to prevent possible damage. Despite using various surface finishes and a highlighted print of the accent color on the edge of the black front area, that is a one-piece part holding all the sensors and cameras. Connecting with this part on top, there's a rubber cover for the USB-C and display connection ports to provide fast data transfer. On the bottom, there's a rubber bumper.

Digital Interface

The digital interface is designed for the use of professionals, intuitively providing relevant information and guidance.

AI & AI Assisted AR Overlays

When using a thermal camera, the user can only measure and see the surface temperature. In the image on the right, there's the thermal view of the illustrated battery module of a vehicle. FUSE creates this 3D visualization digitally by using the "Critical Vehicle Info" that it retrieved from BMS and places it on the scanned and matched 3D data of the vehicle in the AR view window. This way, the user can see the thermal image of the battery through the screen as if it is being measured from the surface of the vehicle.

Using real-world knowledge and the 3D models of the vehicles, FUSE compares them with the scan data, then analyses them. This way, the AI could see the unmatching areas, deformations, possible areas where the tension is built, possible damages to the internal components that are important for the response, etc.

Some tasks are also visible in the AR view if their location is vital when responding. When those tasks are selected and the laser marking is used during that time, the system helps with the adjustment of the projected laser marking for better accuracy.

Documentation & Education

FUSE is connected to the fire command vehicle via WiFi, thus, all the data retrieved and displayed on the screen and available on the site for the firefighters could reach the command center. This would allow for it to be recorded to a server to study and learn from different incidents. That could either be used to train the firefighters and also it could be used to improve the AI. Apart from educational purposes, recording incidents as soon as they occur is also a crucial part of a proper incident investigation.