Four Ways of Testing how to Minimize Risk of Injury in Automatic Train Operation

Date: October 11, 2023

Contribution by IKERLAN

Partner Ikerlan is pursuing the development of two safety functions that will minimize the risk associated with Automatic Train Operation. Two scenarios offer approaches for minimizing the risk of a train running over or injuring people on the track as well as avoiding damages during opening/closing operations on the platform. Four activities are underway to support these scenarios, training dataset generation, track detection techniques research, obstacles detection techniques research, stereo distance calculation.  

Training Dataset generation. A dataset consisting of three sets of labelled images for training, validating and testing neural networks algorithms, such as Yolo (in this case YoloV7 C++ was selected) is being generated. In reality, two pairs of datasets are being generated:

  • Datasets with images based on TrainSimulator © (https://store.steampowered.com/app/24010/Train_Simulator_Classic/)  for People/Obstacles on the tracks(with stereo) and People detection around train doors on the station platform.
  • Datasets with images based in TrainSimulator © but modified with a Stable Diffusion © tool (https://stablediffusionweb.com/), for People/Obstacles on the tracks (with stereo) and People detection around train doors on the Station Platform.

The second dataset, the one to which Stable Diffusion is applied, aims to produce a more realistic dataset. This dataset is basically composed of Train Simulator images followed by their transformation with Stable Diffusion. Figure 1 below shows a Train Simulator scene to which stable diffusion has been applied.

Figure 1 – Stable Diffusion applied to Train Simulator Image to produce a more realistic scene

Track detection techniques research. The railway case study not only needs to detect people and/or obstacles, it also needs to make sure that the people/obstacles detected are on the track (objects outside the track are irrelevant). Several techniques found in the literature are being investigating, ranging from techniques based on vision neural networks (below image) to those based in pure OpenCV algorithms.

Figure 2 – Rail track detection with Rail_Marking network

The image above was generated with experimental implementation proposed in (https://github.com/xmba15/rail_marking).

Obstacles detection techniques research. In the SAFEXPLAIN project, YoloV7 was proposed as a tool to detect people/obstacles on the track. In railway case study, two alternatives are being checked and compared. On one hand, the use of a standard YoloV7 network is applied, as is, to Train Simulator images. On the other hand, two specialised networks will be trained with (a) Train Simulator images and (b) Train Simulator images modified with Stable Diffusion. The images below (Figure 3) show Train Simulator images upon which people have been added using Stable Diffusion tool.

Figure 3 – Images generated with Train Simulator with Stable Diffusion adding people on the track

Stereo Distance Calculation by stereo matching techniques research. In the SAFEXPLAIN railway case study, a distance calculus by stereo matching based on OpenCV functions will be used. A left and right camera in the front of the train will capture stereo images that are fed to a stereo function to estimate distance. Several algorithms provided by OpenCV are being tested (bm and sgbm algorithms) along with post matching different options for filtering types, window size, etc.

Figure 4 – A variety of stereo matching results

Figure 4 above shows two stereo images (top line) and three different results of stereo matching producing Depth Maps (grey colour represents depth/distance) depending on the different parameters applied to the matching.

It is important to emphasize that the railway case study does not intend to develop new AI algorithms for obstacle detection or track detection. Instead, it serves to integrate AI explainability techniques to facilitate the certifiability of the autonomous train driving system based on standard AI system such as Yolo, etc.