Final Report Summary - FTK (Fast Tracker for Hadron Collider Experiments)
Project objective summary: In recent years, there has been substantial development in image detector technology that led to a great increase both in space resolution and in produced data. These detectors target several different application fields from everyday applications (smart phone cameras, web cam...) to complex and demanding applications in the most varied fields (high energy physics, medical imaging, security, ...). All applications require an effective data reduction strategy allowing minimal loss of information. Pattern matching is a common algorithm used for such strategy. Pattern matching looks for a given sequence of tokens (data) that constitute a predefined pattern. This strategy is not limited to image processing, but can be extended to other fields such as data servers (e.g. search engines) and all types of data processing that require identification of patterns. The primary FTK project goal is the construction of a board containing 64 chips, the Associative Memory (AM) chip [1], with each chip able to execute 1 million comparisons every 10ns. The complete system is composed by as many boards as necessary, working in parallel. Such high-performance requirements are typical of event reconstruction at hadron collider experiment in high energy physics (HEP). These HEP experiments search for extremely rare processes hidden in the middle of billions of very frequent not interesting processes. In these experiments these rare processes are identified analyzing the signal produced in the detector elements by the hundreds of particles emerging from the proton-proton (pp) interaction. The high pp intensity necessary to produce the rare processes implies a very high interaction rate. In this harsh environment, the detectors signal therefore corresponds to more than one pp collision much in the same way as a camera integrate all the photons entering its objective during the time the diaphragm is open. The ensemble of all the detector signals collected for one “shot” is named “event” and corresponds to the “image” of all the particles produced in the pp interactions. In ATLAS a single event is produced every 25 ns and consists of the signal of the about 100 million detector elements. The data flow is so massive that only a very small fraction of these events can be stored for later analysis. A drastic real-time data reduction must be obtained with minimal loss of useful information. A multi-level online selection, named trigger, is an effective solution for an otherwise impossible problem. These multilevel triggers quickly identify those events with the greatest scientific potential reducing the rate from the machine event production (40 MHz at the Large Hardon Collider, LHC) down to hundreds of Hz, a rate that satisfy the offline data storage limitations. Tracking devices, with silicon devices as the predominant tracking technology, play an essential role in the identification of interesting events. These devices allow disentangling the different overlapped interactions in the same event. However, these detectors contain up to millions of channels requiring huge computing power for full track reconstruction, an extremely challenging task even for an off-line analysis. Consequently, complete high-quality tracking for real time event selection at very high rates has been considered impossible in LHC experiments. Real-time tracking was planned only for a limited detector region or on a small subset of events, previously selected using information from lower resolution detectors. The electronic boards designed and produced by the FTK project overcome this problem providing real-time tracking for all the events using a massively parallel high performance tracking system. The FTK project aims also to demonstrate how this system, developed for the HEP environment, can be successfully used to solve the image reconstruction problem in other fields. The chosen representative case is the reconstruction of medical images.
Work performed since the beginning of the project: electronic boards were developed for the FTK Processor [2], an approved ATLAS upgrade. The implemented system is based on the optimal mapping of a complex algorithm onto different technologies: a combination of high performances dedicated hardware and distinctive flexibility of general-purpose but lower-performance CPUs. A key role in the architecture is played by high-performance field programmable gate arrays (FPGAs), while most of the computing power is provided by full-custom ASICs named AM. We built a powerful highly parallel dedicated hardware that provides excellent performance, reaching resolutions, efficiencies and fake track rejection typical of offline algorithms. The latencies are short (about one microseconds), power usage is low (the AM chip, has a power consumption below 3 W for 1M fit/10ns), and the system is small (the boards contained in 4 racks perform a task that would require a farm of O(1k) of commercial CPUs). The AM, the central device of our system, store each pattern in a single memory location consisting of 8 independent words of 16 bits each. Each of these words is separately compared to its own input data stream [1]; the coincidence of the 8 words matches, found at any time during data arrival, determines the match of the pattern. FPGAs control, configure and handle the AMs, providing the flexible computing power to process the selected patterns. Distributed debugging and monitoring tools suited for a pipelined, highly parallelized structure and a high degree of configurability have been developed to cope with different applications.
Main goals achieved: We have built the AM [4] and clustering [5] system prototypes, tested and integrated them in the global FTK system at CERN. The design of the AM system was a particularly challenging task for the following factors: the high pattern density (8M patterns/board), required a large silicon area for the AM ASICs; the I/O signal congestion at the board level, required heavy use of serial links; and the power limitation due to the maximum cooling power cooling resulted in setting the maximum power to 250 W per AM board. The extreme challenge presented by the AM chip design forced us to adapt the project schedule however it did not impact any of the project milestones. The infrastructures (crates, heatsinks, fans, power supplies) were studied and optimized [6]. The described system was developed and tested for HEP detectors and it is expected to highly increase the physics discovery potential of ATLAS however this system is essentially a versatile and powerful image reconstruction system adaptable for generic image processing applications. The AM system applies a filter that can be applied to images of different kinds. The AM-based processor can simulate the preliminary stages of image processing performed by the brain for vision, such as the identification of shape edges [6]. In the second two years of the project the main aim was in fact demonstrating the potentiality of the AM-processor to reconstruct medical images. In collaboration with various experts in medical image reconstruction the AM-system has been used to reconstruct Magnetic Resonance Imaging (MRI) of the brain with the aim of recognizing MRI area with sign of degenerative diseases. We have used our AM based pattern reconstruction to produce an image pre-processing that implements a 3D edge enhancing filter. This first stage allows a huge decrease of the image processing time (more than a factor 1000) [8, 9] giving the possibility to make this diagnostic exam applicable to a wider audience of patients.
[1] Annovi, A., et al. "VLSI Processor for Fast Track Finding Based on Content Addressable Memories" - IEEE Trans. Nucl. Sci. 53, 4, Aug(2006), 2428-2433.
[2] Andreani A. et al. "The FastTracker Real Time Processor and Its Impact on Muon Isolation, Tau and b-Jet Online Selections at ATLAS"- IEEE Trans. Nucl. Sci. 59, 2, 348-357.
[4] S. Citraro et al., “Highly Parallelized Pattern Matching Hardware for Fast Tracking at Hadron Colliders” - IEEE Trans. Nucl. Sci., 63, 2, Apr(2016), 1147 - 1154
[5] Sotiropoulou, C.-L. et al. "A Multi-Core FPGA-based 2D-Clustering Implementation for Real-Time Image Processing" - IEEE Trans. Nucl. Sci. 61, 6, Dec(2014), 3599-3606.
[6] IEEE Real Time 2016, Padova. C.Sotiropoulou et al.; “The Associative Memory System Infrastructures for the ATLAS Fast Tracker” to be published IEEE Trans. Nucl. Sci.
[7] Del Viva, M., Punzi, G., and Benedetti, D. 2013. "Information and Perception of Meaningful Patterns" - PloS one 8.7 (July 2013): e69154.
[8] IEEE Real Time 2016, Padova. P. Luciano et al.; “Brain Emulation for Image Processing” to be published IEEE Trans. on Nuclear Science
[9] IEEE NSS/MIC 2016, Strasbourg. C. Sotiropoulou et al. “Medical Image Processing Using Brain Emultion”. Proceeding to be published as IEEE Conference Record.
Work performed since the beginning of the project: electronic boards were developed for the FTK Processor [2], an approved ATLAS upgrade. The implemented system is based on the optimal mapping of a complex algorithm onto different technologies: a combination of high performances dedicated hardware and distinctive flexibility of general-purpose but lower-performance CPUs. A key role in the architecture is played by high-performance field programmable gate arrays (FPGAs), while most of the computing power is provided by full-custom ASICs named AM. We built a powerful highly parallel dedicated hardware that provides excellent performance, reaching resolutions, efficiencies and fake track rejection typical of offline algorithms. The latencies are short (about one microseconds), power usage is low (the AM chip, has a power consumption below 3 W for 1M fit/10ns), and the system is small (the boards contained in 4 racks perform a task that would require a farm of O(1k) of commercial CPUs). The AM, the central device of our system, store each pattern in a single memory location consisting of 8 independent words of 16 bits each. Each of these words is separately compared to its own input data stream [1]; the coincidence of the 8 words matches, found at any time during data arrival, determines the match of the pattern. FPGAs control, configure and handle the AMs, providing the flexible computing power to process the selected patterns. Distributed debugging and monitoring tools suited for a pipelined, highly parallelized structure and a high degree of configurability have been developed to cope with different applications.
Main goals achieved: We have built the AM [4] and clustering [5] system prototypes, tested and integrated them in the global FTK system at CERN. The design of the AM system was a particularly challenging task for the following factors: the high pattern density (8M patterns/board), required a large silicon area for the AM ASICs; the I/O signal congestion at the board level, required heavy use of serial links; and the power limitation due to the maximum cooling power cooling resulted in setting the maximum power to 250 W per AM board. The extreme challenge presented by the AM chip design forced us to adapt the project schedule however it did not impact any of the project milestones. The infrastructures (crates, heatsinks, fans, power supplies) were studied and optimized [6]. The described system was developed and tested for HEP detectors and it is expected to highly increase the physics discovery potential of ATLAS however this system is essentially a versatile and powerful image reconstruction system adaptable for generic image processing applications. The AM system applies a filter that can be applied to images of different kinds. The AM-based processor can simulate the preliminary stages of image processing performed by the brain for vision, such as the identification of shape edges [6]. In the second two years of the project the main aim was in fact demonstrating the potentiality of the AM-processor to reconstruct medical images. In collaboration with various experts in medical image reconstruction the AM-system has been used to reconstruct Magnetic Resonance Imaging (MRI) of the brain with the aim of recognizing MRI area with sign of degenerative diseases. We have used our AM based pattern reconstruction to produce an image pre-processing that implements a 3D edge enhancing filter. This first stage allows a huge decrease of the image processing time (more than a factor 1000) [8, 9] giving the possibility to make this diagnostic exam applicable to a wider audience of patients.
[1] Annovi, A., et al. "VLSI Processor for Fast Track Finding Based on Content Addressable Memories" - IEEE Trans. Nucl. Sci. 53, 4, Aug(2006), 2428-2433.
[2] Andreani A. et al. "The FastTracker Real Time Processor and Its Impact on Muon Isolation, Tau and b-Jet Online Selections at ATLAS"- IEEE Trans. Nucl. Sci. 59, 2, 348-357.
[4] S. Citraro et al., “Highly Parallelized Pattern Matching Hardware for Fast Tracking at Hadron Colliders” - IEEE Trans. Nucl. Sci., 63, 2, Apr(2016), 1147 - 1154
[5] Sotiropoulou, C.-L. et al. "A Multi-Core FPGA-based 2D-Clustering Implementation for Real-Time Image Processing" - IEEE Trans. Nucl. Sci. 61, 6, Dec(2014), 3599-3606.
[6] IEEE Real Time 2016, Padova. C.Sotiropoulou et al.; “The Associative Memory System Infrastructures for the ATLAS Fast Tracker” to be published IEEE Trans. Nucl. Sci.
[7] Del Viva, M., Punzi, G., and Benedetti, D. 2013. "Information and Perception of Meaningful Patterns" - PloS one 8.7 (July 2013): e69154.
[8] IEEE Real Time 2016, Padova. P. Luciano et al.; “Brain Emulation for Image Processing” to be published IEEE Trans. on Nuclear Science
[9] IEEE NSS/MIC 2016, Strasbourg. C. Sotiropoulou et al. “Medical Image Processing Using Brain Emultion”. Proceeding to be published as IEEE Conference Record.