In today’s automobiles, manufacturers competitively blend performance and custom features with cutting-edge safety technology. Tire-pressure monitoring, proactive rollover prevention/mitigation, adaptive headlights and/or night-vision assistance, smart air bags, and emergency response features are becoming industry standards, dramatically increasing our odds of accident survival. Similarly, sensor- or vision-based systems, such as adaptive cruise control and collision mitigation, blind-spot detection and collision warning systems, lane departure warning systems and rear-view cameras for back-up assistance, are quickly becoming mainstream.

In the cost- and risk-sensitive automotive market, vision-based and image-recognition systems are likely to experience strong growth over the coming years due to their cost advantages and multifunction capabilities. For example, cost-effective vision-based systems can offer road user/object classification in addition to highly precise detection, making them ideally suited for blind-spot detection, emergency braking and lane departure warning. More advanced systems can analyze traffic lights or read traffic signs, resulting in even further protection or providing helpful navigation information for the driver. And, with automobile manufacturers integrating radio detection and ranging (radar) or light detection and ranging (lidar) for sensing in addition to vision-based sensors, it is clear that enhancements to the existing vision-processing environment are required. In particular, support for running multiple vision- or sensor-based applications simultaneously is key.

To accommodate these requirements, various issues surrounding the vision processor must be addressed, including true real-time performance and power efficiency as well as software flexibility to account for various applications, range of recognition targets, and changing lighting and weather conditions. While parallelization through single-instruction/multiple-data (SIMD) architectures has speeded up the initial processing of an image, these processors have limited efficiency for the final image-processing steps, which are mainly serialized or require floating-point arithmetic. A new alternative is the utilization of a processor that offers efficient SIMD processing and also can be reconfigured on the fly. This enables sequential processing in multiple-instruction multiple-data (MIMD) operation, thereby providing an efficient and cost-effective way to support multiple applications running simultaneously to provide more comprehensive driver-assistance information.

The Alternatives – SIMD, MIMD or Mixed-Mode

Vision or image-recognition tasks require large amounts of data-level parallelism and real-time responses. SIMD elements are recognized as the most efficient for algorithms that can be designed with a highly parallel structure. The number of operations that can be done in parallel is theoretically limited only by the number of blocks available for processing. SIMD processors can manipulate large amounts of data in a highly efficient manner, enabling implementation of operations in software that conventional digital signal processors (DSPs) find cumbersome. For these reasons alone, the SIMD option can be the best choice for some applications to maximize performance without raising total cost. As a result, a SIMD environment is advantageous for image processing steps that require the same operations to be done on groups of pixels simultaneously. For typical image processing, SIMD is used for image scaling, image filtering and basic image-detection functions.

As mentioned, SIMD processors have limited efficiency for the final image-processing steps that are either mainly serialized or require floating-point arithmetic. Additionally, when compared with general-purpose processors, the reduced amount of control circuitry of a highly parallel SIMD processor reveals a flexibility gap. 

While MIMD processors have their shortcomings for processing data in parallel, they are very well suited for processing steps that can be parallelized on the thread level, but not on the data level. In a typical image-recognition application, the primary filtering and detection will identify areas of the image that may contain useful information. The resulting processing of each area, however, will depend on the result of the primary detection. MIMD architecture enables handling of multiple areas in parallel, with each area processed by algorithms that are tailored toward the potential detection target. As an example, an area identified as a potential traffic light would be processed for confirmation through its shape and for its color, while an area identified as a potential road object would be processed for classification of the object (i.e. vehicle vs. bicycle vs. pedestrian). As some of these algorithms are based on floating-point arithmetic, using floating-point units (FPUs) can significantly improve their processing times.

Developed to eliminate the issues faced when using a pure SIMD architecture, modern mixed-mode solutions provide a robust platform for vision-processing applications. These devices can be used for parallel preprocessing of image data in SIMD mode and then reconfigured on-the-fly to sequentially process different execution threads for the required application in MIMD/multiprocessor mode, providing efficiency with the greatest amount of flexibility. This dynamic reconfiguration is made possible by a slight increase in logic circuit size, which enables a device to process all processing elements of vision-processing algorithms most effectively in the desired mode of operation.

Figure 1. In this example of a reconfigurable mixed-mode architecture, half of the processing elements are combined in groups of four to act as processing units, while the other half of the processing elements operate as SIMD elements.

The reconfiguration enables the core to operate as a SIMD processor (with N processing elements), as a MIMD/multiprocessor processor (with N/4 processing elements), or as a mixed-mode processor (with N/2 processing elements and N/8 processing units). The hardware reconfiguration involves combining four processing elements and their memory blocks to act as a complete processing unit. Each of these processing units therefore acts as an independent processor core, using the processing element’s internal memory as program and data cache. In addition, each of these processing units includes its own FPU, providing the desired acceleration for floating-point routines.

As shown in Figure 1, in the mixed mode, half of the processing elements are combined in groups of four to act as processing units, while the other half of the processing elements operate as SIMD elements.