Lead-free solders in soldered joints carry the risk of increased failure rates. Improved quality control is being used to counteract this. However, conventional methods of camera-based automatic image evaluation have reached their limits. A solution based on AI and deep learning was installed at Siemens Smart Infrastructure.
The conventional methods of camera-based automatic image evaluation have reached their limits because the user has to tread a fine line between excessively high internal or excessively high external error rates when defining the inspection criteria. In most cases, these are camera-based image processing solutions that use suitable software packages to make in-order/not-in-order (IO/NIO) classifications based on image analyses. Their selectivity has not always been satisfactory.
Inconspicuous lifesavers: smoke detectors are safety componentsSiemensSmart Infrastructure manufactures smoke detectors for fire protection in numerous variants and in medium to large quantities on automatic systems. The components used are placed on the circuit board using automatic placement machines and then soldered from above. The EU-wide ban on solder alloys containing lead has forced the use of lead-free solders, which, however, have poorer soldering properties than the solders previously used. This results in increased reject and failure rates.
This makes reliable automatic quality control systems all the more important. Particularly when used for critical safety functions, the test criteria must be trimmed to the 'safe' side, as fire detectors must be extremely reliable. However, this results in increased reject rates with corresponding cost disadvantages.
The Swiss specialist for industrial image processing and robot solutions Compar AG, which Siemens Smart Infrastructure relies on for quality control, therefore worked on ways to reduce reject rates: "We took the approach of also using solutions with 'artificial intelligence' in the form of self-learning neural networks for image analysis," says Lukas Vassalli, who holds a BSc in systems engineering from the University of Applied Sciences Eastern Switzerland (FHO) and is a developer at Compar in Pfäffikon. "It was also about integrating such tasks into higher-level IT structures as part of Industry 4.0 concepts."
AI integration
Compar integrates systems from Cognex as vision hardware. "The image processing specialist offers ready-made software packages as plug-in modules for such tasks under the name ViDi," says Vassalli. In the training phase, a powerful image processor (Graphics Processing Unit/GPU) should be a hardware requirement on the computer used. An essential component of the software library is a neural network, which is already partially pre-structured so that the user can quickly and easily start training. This is necessary before the first use and is done by providing the network with a certain number of images as 'training material'. It can then evaluate new images independently according to the desired criteria. The wealth of knowledge acquired during training is constantly expanded and refined over the course of use, which is expressed by the term 'deep learning'. The application described here involves not only the assessment of solder connections but also the detection of placement errors.
Complete system
Split good and bad part pattern: At the top of the image, the error was detected with a 0.99 error certainty (99 % NOK); at the bottom, the error was detected and evaluated as 'good' with 0.02 error certainty (2 % NOK, i.e. 98 % IO)
In 'supervised learning', the trainer indicates the error with a color marker
"The overall system consists of the camera, an illumination station designed for the application, which also holds the circuit boards, and an industrial PC with the Visionexpert program," says Vassalli. It is supplemented by the ViDi package, which works as a 'black box'. It analyzes the transferred images using its neural network and returns corresponding assessments. This is done instantaneously within milliseconds of the production line cycle. Before the start, the system was pre-configured by Compar using images of sample parts provided. During operation, the system can be trained by the user with new products or retrained with variants of existing products as required.
Thanks to the high computer performance, only a few minutes are required for such training phases. During training, the system can either be 'fed' directly with photos or, in supervisor mode, defects can be highlighted in advance using color markings. After brief training, the customer can carry out such tasks themselves - an essential prerequisite for the success of the project. In this case, around 50 images each of good and bad parts were sufficient.
The ViDi processes
All IO and NOK evaluations up to the respective extreme value are added up and then normalized to 1. The overlap is an indication of insufficient selectivity or the need for additional training. The influence of the threshold setting on the reject rate is clear
Example: Bar chart of the frequencies of IO and NOK assessments. FP = false positive, FN = false negative, TN = total negative, TP = total positive. The threshold value determines which parts are used or rejected
"The ViDi software consists of three modules (red, green and blue), of which the 'red' and 'blue' modules are used in this case," reveals Vassalli. The 'blue' module, known as the 'locator', checks the PCBs for correct placement. It identifies solder joints and component positions as well as imprints. Variances can be specified. ViDi 'red' then takes over the classification into IO or NIO parts. Different approaches can be selected for training, e.g. by specifying only IO parts instead of the two categories IO/NIO. In this case, the AI will automatically classify everything that is not clearly recognizable as IO as NOK.
Selectivity as a reliability feature
"An important feature of the ViDi analysis is the numerical evaluation of the classification of the respective result," says Vassalli. Although the system classifies reviewed images according to the criteria IO and NOK, it always outputs a percentage confidence value. This indicates the percentage of confidence the software has in its judgment. The scale ranges from 0 (= 100 % IO) to 1 (= 0 % IO or 100 % NOK). The frequency distribution of these classifications is displayed statistically in the form of diagrams with, for example, green color for IO and red color for NOK results. They take the form of two bar charts in green or red, which may partially overlap. A simpler representation is obtained by plotting the cumulative scattering ranges normalized to one.
Depending on the task and evaluation criteria, these can either partially overlap or form two clearly separate groups. If the training has run optimally, there is no overlap between the cumulative frequency ranges. This proves that the method has good discriminatory power. If this is not the case, you end up in the decision area between 'false-positive' and 'false-negative' classifications. In such cases, the optimal determination of the so-called Treshhold value plays an important role. If you place it more on the safe side, for example, you minimize the risk of failure of safety-relevant components for the customer. With the opposite strategy, on the other hand, it may be possible to reduce the internal scrap rate.
Inspection processes: The Visionexpert camera and lighting units provide an image of the circuit board. Visionexpert performs its part of the evaluation and at the same time initiates a parallel analysis by ViDi. The results are then incorporated into the Visionexpert evaluation. This results in control commands to the process PLC and messages to the company's higher-level IT structure
This overview prepared by Visionexpert shows a pleasingly wide gap between the IO and NOK assessments assigned. The results are therefore highly reliable
Interaction with Visionexpert
"The integration of the ViDi options described with the Visionexpert image processing software developed by us will be particularly interesting for customers," summarizes developer Vassalli. As the main component, the Compar program initially takes over the external hardware handling, i.e. the connection of the numerous possible camera models and other peripherals. Another task is image data management and the transfer of image data to be analyzed to ViDi. The returned results are used internally, visualized and finally integrated into the decision-making process. Despite all the automation, the human operator always retains the decision-making power by specifying inspection criteria and decision specifications such as the threshold level.
In addition to the results of the ViDi inspection, Visionexpert's own capabilities are also used to analyze and assess a test specimen. In contrast to the ViDi plug-in, this software can, for example, measure dimensions down to the µm range with high accuracy and make decisions based on the results. Finally, Visionexpert also handles communication with the company's higher-level IT. For Compar, which has been developing solutions for visual quality control for decades, the integration of the new AI-based tool is an important step towards further improving its offering.
