Traditional machine vision image processing techniques apply rules-based math convolutions to digital image data sets to extract insightful or actionable information. The information is used to monitor, manage, and verify industrial processes — from counting products traveling down a conveyor for inventory, to guiding robots during product assembly and packaging, to ensuring quality through automated visual inspection.
Developed over decades, machine vision image processing software programs have become incredibly powerful analytical tools used by virtually all manufacturing industries. Today, machine vision software is used in many industrial applications, such as automatically locating one object in a crowded image, making precise measurements at high speed, and analyzing parts for defects — all with greater precision and speed than any human operator. Additionally, each day sees more machine vision technology transitioning to consumer markets as part of embedded solutions for products as varied as interactive toys to facial recognition on cell phones.
But like all tools, machine vision continues to evolve in the never-ending search for manufacturing productivity gains. While machine vision provides advanced manufacturing capability, programming machine vision systems can be complex, require considerable expertise, and consume significant time. Furthermore, traditional machine vision systems excel at analyzing features on images that can be mathematically defined as good or bad. For instance, think of inspecting a million electrical cords to verify the presence of three conductors in every cord. However, when the product or defect varies considerably from one product to the next — such as determining whether a line on a cell phone cover is a defective scratch or merely a cosmetic smudge that can be removed easily — mathematically defining every product and defect is practically impossible.
Artificial Intelligence and Deep Learning
Today, artificial intelligence (AI), also referred to as deep learning in the machine vision industry, is helping to overcome these barriers by defining
good and bad products through the automated development of a digital model that includes information on every defect condition captured in a set of images, regardless of number. These models can be created in hours or days rather than requiring the weeks or months it would take a human programmer to define every defect by position, size, orientation, and so forth.
But even deep learning cannot solve every machine vision challenge by itself. The most powerful machine vision solutions today use a hybrid model that combines traditional 2D and 3D machine vision software with the enhanced defect classification capability of deep learning software. Best-in-class hybrid machine vision solutions combine 2D, 3D, and deep learning software programs with motion control systems to automatically scan a part, create a digital twin, self-optimize the traditional machine vision software to acquire the highest quality images, and make basic measurements. These systems then apply deep learning software for defect identification and classification, reducing false positives and negatives compared to either manual inspection or manually programmed machine vision solutions. As a result, system integrators are solving manufacturing’s most complex quality assurance challenges with hybrid machine vision solutions that only took hours or days to develop, compared to the weeks or months it takes to develop a traditional machine vision solution — and with higher reliability than ever before. This paper will look at how best-in-class hybrid machine vision systems use traditional machine vision algorithms combined with AI/deep learning software to simplify the programming of machine vision solutions for complex inspection tasks.
THE COST OF POOR QUALITY
Humans make mistakes in manual visual inspection. For example, tired workers might miss defects that escape quality screens on the production floor. When defective products make it beyond the factory, manufacturers pay a steep price. The “cost of poor quality” includes the costs of returned or rejected goods, scrap, rework, and in many cases, the negative impact on brand reputation and customer satisfaction. In many supplier-customer relationships, a single defective product could lead to rejection of an entire shipment and potential financial penalties.
A recent Tata Consultancy Services survey listed product quality and the tracking of defects as the major benefits of using big data in manufacturing. Defective products cost major industries on average between 4 and 7% of gross revenue. Rising labor costs also represent a problem in most markets and industries. In markets such as machine tending and welding, a lack of skilled workers presents further challenges. Such shortages of quality-control employees can create a production bottleneck, which negatively impacts revenue.
Deep Learning versus Machine Vision: Stronger Together
Artificial intelligence, or more specifically deep learning, is a new method for programming machine vision systems that removes the need for the designer to define the defect. Rather, experts review images, find and locate potential defects, and label the image as either “good” or “bad.” Deep learning software analyzes from dozens to thousands of these good and bad images, and — much like a human — learns to tell the difference by itself. This step is called “training.”
After the deep learning software creates a model of the part, the software is ready to be deployed as part of a complete industrial machine vision solution. This step is called “inference.” While training is computationally intensive, often requiring server- or workstation-grade computers to generate the deep learning model, inference can be done on the plant floor using commonly available industrial PCs and/ or smart cameras, which contain machine vision software and computational elements inside a unified camera housing.
However, as stated earlier, few if any machine vision applications can be solved by deep learning software alone. For example, traditional machine vision software excels at finding the edge of an object in an image; it can accomplish this task much faster and with less effort than a deep learning machine vision software program. Traditional 2D machine vision software also is effective at making measurements, such as looking 10 mm to the left of an edge to find a specific machined hole for inspection. Traditional 3D machine vision excels at measuring the depth of the hole, or the degree a screw projects from the part — both of which can be a defect condition if not manufactured and assembled to a specific tolerance.
Best-in-class hybrid deep learning machine vision solutions are tightly integrated with traditional machine vision software, making it easier for designers to implement traditional algorithms and pass the processed data to deep learning defect classification software routines. Furthermore, the top echelon of these design environments includes machine vision solution design expertise that — when combined with mobile image capture systems — allows the vision system to self-optimize the best location, camera settings, and illumination techniques to generate the highest quality images for the specific region of interest (RoI) under test. Finally, deep learning software classifies potential defects with accuracy equal to or better than human inspectors and reliability and speed that human operators cannot achieve.
Hybrid machine vision systems, such as the Kitov Planner, follow a six-step self-optimizing design process:
1. Planner software starts with 3D model, CAD file, or digital twin
2. User draws boundary boxes around potentially defective RoI.
3. Planner develops inspection path, including motion control movements and optimal hardware settings for cameras and optics for each unique RoI based on embedded design expertise.
4. Software optimizes inspection plan to include as many RoIs as possible within each image to reduce inspection times and maximize throughput.
5. Software passes RoI information to deep learning software, which learns to classify defects based on previously inspected images.
6. Software is tested against expert inspectors; if inspection reliability is below tolerance, additional tagged defect images are feed into the deep learning system until tolerances are achieved.
3D automated inspection combined with deep learning software enables hundreds of inspections per assembled product. Among other areas, the system inspects all six sides of the product, the silkscreen and pad printing, product edges, light-pipe indicators, screws, labels, the communication port housing, and pins.
Hybrid Vision: Powerful Analysis and Simple Programming
Deep learning and hybrid machine vision solutions are the best — and often the only — solution for automated complex assembly inspection tasks. But how best to solve the application using the two software design environments? Does deep learning software eliminate the need to design high-contrast image acquisition systems to boost the reliability of both traditional machine vision and deep learning image processing? These are not trivial questions.
While a few suppliers do offer unified hybrid machine vision design environments that make it easier to pass data from traditional 2D and 3D algorithms to deep learning software routines, most simply add the complexity of deep learning onto the preexisting complexity of a traditional machine vision software program. Few take a fresh approach to the hybrid machine vision design environment.
For example, most traditional machine vision software programs have adopted “object based” programming approaches that let the designer quickly drag and drop individual algorithms into a flow chart. Tolerances and thresholds for each algorithm are set manually through a spreadsheet program or similar construct that is intuitive to most regular computer users. Some of these machine vision software programs also include pretrained deep learning models. Pretrained deep learning models can be quickly optimized from recognizing a generic “screw” to a specific screw.
But wouldn’t it be better if the self-optimizing routines were applied to traditional machine vision settings as well as the deep learning models?
For example, Kitov’s Planner hybrid machine vision software design environment moves beyond flow chart–based traditional programming with manual tolerances to Semantic Detectors. Semantic Detectors combine multiple traditional machine vision algorithms with common image acquisition and tolerance settings for standard machine vision tasks, such as inspect “screw,” “label,” “surface,” “barcode,” “gaps,” and so forth.
Furthermore, Kitov places the camera, optics, and illumination on a moveable platform — specifically, a robotic arm. By moving the camera around the RoI and changing illumination settings, Kitov Planner can self-optimize the image acquisition process for each RoI without programmer intervention. Once every RoI location is defined, the system can then strive to reduce cycle times by combining as many of the RoIs in a single exposure as possible. Because the image acquisition and robotic motions take the majority of the cycle time for a Kitov inspection routing, combining these two self-optimizing steps for traditional 2D and 3D machine vision image acquisition regularly eliminate days to weeks of development time per application.
Quality Inspection You Can Trust
Hybrid machine vision solutions are not for every machine vision application. ABI Research predicts that of the 94 million machine vision cameras that will be installed worldwide by 2025 (an estimated 16.9 million cameras being installed in 2025 alone), 11% of them will use deep learning capability.
The sweet spot for hybrid machine vision solutions that use deep learning capabilities is complex assemblies and products. Recently, a power inverter for an electric vehicle with more than 100 inspection RoIs was solved within weeks using Kitov Planner with a base set of 200 tagged images. Engineers had previously spent years trying to solve the application using other hybrid software solutions.
Bent pins inside computer I/O ports are among the most common reasons electronic products are returned to the manufacturer as defective. However, inspecting fine wires at the bottom of cavities with reliability and accuracy requires a 3D inspection system, often with additional capabilities, such as deep learning defect analysis. This programming interface begins with the system designer loading a CAD file (shown here) or “golden part” image. The designer can then draw regions of interest around important inspection points. Advanced hybrid solutions evaluate each region on the product to determine the right combination of traditional and deep learning algorithms for inspection.
Kitov Planner achieves previously unattainable automated inspection reliability levels in a fraction of the time needed to design solutions in competing environments because Kitov Planner comes with numerous pretrained models of screws, labels, gaps, ports, and more, reducing the number of images and time required to optimize the defect classification step, and comes with the Semantic Detectors embedded system design expertise. The software includes the ability to add additional pretrained deep learning models and even develop new Semantic Detectors for customer-specific needs.
Finally, deep learning software simplifies the introduction of new products. By simply showing the deep learning training program pictures of the new components, the machine vision solution has the ability to adapt the new component within minutes, or hours, depending on the number of new images to analyze. Traditional design approaches would take additional days, weeks, or even months to solve the same application.
Defects from an automotive wheel/rim are compared to a picture of the same area of a “golden,” defect-free product (outlined in green). The left side shows scratches on the sidewall of the wheel, and the right side shows a dent on the wheel’s edge that is difficult to find with other inspection solutions.
