In the course of digitalization, more and more data is being generated in companies. It is not enough to simply collect this data; it needs to be processed and clearly visualized in order to generate added value. Augmented reality (AR) is a technology that uses an AR app for smartphones to make previously invisible information visible at a specific location, for example on systems. AR can also be used in the context of surface technology - and creates added value.
Augmented reality - definition and significance for the industry
Fig. 1: AR data panel on a smartphone shows system data directly above the systemNew information is constantly flowingatproduction systems in an electroplating plant: which order is currently being processed? What is the current bath current? What is the temperature? To communicate this information to employees, it is printed out or displayed on a monitor at the plant. A new approach to making this information visible in real time is augmented reality (AR). Employees take their cell phone, start an app and then see all the information that is important to them in the camera image directly at the plant, always up-to-date and without an additional screen (Fig. 1).
The fact that new media have the potential to change the world of work has not only been a fact since the fourth industrial revolution, digitalization. Whether telegraph, telephone or computer, when new media find their way into the world of work, processes change. AR is no exception. Augmented reality can be defined in different ways, with Goh et al. providing a practical definition (translated into German here):
"Augmented reality 1) combines real and virtual imagery; 2) is interactive in real time; and 3) displays the virtual imagery in the real environment [1]."
Many people's first contact with AR was probably not in an industrial environment, but rather during leisure activities. One of the most prominent examples of the use of AR is "Pokemon Go", a game app in which users can find a variety of virtual creatures in parks or in the city and photograph them in the real environment using a camera [2]. Other use cases include tourism apps with information on cultural attractions [3] or museums that send their visitors on an exploration tour with tablets [4].
These examples demonstrate the great advantage of AR: visualization of digital information in a real environment. As is so often the case, processes can be rethought with innovative technologies in the context of digitalization. It therefore stands to reason that AR can also be transferred for use in industry, for example in in-house logistics [5] and for maintenance tasks [6]. The fundamental question is: where in the company is there information that is not yet or only insufficiently visible?
AR in surface technology - predictive maintenance, increased process reliability SmARtPlaS research project
The first starting point was to visualize system and maintenance data on site in production in order to enable predictive maintenance. This was one of the declared aims of the SmARtPlaS research project (Smart Augmented Reality Plating Services, Fig. 2).
Fig. 2: Logo of the SmARtPlaS project - which still focuses on smart glasses; smartphones proved to be a better alternative during the project
In the context of SmARtPlaS, the initial aim was to define suitable criteria for how AR could bring benefits for surface technology. In cooperation with partner companies from research and industry, an overall concept was created that takes user feedback into account, as well as criteria for the connection of system controls and exhaust air systems [7].
With the help of the AR app prototype developed on this basis, virtual data panels can be placed in the room and live information can be displayed via interfaces to systems. One challenge was the usability of the app's own editor, which was optimized with the help of the results from a user study [8].
The decision was made early on in the project to develop the prototype for smartphones and not for smart glasses. The ubiquitous availability of smartphones and the experience of future users with smartphone operation are two of the most important aspects for this decision. A detailed comparison between smartphones and smart glasses follows in the Hardware chapter.
Possible applications
After the first experiences with AR in the SmARtPlaS project, it quickly became clear how flexibly the app can be used. In addition to being used for maintenance tasks and displaying technical information, there are various other possible applications for which AR is also suitable.
Training/instruction for employees: Similar to an AR application in a museum or in tourism, data panels can also convey text information and images to the user. This can be used for training purposes, for example. Information on a work step can be anchored in the appropriate place, allowing employees to use the instructions on the virtual boards as an assistance system. For example, picture stories for packaging, safety instructions for operating systems and much more are conceivable. Instructions on the data boards could even be accompanied by short video clips showing the work step. This means that employees always have help on their smartphone without having to search for the right instructions. Notes and instructions can be placed wherever they are needed.
Control center for other apps: If system data has been checked and intervention is necessary, an AR app can also serve as a direct springboard into other apps, for example for system control. If such a link is available, employees can make adjustments directly to the system, then switch back to the AR view and immediately check whether the desired change has been made. This direct feedback saves time and increases process reliability. Another possible connection is a (production data acquisition) PDA app for progress reporting.
Retrofitting systems: When systems are modernized, there is often little space left for new technology or the installation of monitors. AR is therefore particularly suitable for retrofitting in order to visualize new information gained through a modernized control system at important points without having to create new space in production.
Virtual product presentations: To present products such as components to a customer, informative data panels can be used to display product descriptions and technical data. It is also conceivable that AR could be used to present 3D models of components completely virtually, for example to show the finished component in sales meetings long before it is manufactured.
Further applications: In short, AR has a wide range of applications in industry that can be integrated into many environments and work processes. The extent to which AR is used in a company can therefore be flexibly and gradually adapted. Overall, it can be used wherever information needs to be made visible without having to resort to paper or permanently installed monitors.
Generating added value with AR
In order to justify the use of a new technology, it must generate added value for a business, be cost-effective and have a measurable benefit. In the case of AR, there are several advantages, some of which differ depending on the area of application. Here are a few factors:
Greater process reliability: if plant data is monitored on site in production at all times, it is possible to react more quickly to deviations and thus ensure consistently high process and therefore also product quality. If AR is used as an assistance system for employees, this supports employees in carrying out individual work steps correctly, which increases process reliability.
Capture information faster: The visualization of data or information with the help of AR primarily ensures that relevant information is visible at all - and in the appropriate place. As a result, this information can be recorded more quickly, eliminating the need for a long search for the correct values. The relation of the information to the location is particularly practical because an app is not limited to one target group. It can be used equally by a plant operator, worker or maintenance team.
Reduce hardware costs: Especially in the harsh industrial environment of surface technology, where galvanic baths and other chemicals are used in production, the durability of monitors and terminals is limited. Monitors therefore not only have to be purchased, but also regularly maintained and replaced. By using virtual "monitors" via AR, the number of monitors and display panels required can be reduced. Instead, information is available via the virtual data panels.
Sustainability: Saving on hardware or using existing hardware is also advantageous from a sustainability perspective. Switching from permanently switched-on monitors to smartphone displays also saves energy. The smartphone display does not have to be active all the time, but only when it is needed. Information is not constantly displayed, but is available as "information on demand".
Innovation branding: A soft factor that emerged from user discussions in the SmARtPlaS project is the external image as a particularly innovative company. If new innovative technologies are used and this fact is communicated to the outside world, this can be a factor that has a positive effect on employee recruitment.
Increased efficiency: The availability of real-time data directly at the workplace saves walking distances and information for employees is never out of date. This increases the efficiency of individual work steps.
Increased safety: In addition to increased efficiency, safety for employees is also improved. The operating status of machines can be read easily - from a safe distance for all systems - via smartphone; the risk of injuries and accidents is minimized.
Simplified maintenance: If maintenance information is displayed on site, it can be carried out more quickly and easily. Data no longer has to be read from terminals or requested from colleagues.
Flexibility: Augmented reality can be used with a wide variety of machines, regardless of manufacturer and model. Although some modern systems have their own apps, these are isolated solutions for individual systems. AR enables a comprehensive overview of the entire machine park.
Technical requirements for the use of AR
What technology is required to use an AR application and how does the creation of data panels and a spatial model work?
Hardware
The hardware requirements are divided into two areas: Firstly, a device is required on which the AR application runs and secondly, a server that provides the data for displaying the information. A smartphone app was developed for the SmARtPlaS project. Alternatively, smart glasses can be used, but these still have some serious weaknesses, at least today, and therefore cannot compete with smartphones.
Data glasses, such as the Microsoft HoloLens, display information permanently in the user's field of vision and can be operated using gestures or voice control. The weaknesses are that data glasses are significantly more expensive than smartphones and have so far been rather fragile, which makes them even more difficult to use in industrial environments. Short battery life and the unfamiliar operation of the glasses were further reasons for the decision to use smartphones.
However, data glasses are already well suited to remote maintenance tasks, where support is connected via video call and can directly see the perspective of the technician on site [9].
Smartphones are all-rounders that we are already familiar with from everyday life. They have a number of features that make them highly suitable for the use of AR. Important aspects are availability and user-friendliness. As smartphones are often used anyway, there is no need to purchase new devices, which keeps hardware costs low. Employees are already familiar with using these smartphones, as industrial apps can also be operated in a similar way to familiar everyday apps. The operation of the AR app can therefore be explained in a very short time. Another factor is the flexibility of the smartphone. If it is not needed, it can be put in your pocket and does not get in the way. Data glasses, on the other hand, can be annoying, are cumbersome to transport and always provide the image in the user's field of vision.
In addition to displaying the AR app on an end device, the desired data that is later to be visible in the app must also be transmitted. First and foremost, this requires sensors or machine controls, for example, to record the data on site. This data is sent to a server via an interface, which forwards the data to the AR app. A so-called Message Queuing Telemetry Transport (MQTT) server can be rented or hosted independently for this purpose.
Software
Only a few steps are required for a finished data panel to be visible in the app:
- The data is transferred from one or more systems.
- The data is prepared for display on the data panel.
- The data must be displayed in the camera image in the appropriate place.
The individual steps are described in detail below.
Data transmission with an MQTT broker
Fig. 3: Formatted demo data in the AR app with integrated tolerance limits. All these configurations take place in a Smart Factory Interface, which is used to describe interactive displays. These do not have to relate to AR; for example, they can also serve as the basis for a control station Data exchange between plants, control systems, enterprise resource planning (ERP) systems, monitoring and warning systems and much more takes place via interfaces that make it possible to transfer data from one system to another. Just a few years ago, this exchange was complex and by no means a matter of course due to the many different formats.
To avoid the need for individual developments for each interface, there are standard protocols for message transmission. Today, hardware is often already equipped with them and the protocols are used in many control systems. In short, there is a process of progressive standardization. The AR app also uses such a standard: MQTT.
MQTT stands for "Message Queuing Telemetry Transport" and is an open message protocol for machine-to-machine communication. IBM originally developed the protocol to monitor an oil pipeline, where special requirements were placed on the transmission protocol [10].
At the heart of an MQTT infrastructure is a server called an MQTT broker. This is where the data received is managed and forwarded, similar to a post office. As with a letter, there are senders and recipients, in this case called "publishers" and "subscribers".
In our example, publishers are systems that actively transmit data to the broker via an MQTT interface, which can then be retrieved by any number of recipients. Subscribers subscribe to exactly the information packets that are relevant to them; the broker only forwards the desired information. Senders and receivers are therefore clearly separated and do not communicate directly with each other, but use the broker instead.
The MQTT protocol is structured in such a way that sent values are always linked to so-called "topics", which provide a context. To simplify the exchange of data, no units or similar references are sent with a value. For example, if the topic "temperature" is referenced and a value "45" is sent, recipients only know that it must be 45 °C via the context.
Another advantage of the MQTT protocol is that only the most recent data value is ever stored on the server. This means that data on the server is not persistent; old values are simply overwritten. This increases data security. It also reduces the hardware requirements and the administration effort.
With an MQTT interface, systems, devices or sensors send values to a server, which forwards them to the AR app. The data is read out there and can be further processed for display. One problem with machine-to-machine communication is that data is often difficult for humans to read or - due to a lack of contextual information such as units - is simply confusing. This is why the data must be prepared in the next step.
Smart Factory Interface - displaying data clearly
In order to create individual data tables from the transmitted data, it must be defined for each data table which values are to be displayed and how. Particular emphasis was placed on ensuring that the data tables can be implemented in a short time, which is why the simplest possible interface was developed. Outsourcing the creation of new data tables to an interface also offers the advantage that the app itself does not have to be adapted - which reduces the effort involved.
When a new data table is to be created, it is a matter of formatting and structure, for example with headings, texts, images, suitable colors, buttons or buttons. When configuring the tables, tolerance limits can be set for individual values, such as the temperature of a system, which the value should not exceed or fall below. These can be color-coded, for example, by changing the value from green to orange or red (Fig. 3).
With these specifications, a data table can be set up individually so that exactly the values that are relevant can be seen. The data tables can also be quickly customized if additional values are to be displayed.
This completes the content of the data tables. Now they just need to be placed in the right place. This step is carried out in the app itself.
AR app editor - placing data panels in the room
Fig. 4: A wall is transferred to the model in the editor. This superimposition of the virtual and real world - known as "mapping" - creates a natural impression for the viewer. The digital overlay becomes an extension of realityTheAR app is divided into an editor mode and a viewer mode. This makes it more user-friendly. End users do not have to operate the editor, but only use the app to view the data panels. The functions for placing data tables are therefore not required. Placement takes place exclusively in editor mode. A user study was carried out to make the editor as easy to use as possible [7].
In order to be able to spatially locate data panels, the app requires a zero point that does not move. This takes the form of a marker image, usually a QR code, which is stuck to a suitable location in the room. The first step is to scan this marker to create a model. Data panels can then be freely placed in relation to this point. However, this also means that if the marker is moved, the entire model shifts; data panels no longer hang where they should be.
To place a data panel, it needs an anchor point in the model, which is represented as a small sphere. In the real world, this is comparable to the activity of hanging up pictures in a room. A hammer and nail are used to select the place where the picture is to be hung. With the virtual data boards, the app is used to define a point in the room that takes on the function of the nail. The data board is fixed to this point. This basic functionality makes it very easy to use the editor and quickly place individual data panels. At the same time, more complex functions make it possible to model the environment in order to provide an even more realistic experience for the user.
In the real world, we can only see images if they are not obscured by a wall or obstacle. The AR world is even more realistic if it can replicate this effect. This means that data panels only appear when the viewer has actually walked around a wall or entered a room through a door.
For this purpose, objects such as walls, furnishings or machines that restrict the viewer's view depending on their location can also be included in the model in the AR app editor. For this purpose, simple geometric primitives (one- or multi-dimensional shapes) are placed in the room, which block the view in the same way as real bodies (Fig. 4).
AR app viewer
Once the model has been created, all users have to do is start the app and scan the QR code marker. This action triggers the data panels to be loaded and displayed in the right place. This means that the information is available directly in the camera image and the smartphone can be panned from system to system to view the various values.
Several QR code markers can be useful to create additional entry points for the AR app, especially for plant complexes where longer distances are covered. For example, a QR code can be made available at each plant, which can be used to start the app. This gives employees the flexibility to access the app anywhere without having to walk. The markers can also be used to recalibrate the app if data is not loaded correctly. With the AR App Viewer, users can not only view the data, but also interact with it via buttons on data panels. Buttons can contain links to other apps, for example for production data acquisition or system control, so that deviations can be corrected directly.
The advantages of this interaction can be illustrated using an example: A data panel is set up for a system in the AR app, which displays the system's operating temperature, among other things. Tolerance limits are used to define values above which the temperature changes from green to orange to indicate that the tolerance limit has been exceeded. An employee with the AR app notices the low temperature of the system in passing. Using a button on the data panel, he calls up a maintenance app in which he raises the temperature back to the desired value. Back in the AR app, the employee checks the temperature again and the display on the data panel is updated dynamically. It is therefore possible to check the change made immediately.
Simple "switches" directly in the AR app are also conceivable, which can be used to activate or deactivate heating elements on systems, for example. To cover this case, however, information must be exchanged in both directions via the MQTT server. The system must receive changed values that are sent by the AR app. Alternatively, this functionality can be solved via a so-called Representational State Transfer (REST) interface, which is called up in the AR app.
Background to AR technology
From a mathematical point of view, the application is challenging. Objects not only have to be placed in space, their rotation around all three axes also has to be recorded. This poses problems that navigation systems in aerospace or submarines also have to contend with. There are conditions, such as the gimbal lock, which cannot be easily solved with simple vector calculations from analytical geometry. These complications can be avoided with quaternion algebra. In many cases, quaternions allow a mathematically elegant description of three-dimensional Euclidean space, especially in the context of rotations.
On 16 October 1843, the mathematician Sir William Rowan Hamilton had an epiphany while out for a walk in search of a generalization of imaginary numbers. In order to record his thoughts on the spot, he spontaneously carved the multiplication rules for quaternions into the sandstone of Broom Bridge in Dublin, where a plaque commemorates the event to this day [11].
This highly complex mathematics stands in contrast to the result: the representation in the AR app, which behaves naturally, as we would expect according to the laws of physics. The medium can thus be understood as an extension of reality that we can use to make information visible that would otherwise remain hidden from us.
Understanding new media as an extension of our perception, our reality, is a concept that has existed not only since augmented reality as a digital application. Analog examples also show how quickly the human brain adapts extensions and grasps them as a natural part of sensory perception.
This requires a universal concept of media that does not just include technical media. A universal definition of media comes from Marshall McLuhan, who sees media as "extensions of man". In this sense, media enable an extension of the sphere of influence or an expansion of perception or communication possibilities [12]. Tools such as a baton or binoculars can then also be described as media. Items of clothing can also perform this function, for example the tutu in ballet. This shapes the perception of distance to other dancers, the skirt becomes a medium [13]. Tools therefore not only remain tools, but also change our actions and our sensory perception.
Potential of AR in surface technology
To summarize, augmented reality is no longer a dream of the future. The technical foundations are in place and the potential applications are diverse. In surface technology in particular, there are many areas of application in the production environment. Previous hardware solutions are either enclosed in control cabinets and are not directly accessible or devices must have IP classifications, which further increase the cost of hardware. In this harsh, aggressive industrial environment, where high temperatures, humidity and chemicals affect hardware and electronics, it is therefore a smart solution to use less hardware overall. With AR, smartphones are used, which are particularly suitable hardware because they are not permanently exposed to the environment, but only when they are needed. This not only replaces monitors, but also drastically reduces the cost of replacing broken hardware.
Another factor is scalability: a single monitor can only ever hang at one position and display values there. If all systems are to be equipped with display panels, each additional system requires an additional monitor. AR, on the other hand, is scalable: the app can place any number of data panels, all of which can be displayed with the same device, exactly where they are needed. As part of the SmARtPlaS project, these and other possible applications were developed and tested with an app prototype. As of today (March 2023), SmARtPlaS has resulted in several projects in which the AR app is being used in practice in various contexts, for example for exhaust air, electroplating, waste water, coating and drying systems. Softec AG from Karlsruhe is also working on making its ERP solution Omnitec "AR ready" so that not only plant data but also data from the ERP is available for display in the AR app. This means that PDA messages and order data, for example, can also be visualized on data panels.
Literature
[1] Rauschnabel, P.; Rossmann, A.; Dieck, C.: An adoption framework for mobile augmented reality games: The case of Pokémon Go, Computers in Human Behavior, 76 (2017), pp. 276-286
[2] Han, D.; Jung, T.; Gibson, A.: Dublin AR: Implementing Augmented Reality in Tourism, Information and Communication Technologies in Tourism 2014, pp. 511-523
[3] Jung, T.; Dieck, C.; Lee, H.; Chung, N.: Effects of Virtual Reality and Augmented Reality on Visitor Experiences in Museum. Information and Communication Technologies in Tourism, 2016, pp. 621-635
[4] Wang, W.; Wang, F.; Song, W.; Su, S.: Application of Augmented Reality (AR) Technologies in inhouse Logistics, E3S Web of Conferences, 2020, 145. 02018. 10.1051/e3sconf/202014502018.
[5] Palmarini, R.; Erkoyuncu, J.A.; Roy, R.: An Innovative Process to Select Augmented Reality Technology for Maintenance, Procedia CIRP, 59 (2017), pp. 23-28
[6] From electroplating 4.1 to intelligent, service-based electroplating operation, In: WOMag, (2022) 5
[7] Hellmuth, C.; Bachinski, M.; Müller, J.: Interaction Techniques for 3D-positioning Objects in Mobile Augmented Reality. In Proceedings of the 2021 International Conference on Multimodal Interaction (ICMI '21), Association for Computing Machinery, New York, NY, USA, 2021, pp. 604-6102
[8] Introducing Dynamics 365 Remote Assist for HoloLens 2 and mobile devices, accessed 29.07.2022, https://www.youtube.com/watch?v=d3YT8j0yYl0
[9] Transcript of IBM Podcast. November 2011, retrieved on 29.07.2022, https://www.ibm.com/podcasts/software/websphere/connectivity/piper_diaz_nipper_mq_tt_11182011.pdf
[10] Altmann, S.: Hamilton, Rodrigues, and the Quaternion Scandal, Mathematics Magazine, Dec. 1989, 62 (1989) 5, pp. 291-308
[11] McLuhan, M.: Understanding Media: The Extensions of Man, McGraw-Hill Book Company, 1964
[12] Ommeln, M.: Der Cyborg, augmented reality, Google Glass und ihre Umschriftung als Leinwand: Technikphilosophie auf der Grundlage einer Philosophie des Tanzes, Techne, Poiesis, Aisthesis: Technik und Techniken in Kunst und ästhetischer Praxis, IX Congress of the German Society for Aesthetics, February 2015
[13] Goh, E.S.; Sunar M.S.; Ismail A.W.: 3D object manipulation techniques in handheld mobile augmented reality interface: a review, IEEE Access 7, 2019, pp. 40581-40601
