augmented reality techniques for vehicle maintenance

PROCEEDINGS OF THE INSTITUTE OF VEHICLES 2(102)/2015
Marcin Januszka1, Wawrzyniec Panfil2
AUGMENTED REALITY TECHNIQUES FOR VEHICLE MAINTENANCE
1. Introduction
Nowadays, designed and manufactured products are becoming more and more
complex. This is particularly noticeable in the field of mechatronic devices, especially
vehicles. Knowledge associated with such complex technical objects (e.g. with their
servicing and maintenance) is also becoming increasingly complex and difficult to
interpret. Systems supporting servicing of such devices must meet the increasing
demands, what in effect should give savings of time and costs. This can be achieved
through more efficient delivery of knowledge and information to the service personnel.
New demands force the replacement of text semantic information grains with visual and
interactive information grains [8]. Augmented reality systems are the perfect solutions,
which allow for the integration of text grains and interactive models in a common
interface. Augmented reality systems allow to superimpose on a current state of a real
technical object some virtual objects related to information which is important in the
process of such exploitation.
Augmented reality technology was born as a result of drawbacks and imperfections
of virtual reality techniques. The main idea of AR is to facilitate the execution of a task
by the user in the real world and in the context of real objects, that do not provide a
virtual reality techniques. For example, AR system can provide technicians with
information on how to properly perform the process of installing a door in a car in the
context of a real and not a virtual car [6]. This goal is achieved through the provision of
information by several communication channels. It is thus possible visualization (via
video channel) of assembly procedures of a virtual door on the real car, as well as
providing additional sound information. All information is precisely provided at the
moment when there is demand for it (just-in-time) and in a proper place (directly on the
real vehicle for which the maintenance work is carried out).
In this paper the authors present a prototype augmented reality system supporting
servicing and maintenance tasks of various vehicles. Augmented reality mode which is
used to visualize the data and knowledge connected with a particular vehicle can be
more intuitive than traditional ways of delivering knowledge and information using the
printed instructions or instructions on a flat computer screen [5].
2. Augmented reality-based systems for machinery maintenance
One of the first AR system supporting maintenance of devices is presented in [2].
KARMA, despite its simplicity, resulting from the application of visualization based on
a simplified edge models (Fig. 1), well supported a user in his servicing and maintenance
tasks.
1
Marcin Januszka, PhD Eng, Assistant Professor, Institute of Fundamentals of Machinery Design, Silesian
University of Technology
2
Wawrzyniec Panfil, PhD Eng, Assistant Professor, Institute of Fundamentals of Machinery Design, Silesian
University of Technology
37
Fig. 1. KARMA – one of the earliest AR system [2]
Another example of the use of augmented reality in helping to service activities is
ARVIKA [3]. The authors developed a system of several scenarios of application, and
one of them is „Troubleshooting and service on production machines” intended to
support personnel and users of machines in their start-up, operation, maintenance and
repair. Such a support may take place directly (on the machine) or by interaction with a
service center. Examples of applications of the ARVIKA system are aiding a design
process of factory installations, maintenance and repairs of the NATO helicopters or
servicing and repairs of sliding roofs of BMW cars.
System STARMATE [9] is provided to assist the user during the assembly and
disassembly of complex technical facilities and their maintenance and service, as well as
a training tool for service staff. This system allows employees to perform tasks that
require high qualifications (often exceeding their qualifications). Usually their
qualifications raise thanks to use of the system. The system works according to the
“learn by doing” idea. The authors [9] present examples of applications in disassembly
and repair of jet engines, where the user is aided by the AR system indicating following
steps of the task, disassembled parts of the engine, necessary tools, as well as technical
documentation of the engine.
Another, more recent, examples of the AR systems are VTT PALMOS [7] and
context-based system [10]. VTT PALMOS (Plant Model Services for Mobile Process
Maintenance Engineer) system was developed for modern maintenance works carried
out in industrial plants by staff of these plants or external service providers. The system
allows to solve such problems: How quickly locate the plant equipment requiring
maintenance? How to get a concise and clear information on plant equipment and its
state? How to connect and use the knowledge derived from multiple sources of
information without having to use a wide variety of computer software? The second
system [10] is a system based on knowledge. In this system content and placement of
additional information presented the user are dependent on the currently running
tasks/activities. Context in this system is understood as a state of an object, where the
object can be the device, place of a service, a person performing it, etc. The system is
considered as a context-based, if it can gather relevant information, draw conclusions
and adapt to a current situation. The advantage of the system is that it can dynamically
adapt to changing situations while performing maintenance and service tasks.
In the literature, especially in the online sources, you can find many other interesting
examples of use of augmented reality in assisting maintenance and service activities,
e.g. with use of mobile devices [11], ARMAR system applicable in operational use and
repair of military objects [12], in automotive [13, 14, 15].
38
3. Requirements and assumptions for vehicle service supporting system
It is assumed that the AR prototype system supporting maintenance of vehicles
should allow correctly and quickly perform the process of service (e.g. the repair or
replacement of worn out parts). It should be possible for standard and emergency (when
it is not possible for prior training) vehicle servicing situations. A person without a
proper knowledge (prior training), i.e. home user, should be carried out step by step by
the system providing instructions concerning the way and procedure of the service.
Instructions visualized in AR mode may contain:
 visual simulation of the activity (e.g. a way of some part disassembly),
 information about the kinds of tools used during the execution of the necessary
actions (e.g. a type and size of a key needed for a disassembly), including
visualization and simulation of usage during the execution of the task,
 audio records of how to implement a task (with the possibility of enriching
videos).
For a system to be used by any user (not only professional, but also home user), it is
required that the hardware and software components are not expensive and publicly
available. It is assumed therefore to develop a system that can be run on many devices,
including mobile devices, i.e. a tablet or smartphone.
4. The prototype vehicle service supporting system
In this study was developed a prototype system based on augmented reality
techniques, allowing for supporting the various activities (operational and servicing).
Implementation has been made for two types of vehicles: a mobile robot and a car.
Instructions supporting people engaged in service have been limited to a few selected
procedures. i.e. procedure for DC motor replacement of a mobile robot (service activity),
cruise control start-up procedure or headlights operation procedure (operational activity).
The user of the system is supported by appropriate instructions displayed using
elements presented in the form of text, interactive three-dimensional models, movies.
Their visualization is carried out at the site of the real vehicle or it’s component for
which the service is performed. “Step by step” real-time instructions should enable the
efficient conduction of operations also by people without prior training. This way of
instructions presenting is more intuitive than the presentation of the instructions in the
traditional form (as printed instructions).
4.1. Hardware components
In the case of systems based on augmented reality technology a proper selection of
optimal hardware solutions is essential, depending on the purpose of the system. Display
devices are one of the most important elements of AR-based systems. In an AR-based
system some additional computer-generated elements are added to a real environment
that is observed by eyes. There are many ways to present computer-generated virtual
content to the user of an AR-based system. By using advanced displays [1], i.e. Head
Mounted Displays (HMDs), Hand Held Displays (HHDs), Binocular Omni-Orientation
Monitors (BOOMs), and even the standard monitors or projectors, we can present what
we normally do not see. In recent years, more and more knowledge presentation using
AR techniques could be done through mobile display devices, i.e. displays in tablets or
smartphones. The use of mobile devices is both the most effective and cheap for home
users. Therefore, the system presented in this paper uses a PC-based tablet both as a
39
display device and at the same time as a computing unit (computer). However, the
system can be run on any device with Windows, Android, iOS operating systems.
Another very important element of the developed AR system is suitable tracking
system that allows properly integrate virtual objects with images of real world. As in the
case of display devices, there are many types of tracking systems (see [1]). The simplest
and one of the cheapest solution is a system based on image analysis and processing. The
most important component of this system is advanced software. However, for proper
operation of the system also some widely available components (especially cameras), are
required. In the developed AR system, a HD camera (installed on the tablet) captures
images of the real environment (real world), and sends these recorded images in real
time to a user’s computer. The images are analyzed and searched on the characteristic
elements (graphical markers) and then tracked. Next, for such images virtual objects are
added.
4.2. Software components
The presented system consists of several software components (e.g. a CAD system,
a PDM system, an UML editor, a knowledge base, a tracking system and a visualization
system) that are integrated with a single simple application for management (Fig. 2). The
system is available in a desktop version (for personal desktop computers) and in a
mobile version (for tablets or smartphones). The mobile version of the entire system is
stored in the form of a compiled executable application or as server-based application to
download and install on the user’s device.
The most important component of the AR system is a software that allows to
accurately track the real objects and present virtual objects on the real image. Virtual
objects are carriers for conveying knowledge and information. In the developed system,
software that uses image analysis has been applied for tracking. There are two
techniques used for a tracking: a tracking based on 2D images from a camera and a 3D
object tracking with the use of the SLAM (simultaneous localization and mapping)
method. Both methods are supported by gyroscopes in order to increase the quality of
tracking. Software used by the authors, that uses and implements these methods, is called
Metaio SDK (software library) and Metaio Creator (creator for AR scenes).
Fig. 2. Components of the system
40
Metaio software is also used both during the development and the presentation of an
AR scene with the knowledge useful during maintenance operations. The elaboration of
the AR scene involves determining an appropriate orientation, a position (relative to the
real objects, that are tracked), a size, and other parameters of the virtual objects. The last
stage - the application of knowledge - is to present knowledge to the people involved in
maintenance tasks. The most important knowledge and information necessary for the
operation of the system should be obtained from a design process of a vehicle. During
this design process some virtual 3D CAD models are developed. In the presented
system, due to the lack of such models, to reconstruct them reverse engineering methods
have been applied. In the process of reconstruction and 3D modelling a system
CATIA V5 was used. The 3D models were then stored in Product Data Management
(PDM) system. A repository that contains virtual models of the tools used during the
operation and animated models of typical maintenance operations (models with move of
a hand, operations of unscrew the left / right, clamping, gripping, etc.) (Fig. 3) is also
stored in the PDM system. Due to requirements of the AR scene all models are
converted to the format supported by the used software (mainly the FBX format).
Fig. 3. Examples of models of tools and maintenance tasks from repository
Except that the user of AR system can see virtual 3D and interactive models
superimposed on the real world, he/she can also see knowledge connected with
maintenance activities in form of diagrams. Activity diagrams are created in accordance
with the notation of the Unified Modeling Language (UML). This knowledge is stored in
a knowledge base in two different forms: graphical (as .png graphical file format), and
textual (as an XML file). For the purposes of the research carried out by the authors only
a graphical form (with the use of UML diagrams shown in AR mode) was applied. The
XML file can be used in future knowledge processing systems or knowledge based
systems (expert systems). Presentation of knowledge with diagrams and 3D models may
be supplemented with additional sources of knowledge, i.e. audio descriptions, photos,
videos that could be also stored in the PDM system.
4.3. Functionality of the system
If someone needs to perform some maintenance activities, he/she should launch the
main application for the AR system management (for PC) or standalone application (for
mobile devices). When the application is running, a searching for an appropriate
instruction of a maintenance task is available.
If the user finds a right instruction and points a main unit (tablet, smarthone, PC
camera) at a vehicle (often at a fragment of the vehicle), for which instructions should be
displayed, then the images with an indicated key parts of a vehicle are automatically
41
displayed. The user can analyze (regarding to position and part name etc.) these parts.
Next, by pressing "Start" button it is possible to run the instruction that presents
maintenance activities. The user sees on the screen the visualization of a first operation
for a procedure (e.g. unscrewing the fixing screws). At the same time on the screen could
be presented (Fig. 4):
 UML diagram with an indicated current operation from a whole procedure
(UML activity block with the name of the operation),
 models (that are usually animated) of parts (components) related to the
operation,
 tools that are necessary to carry out the current operation,
 animation of activities performed by the user (also with the use of 3D models).
Additionally, the user can listen to the audio description of the operation or refer to
additional sources of knowledge, i.e. movies, pictures, drawings, if they have previously
been prepared for the operation. After the initial operation, the user can - by selecting a
button from the UML diagram - go to the next operation and so on until the end of the
procedure.
Fig. 4. Example of elements presented to the user on a screen
The system presented in this paper allows the user to develop both the basic and
advanced skills in maintenance of vehicles with a maximum reduction of the risk of
damages (caused by a lack of knowledge). The system allows the user to get acquainted
with a complex construction of a vehicle and with main components that are parts of the
vehicle (Fig. 1). However, the most important function of the AR system is presentation
of maintenance instructions regarding to a vehicle, including:
 operating instructions for operators,
 maintenance instructions for technicians (repairs, part replacements).
The functionality of the AR system allows a presentation of the visible interior of
the vehicle. Moreover, it is possible to visualize the position of individual elements
(Fig. 5), including those which are normally not visible from outside (e.g. are hidden
under the covers) or those for which an access is difficult. Such information is a hint for
an inexperienced user, what is the exact location of a part (useful e.g. in the case of need
of part replacement or its repair). Likewise, various types of data and information
(e.g. parameters, marks etc.) connected with selected components may be also displayed.
42
For example, information regarding to a part mark allows proper selection of
a replacement part during exchanging procedure.
Fig. 5. Presentation of elements’ positions and descriptions
The most important feature of the system is connected with a possibility of
maintenance tasks supporting (Fig. 6). Generally, the user of the AR system is supported
by displaying the appropriate instructions, using interactive 3D models and diagrams
superimposed on a physical object. “Step by step" instructions with visualization of realtime tasks allow to efficiently carry out complex maintenance operations, including
those with no prior training. This way of instructing the user should be more intuitive
than the presentation of the instruction in the traditional form (as printed instructions).
Instructions visualized in AR mode may contain knowledge for which different virtual
objects could be used as carriers (see section 3). Knowledge and data for the system are
derived from the design stage in which many virtual 3D CAD models are elaborated.
Fig. 6. Examples of visualisations of maintenance tasks
5. Evaluation of the AR system
The aim of the validation studies in relation to the developed system was to evaluate
the operation and usefulness of the system in the context of its use to facilitate
maintenance activities for vehicles with the use of AR techniques.
To evaluate the system quantitative (time of maintenance tasks) and qualitative
measures (satisfaction of the user) were selected. Experimental studies have been carried
43
out in order to evaluate the functionality and usefulness of the developed system using
AR technology. Experiment consisted of four maintenance tasks (for two types of
vehicles, i.e. a mobile robot and a car), including: engine replacement for a mobile robot
(task connected with robot service), a light bulb replacement in a car (task connected
with car service), launching cruise control in a car (task connected with car operation),
running lights in a car, in the correct order, i.e. traffic lights, high beam, position lights,
front fog lights, rear fog lights (task connected with car operation). 10 random people
were invited to take part in the experimental study. Before the experiment they have
never been dealing with vehicles involved in the experiment. Each participant carried out
two maintenance tasks using the AR system and another two tasks using traditional
printed user's manual supplied with the vehicle. User randomly sketched which of two
tasks were carried out by them with the use of the AR system, and which another two
without the AR system.
The validation of the system was assessed with the use of induction method,
according to the following plan:
1. A short training for participants, explaining the purpose of the experiment;
2. Carrying out the maintenance tasks for two types of vehicles;
3. An evaluation of the system in terms of functionality and usefulness of the AR
system.
The evaluation of the system has both quantitative and qualitative form.
Quantitative assessment was made as a result of direct observation of time of the task.
A test solely included the duration time of operator actions, assuming that the other
times (e.g. time of preparation and time to search for information, etc.) are at a very
similar level, both in the case of the AR system and the printed manual. To assess the
quality of the AR system a personal interview method was used. The participants
evaluated the usefulness and functionality of the AR system and traditional manuals,
during the interview, in three categories: an efficiency, a quality of visualization, an
overall evaluation of the usefulness. The participants responded to statements using a
three-point Likert scale. The scale rating was determined by evaluation from
-1 – negative rating, 0 – I do not have an opinion, up to 1 – positive rating.
The average saving times thanks to the using of the AR system for computer-aided
maintenance are presented in Fig. 7. Analyzing the duration times of maintenance
operations realized by individual users it can be seen that by using the AR system it is
possible to achieve a significant shortening of the tasks execution. The mean shortening
of the execution time of the tasks achieved the level of 23.7%. This result is largely
consistent with the results for this type of maintenance tasks presented in [4].
In the case of qualitative assessment the AR system received higher average scores
than traditional printed user manuals in all three categories. The results of the qualitative
assessment are presented in Fig. 8. It can be seen that only in the category of ‘quality of
visualization‘ results were most similar. This situation was probably due to some
disadvantages of AR technology in the field of visualization, i.e. limitations related to
image resolution, and tracking stability during the superimposing the virtual objects on
the real-world images.
44
Fig. 7. The average duration of the maintenance tasks and time savings
Fig. 8. Comparison of quality assessment of the AR system and the traditional user
manual.
Summarizing, the participants of the experiment positively evaluated the AR-based
system aiding maintenance of vehicles. This group clearly indicated the usefulness of
this system in the maintenance tasks.
6. Summary
The need for the presentation in a comprehensive manner of knowledge related to
the handling of complex technical means, which are vehicles, requires advanced
visualization methods. A proper visualization allows more flexibility and freedom in
decision making, speeds up the maintenance and helps to avoid errors. In this respect
great potential involves the application of the augmented reality techniques, which
attempt is presented in this article. As demonstrated by studies concerning the use of AR
system supporting a user in performing servicing and operational activities on different
groups of vehicles, it should reduce the time of these activities (in relation to the time
they will be fulfilled with supporting by traditional printed user manuals). A significant
benefit of the use of the augmented reality is also that the operating instructions
developed in this way are much more readable for people using them. The knowledge
represented in the form of 3D models can be accurately presented at the time of a
demand for it and in such a way that the user is not disturbed during his duties. These
45
advantages have a positive effect on the shortening of executed operations, and thus
allow for some costs savings.
As part of a future work is expected to conduct verification tests on a larger group
of people, as well as among professional technicians in garages. Of particular interest
may be the results of a group of professional mechanicians who have a lot more
knowledge and experience than the standard user of the vehicles. The goal is to
demonstrate the usefulness of the system also for such users. Benefits
of
using
augmented reality techniques allow us to hope that one day such systems become widely
used and will contribute to the further rationalization of maintenance and servicing of
vehicles, despite still present some technological (quality of visualization and tracking
accuracy could be improved) and economic (currently high cost of equipment) problems.
References:
[1]
Azuma R.T.: A Survey of Augmented Reality. Teleoperators and Virtual
Environments 6 (4), 1997, s. 355-385
[2]
Feiner S., Macintyre B., Seligmann D.: Knowledge-based Augmented Reality.
Communications of the ACM, 36(7), s. 53-62, 1993
[3]
Friedrich W.: ARVIKA-augmented reality for development. Production and
Service, International Symposium onMixed and Augmented Reality, s. 3–4, 2002
[4]
Januszka M.: Metoda wspomagania procesu projektowania i konstruowania z
zastosowaniem "poszerzonej rzeczywistości", Zeszyty, Nr 147, Politechnika
Śląska, Instytut Podstaw Konstrukcji Maszyn, Gliwice, 2012
[5]
Moczulski W., Panfil W., Januszka M., Mikulski G.: Applications of augmented
reality in machinery design, maintenance and diagnostics. In: R. Jablonski, M.
Turkowski, R. Szewczyk, eds., Recent Advantages in Mechatronics, SpringerVerlag, Berlin Heidelberg, s. 52-56, 2007
[6]
Reiners D., Stricker D., Klinker G., Muller S.: Augmented reality for construction
tasks: Doorlock assembly. 1st IWAR98, pp.31–46, 1998
[7]
Savioja, P., Järvinen, P., Karhela, T., Siltanen, P., Woodward, C.: Developing a
mobile, service-based augmented reality tool for modern maintenance work. In:
Proc. of the 2nd Int. Conference on Virtual Reality, Beijing, s. 554–563, 2007
[8]
Skarka W., Moczulski W., Januszka M.: Interaktywne technologie w procesie
kształcenia. Szybkobieżne Pojazdy Gąsienicowe (29) nr 1, 2012
[9]
Schwald, B., et al.: STARMATE: using augmented reality technology for
computer guided maintenance of complex mechanical elements. In: Proceedings
of eBusiness and eWork Conference (e2001), Venice, s. 17–19, 2001
[10] Zhu J., Ong S. K., Nee A. Y. C.: A context-aware augmented reality system to
assist the maintenance operators, International Journal Interact Des
Manufacturing, vol. 8, s.293–304, 2014
[11] Augmented reality in action - maintenance and repair {Available - 28.02.2015:
http://www.pocket-lint.com/news/108887-augmented-reality-maintenance-andrepair}
[12] Augmented Reality To Help Military Mechanics Fix Vehicles {Available 28.02.2015:
http://singularityhub.com/2010/01/11/augmented-reality-to-helpmilitary-mechanics-fix-vehicles-video/}
[13] End of the mechanic? BMW smart glasses make it possible for ANYONE to spot
and fix a car engine fault just by looking at it {Available - 28.02.2015:
46
[14]
[15]
http://www.dailymail.co.uk/sciencetech/article-2543395/The-end-mechanicSmart-glasses-make-possible-fix-car-engine-just-looking-it.html}
Metaio and Volkswagen announce MARTA augmented reality service support
for
new
XL1
concept
car
{Available
28.02.2015:
http://www.metaio.com/press/press-release/2013/volkswagen-marta-augmentedreality-service-support/}
I-Mechanic, the AR App that turns yourself into a Mechanic {Available 28.02.2015: http://www.armedia.it/i-mechanic}
Abstract
Knowledge of modern technical means, connected for example with their using or
servicing becomes increasingly complex and difficult to interpret. Systems supporting
servicing of such technical means are expected to fulfill still increasing requirements,
which in consequence should result in time and money savings. The paper presents an
augmented reality (AR) system based on a mobile device such as tablet. The main task
of the system is to aid a normal use and maintenance tasks of the technical means by
enhancing the image captured and displayed on the tablet with virtual 3D objects. These
objects are properly located with respect to the real objects. The aiding consists in
displaying invisible parts, activity diagrams of the tasks, and tools necessary during the
tasks. Verification of the system have been carried-out on two objects, i.e. a mobile robot
and a car. In the first case the task was to replace a drive of the mobile robot. In the
second case the task was to replace a bulb or to launch a cruise control system or switch
the lights. There was done a quantitative assessment of the AR system based on
comparison of the time of these operations in carried-out in a conventional manner (with
use of printed manuals) and using the developed AR system. There was also made a
qualitative assessment of the AR system based on subjective evaluation of system’s
users. Both assessments (quantitative, qualitative) proved beneficial effect of using the
AR system to perform maintenance and simple car driver tasks.
Keywords: augmented reality, vehicle maintenance, computer-aided
maintenance, interactive systems, knowledge-based systems, visualization
vehicle
TECHNIKI POSZERZONEJ RZECZYWISTOŚCI DO
WSPOMAGANIA OBSŁUGI POJAZDÓW
Streszczenie
Wiedza dotycząca współczesnych środków technicznych, związana np. z ich
obsługą serwisową/techniczną lub operatorską staje się coraz bardziej złożona i trudna w
interpretacji. Przed systemami wspomagania obsługi takich środków technicznych
stawiane są coraz wyższe wymagania, które w efekcie mają na celu uzyskanie
oszczędności czasu, a tym samym kosztów. W artykule zaprezentowano mobilny system
poszerzonej rzeczywistości (ang. Augmented Reality, AR) wykorzystujący tablet,
którego głównym zadaniem jest wspomaganie obsługi serwisowej i operatorskiej
poprzez wzbogacanie obrazu rejestrowanego i wyświetlanego przez ten tablet obiektami
wirtualnymi. Obiekty te są odpowiednio zorientowane względem obiektów
rzeczywistych. Wspomaganie polega m.in. na wyświetlaniu niewidocznych obiektów,
diagramów czynności dla wykonywanych zadań, a także narzędzi potrzebnych w trakcie
tych czynności, Weryfikację działania systemu przeprowadzono dla dwóch obiektów,
47
tj. robota mobilnego oraz samochodu osobowego. W przypadku robota mobilnego
dokonywano czynności serwisowej polegającej na wymianie jego silnika napędowego,
natomiast w przypadku samochodu dokonywano czynności serwisowej (wymiana
żarówki), a także czynności operatorskich (uruchomienie tempomatu oraz włączenie
świateł). Dokonano oceny ilościowej systemu AR na podstawie porównania czasów
wykonywania wspomnianych czynności w sposób tradycyjny (z zastosowaniem
instrukcji drukowanych) oraz z wykorzystaniem opracowanego systemu. Dokonano
również oceny jakościowej systemu na podstawie subiektywnej oceny jego działania
poprzez jego użytkowników. Obie oceny wykazały korzystny wpływ zastosowania
systemu AR na wykonywanie czynności obsługowych.
Słowa kluczowe: poszerzona rzeczywistość, komputerowe wspomaganie eksploatacji
pojazdów, systemy interaktywne, systemy bazujące na wiedzy, wizualizacja
48