A mixed reality application for the Microsoft Hololens was developed to display real-time data from the sensors to engineers.
As part of my Year In Industry, I led training to engineers to use the Microsoft HoloLens so that virtual inspections could be conducted without traveling- especially crucial during the Covid-19 pandemic. This barely grafts the capabilities of said device, and there is untapped potential in how we could use the HoloLens; hence, I want to develop an application that will help the engineers.
One way to help engineers is to provide tools to help them diagnose faults and see the current status, which could be achieved by displaying live data from the sensors on the HoloLens. This will provide engineers with relevant information in their field of view, rather than holding a separate device, increasing productivity.
As this is an IoT application, the data flow is essential to the solution. Based on my literature review, I have designed an architecture that is scalable and can be implemented within the budget constraints:
The Architecture of the data flow can be broken into three components: • The Raspberry PI –uses sensors such as a thermometer to measure the temperature • The Azure Cloud –is responsible for receiving, process, storing and broadcasting the data • The HoloLens app- receives the data and uses it to display the data
This device will retrieve the data from the sensors and send it to the cloud. A data simulator with a Python script that sends data to the IoT cloud will be used for development. It can also be replaced by any other service that can send data to the cloud.
I have used as many serverless services as possible as it allows me to scale when multiple sensors are introduced, as it is not tied to the specification of a machine. This will mean that the architecture can achieve the requirement to scale to thousands of devices.
This is where the data collected from the Raspberry PI will be sent to. It is a message broker that is suited to IoT messages and devices. As it is a message broker, I can redirect the messages to the relevant services.
My research shows that Azure Digital twins allow us to represent objects digitally. Although it would be simpler to bypass this and store the values in a database and retrieve the latest readings, modeling the power station opens up more options for future versions. We could run machine learning to predict faults before it happen and infer where the fault could lie, i.e., related joined components.
One of the requirements is to be able to scale. Azure Event grid would allow us to fire off an action every time there is new data. Thus it makes it scalable as it is not dependent on the server specifications to run and can increase automatically when the load increases. This makes an event-driven architecture.
SignalR would allow broadcasting the data to all the clients connected. This is a better method, as the device does not have to poll for new data, so there will be a lower delay in retrieving the live data and lower service usage. This is because the Signal IR broadcasts it continuously rather than a device making continuous API calls to refresh the data. An analogy to this method is that satellite TV is distributed, the satellite broadcasts it, and the satellite box decodes, compared to network streaming, where the device asks the server to send the data.
For the application that will be developed, I decided to go for a Unity 3D application. This is mainly due to the documentation that Microsoft provides being catered for it. The Mixed Reality Toolkit that provides UI components for the HoloLens is also developed for Unity. This application will subscribe to the Azure SignalR broadcast, allowing it to display live data to the user in dashboards. These dashboards will be fixed onto a location using Azure Spatial Anchors.
Please see Dissertation or Presentation for more information.
1.Anchor.to.Walls.1.mp4
2.Alert.and.Arrow.1.mp4
3.Close.Dashboard.mp4
As this was an application developed under Uniper sponsorship, the source code is not available publically.