Our visualization pipeline:

At first we were uncertain about what system we wanted to use to visualize Cabot. At first we tested simple visualization and a GUI in CodeSys, but as CodeSys is a slow program we wanted to use third-party software. We knew that we had two main options; Ignition and CDP-Studio. Both of which are new software packages to all of the group members and it would require a communication protocol. As the project went on we deprioritized the visualization and GUI, until we heard from another group that they were using Blender. Blender is a 3D graphics program with no built in support for communication with other software, but has great modelling tools and rendering capabilities. Since blender has a Python API, and we now had MQTT setup, we decided to give it a go, even though it is not software made for this, its possible to adjust the program to our needs. While the 24_7 CDP-Studio Dashboard was being developed, the blender script and visualization was developed in parallel. Lets go into the details of the Blender pipeline.

Responsible: Magnus

This page is best viewed in full screen landscape

Digital-Twin in Blender:

First of all we created the digital double as seen in the image on the right. As most parts of our digital double does not need to be dimensionally accurate, such as the motor size and the end-effector, it was a simple modelling job. The dimensions of the aluminum profiles has to be accurate though, so they were created accordingly. Motor and driver 3D-models were taken from the official Omron website, where they offer CAD files for free. Now for the important part, the communication and methods we implemented to make the visualization behave as a digital twin and UI for our system.

We wanted to make the position of the end-effector of our digital representation match the actual end-effector position on Cabot. A communication fieldbus was needed, and we chose MQTT because of its python libraries we could use in Blender. Creating the script requires some knowledge about the BPY (Blender python api). As we had experience with this from earlier projects, we made a subscriber script first, which listens to updates from the PLC publisher. The messages are formatted in the most logical way: "x,y,z". Some adjustments were also made to make the coordinate system from the system work in the visualization.

The python libraries used for the MQTT software was paho-mqtt. This library helps make subscriber and publisher clients, which is exactly what we needed. After testing of the subscriber went well, we figured why not make blender our main UI system as well. This would make the interaction with the system more natural, because of its complex dynamics could be visualized at the same time as using it as an input. The first idea was to simply publish the position of the blue cube to the PLC, and it should then dynamically move to the target position. The implementation of this went well, and we decided to add more features. A feature list of what can be done from blender now is:

  1. Send "real-time" update of the blue cube's position to update the target position on the system.
  2. Subscribe to estimated position from forward kinematic and link that position to the red cube.
  3. Add step to path.
  4. Remove step to path.
  5. Show full path.
  6. Send full path.


How the visualization looks to the end-user:

To make it easier to visually understand what is happening in the system at all times, the viewport is split into 4 unique views. In this setup the top left view is from the front, to the right is from the side and the last two are from the top view and one is free to move to interact with the camera at will. This is the typical setup when running the program, though all of this is completely adoptable.

Basic script explanation:

First the script is initialized ran in blender, when it starts up it creates the UI buttons on the side of the toolbar in the 3D-viewport. As can be seen in the code the UI elements/buttons have functions that run when they are pressed. The MQTT part of the code mostly consists starting the subscriber, stopping the subscriber, and publishing when the "send motion" button is pressed. 

When running the script multithreading was used as Blender needs this to update positions in realtime. The code from blender is found here, in the codeblock below. 










Python Code in Blender 
import bpy
import threading
import paho.mqtt.client as mqtt
import time

# Global MQTT
mqtt_client = None
mqtt_running = False
mqtt_topic = "cabot"
broker_address = "localhost"
broker_port = 1883

# path globals
path = []  # path steps

# MQTT helpers, runs via the the mqtt class
def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected with result code "+str(rc))
        client.subscribe(mqtt_topic)
    else:
        print("Failed to connect with result code "+str(rc))

def on_message(client, userdata, msg):
    decoded_message = msg.payload.decode()
    print(f"Message mottatt: {decoded_message}")
    coordinates = decoded_message.split(',')
    if len(coordinates) == 3: # example of message -m "x,y,z"
        try:
            position = tuple(float(coord) for coord in coordinates)
            bpy.app.timers.register(lambda: set_object_position("End-Effector", position), first_interval=0.1)
        except ValueError:
            print("Error: ikke tall LOL")

def set_object_position(obj_name, position):
    obj = bpy.data.objects.get(obj_name)
    if obj:
        obj.location = position
    return None

# MQTT Client Thread init
def run_mqtt_client():
    global mqtt_running, mqtt_client
    mqtt_client.loop_start()
    while mqtt_running:
        continue
    mqtt_client.loop_stop()
    mqtt_client.disconnect()

# Operator to start MQTT subscription
class StartMQTTSubOperator(bpy.types.Operator):
    bl_idname = "mqtt.start_sub"
    bl_label = "Start MQTT Subscriber"

    def execute(self, context):
        global mqtt_client, mqtt_running
        if not mqtt_client:
            mqtt_client = mqtt.Client()
            mqtt_client.on_connect = on_connect
            mqtt_client.on_message = on_message
            mqtt_client.connect(broker_address, broker_port, 60)
        if not mqtt_running:
            mqtt_running = True
            threading.Thread(target=run_mqtt_client).start()
        return {'FINISHED!'}


class StopMQTTSubOperator(bpy.types.Operator):
    bl_idname = "mqtt.stop_sub"
    bl_label = "Stop MQTT Subscriber"

    def execute(self, context):
        global mqtt_running
        mqtt_running = False
        return {'FINISHED'}

class AddStepOperator(bpy.types.Operator): # for path, må fullføres for å animere path i blender under show motion
    bl_idname = "mqtt.add_step"
    bl_label = "Add Step"

    def execute(self, context):
        target = bpy.data.objects.get("Target")
        if target:
            path.append(target.location.copy())
            print("Step added: ", target.location)
        return {'FINISHED'}

class RemoveStepOperator(bpy.types.Operator):
    bl_idname = "mqtt.remove_step"
    bl_label = "Remove Step"

    def execute(self, context):
        if path:
            removed = path.pop()
            print("Step removed: ", removed)
        return {'FINISHED'}

# denne må oppdateres. nå fryser blender viewporten når jeg "animerer" posisjonen for å simulere. 
# trenger også å animere en path fra x1,y1,z1 til xN,yN,zN avhengi av hvor mange i punkter i path, HIGH PRIORITY
class ShowMotionOperator(bpy.types.Operator):
    bl_idname = "mqtt.show_motion"
    bl_label = "Show Motion"

    def execute(self, context):
        target = bpy.data.objects.get("Target")
        if target and path:
            for pos in path:
                target.location = pos
                time.sleep(0.5)  
        return {'FINISHED'}


class SendMotionOperator(bpy.types.Operator):
    bl_idname = "mqtt.send_motion"
    bl_label = "Send Motion"

    def execute(self, context):
        if path: # sends a path to the topic
            message = ",".join(["{:.2f},{:.2f},{:.2f}".format(*p) for p in path]) # ChatGPT food. husket ikke syntax for formatering
            mqtt_client.publish("cobot/input/targetposition", message)
            print("Published path: ", message)
        return {'FINISHED'}


class CobotSystemInterfacePanel(bpy.types.Panel):
    bl_label = "Cobot System Interface"
    bl_idname = "MQTT_PT_cobot_system_interface"
    bl_space_type = 'VIEW_3D'
    bl_region_type = 'UI'
    bl_category = 'Cobot System Interface'

    def draw(self, context):
        layout = self.layout

        # MQTT Section / Buttons
        layout.label(text="MQTT")
        layout.operator("mqtt.start_sub", text="Start MQTT Subscriber")
        layout.operator("mqtt.stop_sub", text="Stop MQTT Subscriber")

        # Operations Section / BUttons
        layout.label(text="Operations")
        layout.operator("mqtt.add_step", text="Add Step")
        layout.operator("mqtt.remove_step", text="Remove Step")
        layout.operator("mqtt.show_motion", text="Show Motion")
        layout.operator("mqtt.send_motion", text="Send Motion")

def register(): # Blender shiT
    bpy.utils.register_class(StartMQTTSubOperator)
    bpy.utils.register_class(StopMQTTSubOperator)
    bpy.utils.register_class(AddStepOperator)
    bpy.utils.register_class(RemoveStepOperator)
    bpy.utils.register_class(ShowMotionOperator)
    bpy.utils.register_class(SendMotionOperator)
    bpy.utils.register_class(CobotSystemInterfacePanel)

def unregister(): # MER Blender shiT
    bpy.utils.unregister_class(StartMQTTSubOperator)
    bpy.utils.unregister_class(StopMQTTSubOperator)
    bpy.utils.unregister_class(AddStepOperator)
    bpy.utils.unregister_class(RemoveStepOperator)
    bpy.utils.unregister_class(ShowMotionOperator)
    bpy.utils.unregister_class(SendMotionOperator)
    bpy.utils.unregister_class(CobotSystemInterfacePanel)

if __name__ == "__main__":
    register()




The UI interface:

The items mentioned in the list above are also made as UI-elements within the toolbar interface in the software. Since its the only way we found to press buttons to run simple commands from the interface. The UI-interface is implemented as illustrated in the image completely to the right in this section. 



  • No labels

Industrielle styresystemer - NTNU Ålesund - Automasjon