Hi I am wondering if anyone may be able to help

I am attempting to impliment a data stream from an old software called Faceshift Studio for mocap of a face using my Kinect with a depth sensor. Apple bought the software back in 2015 but I found an old version of the software which does work very well, and I do not have an Iphone to use in this modern day so thought it would be an option to try this I have found that faceshift has a streaming option which I can get the data from to pass to blendshapes on a model face. So far I can receive data from the software and blender can receive it and parse to the model, how ever the model behaves erratically and I think it is to do with the data conversion from the software.

Here is my code so far

I am an artist and not a programmer and know there must be someone out there who could assist

import bpy
import socket
import threading
import struct
# Define the UDP IP address and port
UDP_IP = ""  # Localhost (change to the IP address Faceshift Studio is sending to)
UDP_PORT = 5005       # The port Faceshift Studio is sending to
# Create a socket to listen for UDP packets
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((UDP_IP, UDP_PORT))
# List of blendshape names
blendshape_names = [
    "EyeBlink_L", "EyeBlink_R", "EyeSquint_L", "EyeSquint_R", "EyeDown_L", "EyeDown_R",
    "EyeIn_L", "EyeIn_R", "EyeOpen_L", "EyeOpen_R", "EyeOut_L", "EyeOut_R", "EyeUp_L",
    "EyeUp_R", "BrowsD_L", "BrowsD_R", "BrowsU_C", "BrowsU_L", "BrowsU_R", "JawOpen",
    "LipsTogether", "JawLeft", "JawRight", "JawFwd", "LipsUpperUp_L", "LipsUpperUp_R",
    "LipsLowerDown_L", "LipsLowerDown_R", "LipsUpperClose", "LipsLowerClose", "MouthSmile_L",
    "MouthSmile_R", "MouthDimple_L", "MouthDimple_R", "LipsStretch_L", "LipsStretch_R",
    "MouthFrown_L", "MouthFrown_R", "MouthPress_L", "MouthPress_R", "LipsPucker", "LipsFunnel",
    "MouthLeft", "MouthRight", "ChinLowerRaise", "ChinUpperRaise", "Sneer_L", "Sneer_R", "Puff",
    "CheekSquint_L", "CheekSquint_R"
def udp_listener():
    while True:
            # Receive data from Faceshift Studio
            data, addr = sock.recvfrom(1024)  # Buffer size is 1024 bytes
            print(f"Received message: {data} from {addr}")
            # Process the binary data (replace with actual parsing logic)
            parsed_data = parse_binary_data(data)
            # Update the blendshapes in Blender
            if parsed_data:
                bpy.app.timers.register(lambda: update_blendshapes(parsed_data))
        except Exception as e:
            print(f"Error in udp_listener: {e}")
def parse_binary_data(data):
    # Determine the number of blendshape weights expected
    num_blendshapes = len(blendshape_names)
    # Calculate the expected data size (num_blendshapes * 4 bytes per float)
    expected_data_size = num_blendshapes * 4
    # Check if the received data matches the expected size
    if len(data) >= expected_data_size:
        # Unpack the data into floats
        weights = struct.unpack('f' * num_blendshapes, data[:expected_data_size])
        return weights
        print(f"Received data size ({len(data)}) does not match expected size ({expected_data_size})")
        return None
def update_blendshapes(weights):
    # This function updates the blendshapes of a Blender object based on received weights
    obj = bpy.data.objects.get("Head_Mesh")  # Replace with your mesh object name
    if obj and obj.data.shape_keys:
        for i, weight in enumerate(weights):
            shape_key = obj.data.shape_keys.key_blocks.get(blendshape_names[i])
            if shape_key:
                shape_key.value = weight
# Start the UDP listener in a separate thread
thread = threading.Thread(target=udp_listener)
thread.daemon = True
print("Started UDP listener...")
# Keep the script running
def keep_running():
  • $\begingroup$ Hello, my best bet is that weight is not bound to 0...1 when that's the values blender shape keys expect in general. Try putting a print(weight) in the def update_blendshapes for loop $\endgroup$
    – Gorgious
    Commented Jun 26 at 8:34
  • $\begingroup$ I found a script for an old version of blender, but it uses a makehuman model which isnt what I have. I am wondering if this script would be able to be altered to work? As any online help I am getting is saying possibly the parsing of the data out from faceshift needs to correspond, but as there is no documentation any more nor source code, I am wondering how I can find the struct of the data? $\endgroup$
    – Ectomia
    Commented yesterday


You must log in to answer this question.

Browse other questions tagged .