How to extract/convert data from blender cache files .bphys into a human readable form via python

Question

I am trying to extract position, velocity and rotation data from smoke simulation cache files .bphys.

i have been attempting to convert the .bphys file into a more human readable form via python. With the eventual goal of using the location velocity and rotation data to create an .fga vector field that could be imported into unreal engine 4. If someone could tell me how to convert the binary data into usable (human readable) float values using python that would be amazing. I am having problems separating the data properly into floats.

(here is an overview of the contents of the files. https://developer.blender.org/rBAC17a9b3e44ca0e48ef0244df19f2f252198a56919 )

(additional info) struct.unpack is currenly what i have been trying to use. however there seems to be some non float data at the start of the file. i have not been able to work out exactly where it ends, and i think it might be throwing off the unpack. also unpacking with "f"

struct.unpack("f",file.read(4)

is resulting in strings of numbers with what i assume to be scientific notation at the end (**10) e.g 4.203895289274451e-45, which do not look like the sort of values i was expecting to find.

Thanks

Answer 1

the first 9 bytes of the bphys file is the Header of the file. the first 8 are 'BPHYSICS' after which there is a unsigned int describing the type of data:

0 softbody
1 particles
2 cloth
3 smoke_domain
4 smoke_highres
5 dynamicpaint
6 rigidbody 

the link you posted is in relation to Particles, so i will assume you are only looking at accessing the particle data.

after the header, each particle has the following data:

index , 4
location , 12 
velocity , 12 
rotation , 16 
avelocity , 12
size , 4
times , 12
boids , 20

however, the pointcache reader will only write/read boid data if the particle system has boids enabled. so if you're not using boids, you just dont read that data.

assuming there is no boid data you will want to read the following data:

unsigned int, // BPHYS_DATA_INDEX
3 * float, // BPHYS_DATA_LOCATION
3 * float, // BPHYS_DATA_VELOCITY
4 * float, // BPHYS_DATA_ROTATION
3 * float, // BPHYS_DATA_AVELOCITY / BPHYS_DATA_XCONST
float, // BPHYS_DATA_SIZE
3 * float, // BPHYS_DATA_TIMES  

all this information can be found in pointcache.c file in blender's code.

Answer 2

I have the opinion that the format for the external physic-storage is no format that is fixed for every version of blender. I have looked at a file and found that the location-vectors, that should use 12 bytes, use 16 bytes. I believe that this is determined by the compiler and may differ for 32bit and 64-bit-versions. The best way for getting coordinate and rotation-info for particles is in my opinion to edit the source of blender (pointcache.c) and add a saving of files with textual info, such as fprintf(file,"XYZ %lf,%lf,%lf\n",x,y,z) with names like the cache files. I will try this out, ihave same ideas about getting particle-infos as animation-pathsinto blender for creating animations.

Hexview of file http://www.mingsing.de/AkAZimbo/Hex.jpg

My first version of a gambas-program (basic on linux) works an can read the coordinates. The long-value on position 12 tells the count of particles inside the framefile, position 20 is the number of the following particle, 24 is x as single, 28 y as single and 32 z as single-float, little endian. The next data is following with an add of 76 bytes for each next particle until all particles in ths frame are done. Multiple particles in a scene are stored in multiple files. The name bervore the frame-numbering is different for each particle-system that has a different object. When you use more than one particle-system for one object by adding more slots, the files have a different naming just at the end, the first system has the 00, the secondt the 01 and so on.

Answer 3

It is a little more complicated than as explained by Wardrums. As of Blender 2.77 my experiments yielded the following results:

b'BPHYSICS'
flavor : i32 #=1 for point cache
count : i32
data_type_flags : i32 #I think this relates to BPHYS_DATA_* in DNA_object_force_h
for i in range(count):
     index : i32
     location : i32[3]
     velocity : i32[3]

For my particular sample file the data_type_flags==7 and the table only contained the index, location, and velocity. This corresponds to (1<<BPHYS_DATA_INDEX)|(1<<BPHYS_DATA_LOCATION)|(1<<BPHYS_DATA_VELOCITY) . I suspect that particle systems that use an object group for rendering instead of a halo will have PHYS_DATA_ROTATION, and other particle system features will bring in other flags requiring more of the columns that Wardrums documented. Likewise other cache formats (cloth, smoke, rigid body) will make use of columns not used by particles.

One other thing is that the count and data_type_flags are read by pid->read_header which as far as I can tell is only ptcache_basic_header_read but there might be a mode where it is a different function, and things might change in the future.

I have also noticed that the very first file (_000000_00.bphys) has only time information (data_type_flags=0x40) and in my file the rows looked like

{'times': (1.0, 51.0, 50.0)}
{'times': (1.1990000009536743, 51.19900131225586, 50.0)}
{'times': (1.3980000019073486, 51.39799880981445, 50.0)}

which I assume to be (start,end,lifetime) numbers as frames. This file may or may not be important if you're attempting to synthesize your own point cache files from scratch.

Just for completeness, here is an early draft of http://web.purplefrog.com/~thoth/blender/python-cookbook/dump-point-cache.html I used to dump my particle cache:

import struct

def dump_one_file(fname):
    f = open(fname, "rb")

    magic = f.read(8)
    if magic != b'BPHYSICS':
        raise Exception("not a blender physics cache")

    flavor = f.read(12)
    (flavor,count,something) = struct.unpack("iii", flavor)

    print( "%d\t%d\t%d"%(flavor,count,something))

    if flavor==1: # point cache

        rec_len = 28
        while True:
            chunk = f.read(rec_len)

            if chunk is None or len(chunk)==0:
                break
            if len(chunk) != rec_len:
                raise Exception("short read (%d<%d)"%(len(chunk), rec_len))

            all = struct.unpack("i fff fff ", chunk)
            print( "%d\t<%f,%f,%f>\t<%f,%f,%f>"%all)

dump_one_file("/var/tmp/blendcache_particle/pants_000238_00.bphys")

Answer 4

You might be able to build on the techniques from Reading 32 bit signed ieee 754 floating points from a binary file with python?. I don't have any physics caches lying around to experiment upon, but hopefully this will get you started.