2
$\begingroup$

I've a scene and an image matte built from an image with a green rectangle depicting a hole. I'd like to have the camera view resized to fit the hole.

The image with the hole may change, the hole may be sized and located differently (no input about that, only the green color of the pixels).

I have used a Transform node at the location indicated with ?. I have to adjust the offsets and the scale factor visually to position the camera view exactly in the hole. I'm looking for something automatic if this exists, or the most effective solution in any case.

I'm a true beginner.

enter image description here

Assumptions:

  • This final result is an image, not an animation.
  • The green rectangle and the camera have the same W/H ratio.
  • The image appears in the foreground, normal to the camera view axis, in the composited result. No perspective is involved.
$\endgroup$
  • $\begingroup$ So is the rectangle green hole moving across and changing size on each frame? If so you should edit your question to reflect all requirements. Is this question related to camera tracking? $\endgroup$ – atomicbezierslinger Sep 10 '15 at 16:10
  • $\begingroup$ Without revealing events of national or corporate or any security, how do you determine this green rectangle? Manually via human inspection or some other mechanism over which you have little control? If it is manual the proposed below answer is also manual. $\endgroup$ – atomicbezierslinger Sep 10 '15 at 16:16
  • $\begingroup$ For this exercise it seems you are motion tracking. You may have made this clear before. You want to track a green rectangle ... image recognition on some small scale. Camera Tracking a plane may be an excessively powerful alternative. Searching blender.org/manual/motion_tracking/index.html $\endgroup$ – atomicbezierslinger Sep 10 '15 at 16:28
  • $\begingroup$ In your random scenario, could the green rectangle, occupy .01 area of the entire image and move from the upper left corner to the bottom right corner of the screen in one frame? Small area and large movement? Truly random movement from one frame to next? $\endgroup$ – atomicbezierslinger Sep 10 '15 at 17:00
5
$\begingroup$

Here are some mathematics governing the transform:

  • W = pixel width of final scene
  • H = pixel height of final scene
  • x1,y1 = min coordinate of green screen
  • x2,y2 = max coordinate of green screen

You will feed the inside image into a scale node and then a translate node.

The values for the scale node will be

  • scale_x = (x2-x1)/W
  • scale_y = (y2-y1)/H

The values for the translate node will be

  • translate_x = (x2+x1-W)/2
  • translate_y = (y2+y1-H)/2

If you want to use the Relative checkbox, then divide those values by W and H respectively.

Keep in mind that blender's coordinate system is mathematical, not page coordinates, so max_y is at the top of the screen. (page coordinates usually put y=0 at the top instead)

compositor node setup and PiP

This math applies after you have determined the coordinates of the green rectangle.

Here is an operator that will let you pick a file and a color and the operator will rewrite your compositor node tree to mix the current 3D scene with the matte image. Be warned, this can be misled by stray matching pixels, so if your source images aren't properly constructed, you are boned:

http://web.purplefrog.com/~thoth/blender/python-cookbook/find-green-rectangle.html

bl_info = {
    "name" : "Composite into Matte image",
    "description": "An operator that configures the compositor to substitute the 3D scene into a chroma-keyed subset of the matte image chosen by the operator",
    "author" : "Robert Forsman <blender@thoth.purplefrog.com>",
    "version": (0,1),
    "blender": (2,74,0),
    #"location": "????",
    "warning": "",
    #"wiki_url": "http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/???",
    "category": "Node",
}
import bpy
from math import *

def get_image(path):
    matches = [ img for img in bpy.data.images if img.filepath==path]
    if len(matches)>0:
        return matches[0]
    rval = bpy.data.images.load(path)
    return rval


def matches( c1, c2, epsilon = 0.02):

    for i in range(len(c1)):
        if i<len(c2):
            b = c2[i]
        else:
            b=1
        if epsilon < abs(c1[i]- b):
            return False
    return True

def find_green_rectangle(img, chroma_key):
    # hold down the left mouse button in the UV/Image editor window to get the values for chroma_key

    width = img.size[0]
    #print (img.pixels[:4])
    pix = img.pixels[:]
    min_x = None
    max_x = None
    min_y = None
    max_y = None
    for i in range(0,len(pix), 4):
        one_pixel = pix[i:(i + 4)]
        if matches(one_pixel, chroma_key):
            x = floor(i/4)% width
            y = floor(i/4/ width)
            if min_x is None or x<min_x:
                min_x = x
                print([ i,x,y , one_pixel])
            if max_x is None or x>max_x:
                max_x = x
            if min_y is None or y<min_y:
                min_y = y
            if max_y is None or y > max_y:
                max_y = y
        #if i%100000 ==0:
        #    print(i)
    return [ min_x, min_y, max_x, max_y, width, img.size[1]]


def convert_bounds_to_node_parameters(scn, x1, y1, x2, y2, W, H):
    translate_x = (x2+x1-W)/2 /W
    translate_y = (y2+y1-H)/2 /H
    scale_x = (x2-x1)/W
    scale_y = (y2-y1)/H

    #scale_nodes = [ n for n in scn.node_tree.nodes if n.type=='SCALE']
    # use the N-panel in the compositor nodes to give your scale node this name
    scale_nodes = [ n for n in scn.node_tree.nodes if n.name=='matte scale']
    translate_nodes = [ n for n in scn.node_tree.nodes if n.name == 'matte translate']

    scale_nodes[0].inputs[1].default_value = scale_x
    scale_nodes[0].inputs[2].default_value = scale_y

    translate_nodes[0].inputs[1].default_value = translate_x
    translate_nodes[0].inputs[2].default_value = translate_y
    translate_nodes[0].use_relative = True

def wipe_and_rebuild_compositor_nodes(scn, img, wipe=True):

    scn.use_nodes = True
    while wipe and len(scn.node_tree.nodes)>0:
        scn.node_tree.nodes.remove(scn.node_tree.nodes[0])

    nodes = scn.node_tree.nodes
    links = scn.node_tree.links

    dx = 200
    image_n = nodes.new("CompositorNodeImage")
    image_n.location = (0,0)
    image_n.image = img

    #matte_n = nodes.new("CompositorNodeColorMatte")
    #matte_n.location = (dx,0)
    #links.new(matte_n.inputs[0], image_n.outputs[0])

    scale1_n = nodes.new("CompositorNodeScale")
    scale1_n.location = (2*dx,0)
    links.new(scale1_n.inputs[0], image_n.outputs[0])
    #links.new(scale1_n.inputs[0], matte_n.outputs[0])
    if True:
        scale1_n.space = 'RENDER_SIZE'
        scale1_n.frame_method = 'STRETCH'
    else:
        scale1_n.inputs[1].default_value = scn.render.resolution_x / img.size[0]
        scale1_n.inputs[2].default_value = scn.render.resolution_y / img.size[1]

    y2=-250
    layer_n = nodes.new('CompositorNodeRLayers')
    layer_n.location = (0,y2)

    scale2_n = nodes.new("CompositorNodeScale")
    scale2_n.location = (dx,y2)
    scale2_n.name = "matte scale"
    links.new(scale2_n.inputs[0], layer_n.outputs[0])

    translate_n = nodes.new("CompositorNodeTranslate")
    translate_n.location = (2*dx, y2)
    translate_n.name = "matte translate"
    links.new(translate_n.inputs[0], scale2_n.outputs[0])

    mix_n = nodes.new("CompositorNodeAlphaOver")
    mix_n.location = (3*dx, 0)
    links.new(mix_n.inputs[1], scale1_n.outputs[0])
    links.new(mix_n.inputs[2], translate_n.outputs[0])

    output = nodes.new('CompositorNodeComposite')
    output.location = (4*dx,0)
    links.new(output.inputs[0], mix_n.outputs[0])




#
#

class CompositeSceneIntoMatte(bpy.types.Operator):
    bl_idname = "compositor.scene_into_matte"
    bl_label = "Composite Scene Into Matte"
    bl_options = {'REGISTER', 'UNDO'}

    matte_filename = bpy.props.StringProperty(name="Matte file path",
                                              description="path to the image that provides the matte image",
                                              #default="/var/tmp/matte.png",
                                              subtype='FILE_PATH')
    matte_color = bpy.props.FloatVectorProperty(
            name="Matte chroma key",
            subtype='COLOR',
            default=(0, 1, 0),
            min=0.0, max=1.0,
            description="color from the matte image that marks where the 3D scene should be overlaid"
           )

    def execute(self, ctx):
        try:
            try :
                img = get_image(self.matte_filename)
            except :
                self.report({'WARNING'}, "failed to load image from filename %r"%self.matte_filename)
                return {'FINISHED'}

            bounds = find_green_rectangle(img, self.matte_color)
            if bounds[2] is None:
                self.report({'WARNING'}, "matte color %r does not appear in image"%self.matte_color)
                return {'FINISHED'}
            wipe_and_rebuild_compositor_nodes(ctx.scene, img)
            convert_bounds_to_node_parameters(ctx.scene, bounds[0], bounds[1], bounds[2]+1, bounds[3]+1, bounds[4], bounds[5])
            return {'FINISHED'}
        except BaseException as e:
            self.report({'ERROR'}, e.args[0])
            return {'CANCELLED'}

#
#

def register():
    bpy.utils.register_module(__name__)

def unregister():
    bpy.utils.unregister_module(__name__)

if __name__ == "__main__":
    unregister()
    register()

if False:
    bounds = find_green_rectangle(get_image("/var/tmp/matte.png"))
    print(bounds)

    scn = bpy.context.scene
    convert_bounds_to_node_parameters(scn, bounds[0], bounds[1], bounds[2]+1, bounds[3]+1, bounds[4], bounds[5])

To use this, follow the instructions in the Install the Addon section of the Addon Tutorial . Then use the Spacebar pop-up menu to invoke the Composite Scene Into Matte operator.

$\endgroup$
  • 1
    $\begingroup$ blender.org/manual/extensions/python/text_editor.html explains how to use the text editor, and the Run Script button is your friend. $\endgroup$ – Mutant Bob Sep 10 '15 at 18:27
  • $\begingroup$ I have expanded the example to use the formulas I outlined in the very beginning and actually alter the nodes in the compositor tree. If you have more than one scale and translate node, it might alter the wrong one, but that's a complication for another day. $\endgroup$ – Mutant Bob Sep 10 '15 at 21:03
  • $\begingroup$ Well done! It works exactly how I wanted. As I have two Scale, I have the problem of wrong node index as you mentioned, so you last touch to the code will be appreciated, however I'll select your answer right now. A note for other users: Be sure to adjust the code for your files (I changed get_image() as name = "xxx.png" return bpy.data.images.get(name) to simplify, but this will generate an error later if xxx.png is not found. Last: How to run the script automatically after the image is loaded and the matte is created? $\endgroup$ – mins Sep 11 '15 at 15:02
  • $\begingroup$ The new version is an interactive operator. $\endgroup$ – Mutant Bob Sep 11 '15 at 18:58
  • $\begingroup$ Wow, that was a champion derp on my part. Here's a new version with a better choice for the scaling of the matte. $\endgroup$ – Mutant Bob Sep 12 '15 at 14:14
6
$\begingroup$

You can use a Plane Track Deform: in doing so, you'll be able to apply the scene both on a rectangle or a deformed plane.

enter image description here

  1. Open the "matte image" in the Movie Clip Editor on "Tracling" mode
  2. Add 4 markers near the corners

enter image description here

  1. Select all the markers and add a Plane Track

enter image description here

  1. Drag the corners to fit the "hole" you want to fill with your scene
  2. Open the Node Editor and the Plane Track Deform (add > distort > plane track deform)

enter image description here

$\endgroup$
  • $\begingroup$ Good contribution. $\endgroup$ – atomicbezierslinger Sep 10 '15 at 16:58
  • $\begingroup$ Appreciate your answer, as it introduces me to another feature of Blender. However it seems not fit my scenario (I have added details in my post to better describe it). 1/There is no perspective involved, a Transform node might be good too for this point, 2/I'm looking for something automated, where the scene rendered is translated and scaled automatically based on the "hole" characteristics. $\endgroup$ – mins Sep 10 '15 at 17:08
  • $\begingroup$ You can use a rectangular shape (not distorted). The Plane Track Deform will translate and scale your scene automatically $\endgroup$ – parameciostudio Sep 10 '15 at 17:16
  • $\begingroup$ @cegaton you are right. I have changed the images in the answer $\endgroup$ – parameciostudio Sep 10 '15 at 18:57
  • $\begingroup$ @parameciostudio: Drag the corners to fit the "hole" seems a manual operation. What does it add to using a Transformnode and using the sliders instead of moving the 4 markers? $\endgroup$ – mins Sep 10 '15 at 19:46
0
$\begingroup$

Consider two planes in the 3d View, Orthogonal Camera. Both planes have image textures, are shadeless, have transparent portions, and thus illuminated at all times.

The further view is the background. The closer view is the foreground.

You can animate location on the X, Z axis, and size of the foreground. We assume your permanent view is View 1 Front Orthogonal.

$\endgroup$
  • $\begingroup$ Appreciate your effort in this crazy scenario :-) Ok I see what you suggest, but actually it is equivalent to using Transform, right? You play with the perspective if I understand. $\endgroup$ – mins Sep 10 '15 at 16:22
  • $\begingroup$ You are the judge if it fails as a solution firstly. But it allows keyframes on two different and potentially distant frames, thus less work potentially. Assumptions made by lack of understanding of the random element of your question. $\endgroup$ – atomicbezierslinger Sep 10 '15 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.