Create the matrix world
The rotation part of an objects world matrix is an orthogonal matrix where the columns represent the local axes.
To demonstrate, in test code below, I have added a plane and removed one vertex to visualize the three points A, B and C ensuring the angle between BA and CA is 90 degrees.
Cube (orange wireframe) aligned to 3 vertex, 2 edge "plane" Vertex 0 for B, 1 for A and 2 for C
The vectors BA and CA, normalized,, are used for the x and y axis directions.
The cross product of x and y is the z axis. (The negated value would be the "flipped" option)
Finally build the matrix world from the 3 axis vectors, make it 4x4 and the translation A.
Test script. Move and rotate the "plane" then run script to align "Cube" such that its origin is at A, and the x and y axis align.
import bpy
from mathutils import Matrix
context = bpy.context
scene = context.scene
tri = scene.objects.get("Plane")
# global coords of 3 verts in plane.
b, a, c = [tri.matrix_world @ v.co for v in tri.data.vertices]
x = (b - a).normalized()
y = (c - a).normalized()
z = x.cross(y)
M = Matrix([x, y, z]).transposed().to_4x4()
M.translation = a
# Cube is the test object
ob = scene.objects.get("Cube")
ob.matrix_world = M
Note I have used blender 2.8. Replace matrix - vector multiplication @
with *
for prior versions.