Intersect point with line.
Rather than use the bmesh scale operator, another technique using the method
mathutils.geometry.intersect_point_line(...)
Takes a point and a line and returns a tuple with the closest point on
the line and its distance from the first point of the line as a
percentage of the length of the line.
To find the radial from each vertex to the axis of scale ie the active edge
. For each vertex v
in selection, find the closest point o
to the line defined by the edge . Translate the vert along vector v.co - o
from the point o
by a scale factor.
Quick code, the verts of the edge could be removed from the selection. Since they are on the line there will be no (except error tolerance) transformation anyway.
import bpy
import bmesh
from mathutils.geometry import intersect_point_line
context = bpy.context
ob = context.edit_object
me = ob.data
bm = bmesh.from_edit_mesh(me)
scale_factor = 2
edge = bm.select_history.active
# if isinstance(edge, bmesh.types.BMEdge):
verts = [v for v in bm.verts if v.select]
for v in verts:
o, pc = intersect_point_line(v.co, edge.verts[0].co, edge.verts[1].co)
v.co = o + scale_factor * (v.co - o)
bmesh.update_edit_mesh(me)

Result of running on default cube, with active edge and its diagonally opposed edge selected