I checked out your scene and adjusted the physics to be more in line with the situation you asked for.
You used only one empty in the hinges each, but since they work as a position marker on the first object selected in their constraint options, it was kinda the wrong way.
If you want the board(you called it physics) object to be anchored with 4 points, you need to place 4 empties on each corner of the board, (UPDATE: not needed to parent them, can cause glitches) and use the board itself as first target in the "rigid body constraint".
Afterwards you set the empties as "generic springs" and select the hinge on the corresponding side as second target to the empties each. You should now have 2 empties on each side of the board connected to one hinge each.
You can find the set up here:
(keeping this version available, newer file below!)
I changed one of the constraints to be weaker making the board slightly lower on that edge, and the ball rolls down. You can adjust the constraints and tinker to get what you may want it to look like.
Looks like this:
UPDATE: un-parented the constraint empties and added visual connections.
Having the constraints parented under the board caused the glitching.
They don't need to be parented to work, so easy solution, only downside is if you want to move the board, you need to move the constraint-empties as well(select all and you should be fine). May be a bug, that it glitches if parented, don't know.
Here the new version: