I am not new in Blender but in rigid physics.
How to get a pendelum swinging down / rotating around the axis in case of model weight?
Blender Stack Exchange is a question and answer site for people who use Blender to create 3D graphics, animations, or games. It only takes a minute to sign up.Sign up to join this community
Use a Rigid Body Constraint to fix the pendulum to the pivot as a hinge instead of relying on the geometry (select the pendulum and thr pivot, in the physics toolbox panel select ‘Connect’ and set as ‘hinge’ - you might then need to re-orient the hinge so it swings on the correct axis (it will rotate around the Z-axis of the constraint)).
Finally, you need to move the origin of the pendulum to its centre of mass (currently it appears to be at the pivot) - select the pendulum, click Set Origin and select ‘Center of Mass’.
See also https://blender.stackexchange.com/a/109136/29586 - it doesn’t include a pendulum but does explain the Rigid Body Constraints. Also https://blender.stackexchange.com/a/101880/29586 for an explanation as to why to use constraints for pivots and axles rather than mesh surface interactions.