I always wondered how to deform an object being grabbed , i usually use a shrinkwrap modifier for more simple collision, but it do not works very well for more complex cases...I also tried cloth simulation a few times , but the object always end up falling toward infinity..
The blend file if you have an idea: https://uploadnow.io/f/yj0lLWD
It's going to depend heavily on the details of the shapes involved, the quality required, whether you're making a still or an animation.
Before looking at options, we should talk first about topology. We have to care about topology for any meshes that are going to deform, because they're only ever going to deform along the edges of the mesh-- so how those edges are laid out is very important. We can use a lot of vertices to make a lot more edges, but we'll probably be using subdivision to do that, and subdivision depends on certain properties of topology as well.
You're using a UV sphere for your ball mesh. This won't deform well or subdivide well. Instead, let's use a cube-sphere. This will have evenly distributed faces and angles. Make a cube, give it a few levels of subdivision, apply the modifier, use a to-sphere operation in edit (also doable with a cast modifier) to make it fully spherical.
In this case, yes, shrinkwrapping is doable, but we need to tune some parameters of the shrinkwrap:
We subdivide before the shrinkwrap to generate additional vertices.
The shrinkwrap is set to "project" mode which is appropriate for a sphere like this-- it will only ever project along its normals. It is set to cull backfaces, so it will only shrinkwrap to front faces of the hand (rather than shrinking to the back of of the hand.) It will project only in its negative normal, so it will only ever get smaller with this shrinkwrap, not bigger.
We subdivide one more time after the shrinkwrap to even out the shrinkwrap. Then, we tune the shrinkwrap's offset to eye.
This is not always a great technique. It won't be good for animation, because shrinkwraps don't interpolate smoothly-- they depend too heavily on high-frequency details of both meshes.
Physics is an option for animation, but it's not very appropriate here, because your hand is already deeply in collision with the ball. If I were going to use collision to do this, I would probably use soft body physics like so:
I've extruded only vertices, then merged them at the mesh, and assigned this vertex to a vertex group. This creates internal forces for a soft body to hold it together (otherwise, it acts more like a hollow ball.) I can use the vertex group to assign this center vert a higher goal setting than the rest of the mesh and to mask away this vertex after running physics. For the physics, we'll want to use face collision, and we would be best off using a much larger scale than you're using in this file (applying scale on all meshes involved in the simulation.)
But one of the best solutions is, make a shapekey and manually edit the mesh. We could make that shapekey, if we wanted, by starting with the shrinkwrap that we already had and applying it as a shapekey-- then, we could make manual edits on top of it; if we needed to, we could mix this with shapekeys in other positions.
