setuptools fails installing add-on dependency

Question

I have a in-house python add-on tool that requires some 3rd party dependencies to work. One of them, a proprietary one, seems to fail specifically on my machine. With Ubuntu 20.04 where Blender 2.93 LTS is used.

I've managed to reproduce the error with a dummy package, that installs in the same way as the conflicting package does, to work as dependency with the following procedure.

 $ mkdir -p ~/mockpkg/pkg
 $ touch ~/mockpkg/pkg/__init__.py
 $ cat <<EOF > ~/mockpkg/setup.py
 > from setuptools import setup
 > 
 > setup(name='mockpkg',
 >       version='0.0.1',
 >       packages=['pkg'])
 > EOF

The dependency resolution is handled by the add-on itself when configuring it in the preferences menu. The following script installs the dependency in the same way the add-on does. Running it inside Blender reproduces the same issue I hit.

import bpy
import os
import sys
import subprocess

class InstallDependency(bpy.types.Operator):
    bl_idname = "object.install_dependency"
    bl_label = "Install Dependency"

    def execute(self, context):
        os.environ["PYTHONNOUSERSITE"] = "1"
        subprocess.run([sys.executable, "-m", "pip", "install",
           os.environ["HOME"] + "/mockpkg"], check=True)
        return {'FINISHED'}

def register():
    bpy.utils.register_class(InstallDependency)

def unregister():
    bpy.utils.unregister_class(InstallDependency)

if __name__ == "__main__":
    register()
    bpy.ops.object.install_dependency()

Resulting in the following error.

Switching to fully guarded memory allocator.
Blender 2.93.8
Build: 2022-02-02 00:34:48 Linux release
argv[0] = Programs/blender-2.93.8-linux-x64/blender
argv[1] = -d
read file 
  Version 293 sub 18 date 2021-04-16 16:00 hash 463b38b0e0b0
Processing ./mockpkg
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Python: Traceback (most recent call last):
  File "/dependency.py", line 12, in execute
  File "/home/user/Programs/blender-2.93.8-linux-x64/2.93/python/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/user/Programs/blender-2.93.8-linux-x64/2.93/python/bin/python3.9', '-m', 'pip', 'install', '/home/user/mockpkg']' returned non-zero exit status 1.

location: /home/user/Programs/blender-2.93.8-linux-x64/2.93/scripts/modules/bpy/ops.py:132

However if I resolve the dependency manually, that is in the following manner, it does install without any trouble.

 $ /home/user/Programs/blender-2.93.8-linux-x64/2.93/python/bin/python3.9 -m pip install /home/user/mockpkg

While this helps me to keep working in the meantime. I'll like to address the problem to avoid the user requirement to type any commands when deploying the add-on.