mlx4_core: fix FMR unmapping to allow remapping afterward

mlx4_core: fix FMR unmapping to allow remapping afterward

The FMR common use flow (as implemented in fmr_pool) is:
 - Allocate FMR (ib_alloc_fmr)
 - Use the FMR to remap DMA memory until remaps limit
   exceeded (ib_map_phys_fmr)
 - Unmap the FMR (ib_unmap_fmr)
 - Use the FMR to remap DMA memory until remaps limit
   exceeded (ib_map_phys_fmr)
 - ...

The current implementation of mlx4_fmr_unmap is not following
this use flow since it is using the HW2SW MPT command.
The HW2SW MPT command notifies the FW that the MPT entry is
not used by HW anymore. The FW may act according to this information,
therefore it is not safe for the driver to manipulate the MPT
directly.  The patch fixes this by manipulating the MPT directly
to unmap the memory instead of using the HW2SW MPT command.

Signed-off-by: Moshe Lazer <moshel@mellanox.com>
(Ported from Mellanox OFED 2.4)

Signed-off-by: Mukesh Kacker <mukesh.kacker@oracle.com>