ice: Add support for devlink local_forwarding param

ice: Add support for devlink local_forwarding param

Add support for driver-specific devlink local_forwarding param.
Supported values are "enabled", "disabled" and "prioritized".
Default configuration is set to "enabled".

Add documentation in networking/devlink/ice.rst.

In previous generations of Intel NICs the transmit scheduler was only
limited by PCIe bandwidth when scheduling/assigning hairpin-bandwidth
between VFs. Changes to E810 HW design introduced scheduler limitation,
so that available hairpin-bandwidth is bound to external port speed.
In order to address this limitation and enable NFV services such as
"service chaining" a knob to adjust the scheduler config was created.
Driver can send a configuration message to the FW over admin queue and
internal FW logic will reconfigure HW to prioritize and add more BW to
VF to VF traffic. An end result, for example, 10G port will no longer
limit hairpin-bandwidth to 10G and much higher speeds can be achieved.

Devlink local_forwarding param set to "prioritized" enables higher
hairpin-bandwitdh on related PFs. Configuration is applicable only to
8x10G and 4x25G cards.

Changing local_forwarding configuration will trigger CORER reset in
order to take effect.

Example command to change current value:
devlink dev param set pci/0000:b2:00.3 name local_forwarding \
        value prioritized \
        cmode runtime

Co-developed-by: Michal Wilczynski <michal.wilczynski@intel.com>
Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
Reviewed-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Signed-off-by: Pawel Kaminski <pawel.kaminski@intel.com>
Signed-off-by: Wojciech Drewek <wojciech.drewek@intel.com>
Tested-by: Rafal Romanowski <rafal.romanowski@intel.com>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>