xen-netback: copy buffer on xenvif_start_xmit

xen-netback: copy buffer on xenvif_start_xmit

Normally packets are enqueued from ndo_start_xmit into rx_queue
which is internal to netback. The guestrx thread will then pick
these up, create the grant copy ops (while coaslescing them as
much) and notify frontend. Although most packets now endup being
memcpy-ed directly from netback (instead of through Xen). As a result
guestrx thread ends up waiting more (and woken up by transmit function)
which leads to higher contention on the wait queue as seen with
lock_stat:

class name    con-bounces    contentions   waittime-min   waittime-max
waittime-total   waittime-avg    acq-bounces   acquisitions   holdtime-min
holdtime-max holdtime-total   holdtime-avg
--------------------------------------------------------------------------
&queue->wq:   792            792           0.36          24.36
1140.30           1.44           4208        1002671           0.00
46.75      538164.02           0.54
----------
&queue->wq    326          [<ffffffff8115949f>] __wake_up+0x2f/0x80
&queue->wq    410          [<ffffffff811592bf>] finish_wait+0x4f/0xa0
&queue->wq     56          [<ffffffff811593eb>] prepare_to_wait+0x2b/0xb0
----------
&queue->wq    202          [<ffffffff811593eb>] prepare_to_wait+0x2b/0xb0
&queue->wq    467          [<ffffffff8115949f>] __wake_up+0x2f/0x80
&queue->wq    123          [<ffffffff811592bf>] finish_wait+0x4f/0xa0

with staging grants:

&queue->wq:   61834          61836           0.32          30.12
99710.27           1.61         241400        1125308           0.00
75.61     1106578.82           0.98
----------
&queue->wq     5079        [<ffffffff8115949f>] __wake_up+0x2f/0x80
&queue->wq    56280        [<ffffffff811592bf>] finish_wait+0x4f/0xa0
&queue->wq      479        [<ffffffff811593eb>] prepare_to_wait+0x2b/0xb0
----------
&queue->wq     1005        [<ffffffff811592bf>] finish_wait+0x4f/0xa0
&queue->wq    56761        [<ffffffff8115949f>] __wake_up+0x2f/0x80
&queue->wq     4072        [<ffffffff811593eb>] prepare_to_wait+0x2b/0xb0

To avoid such contention we copy the packet directly on ndo_start_xmit
which avoid the kicking of this thread. Although with recycling of grants
it is *not* fully guaranteed that all copies will be done by netback, and
as a result handling all packets on start_xmit could potentially lead to
a very high number of hypercalls per packet and therefore affect
throughput. Only with a copy frontend would we have this guarantee. Hence
for now we hide this behind a module parameter (skip_guestrx_thread)
which by default is false.

Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Reviewed-by: Shannon Nelson <shannon.nelson@oracle.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Orabug: 26107942