xen-netfront: generalize recycling for grants

xen-netfront: generalize recycling for grants

Takes the already existent mechanism for recycling pages and leverages it
for grant references too. The difference though is that pages permanently
granted to the backend cannot be revoked (because those are mapped by the
other side) and hence these need to go to a separate quarantine pool, until
the point these pages can be consumed. The strategy is: 1) Get a page by
fetching oldest entry in rx_pool 2) If it's not granted then the page is
freed at the head 3) if it's reusable return the page otherwise add it to
quarantine pool 4) fetch oldest entry in quarantine pool and finally 5) if
all else fails then we resort to allocating a new page. Worst case scenario
if we have two atomic read op added on packet path when allocating a new
page for Rx requests.

This page reuse strategy allows us to remove a copy for each page handed
over by the backend leveraging guest RX performance to ~42-47 Gbit/s when
testing backend -> frontend. The measured recycling percentage is about
30% on TCP streams if pool size == ring size; and with pool size == 2 *
ring size these rises up to 80 - 100%.  This shows that bigger ring sizes
should allow for better recycling, which remains to be explored.

The only downside of this approach is that it is not 100% guaranteed that
the Rx requests provided to the backend will be already mapped; in other
words, backend may need to do a grant copy on 1% of the packets.
This is not the case though when we are in full copy mode whereby we always
reuse the same grants while copying into new pages into the upper layers.

Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Reviewed-by: Shannon Nelson <shannon.nelson@oracle.com>
Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Orabug: 26107942