rds: re-entry of rds_ib_xmit/rds_iw_xmit

rds: re-entry of rds_ib_xmit/rds_iw_xmit

The BUG_ON at line 452/453 is triggered in function rds_send_xmit.

 441                         while (ret) {
 442                                 tmp = min_t(int, ret, sg->length -
 443                                                       conn->c_xmit_data_off);
 444                                 conn->c_xmit_data_off += tmp;
 445                                 ret -= tmp;
 446                                 if (conn->c_xmit_data_off == sg->length) {
 447                                         conn->c_xmit_data_off = 0;
 448                                         sg++;
 449                                         conn->c_xmit_sg++;
 450                                         if (ret != 0 && conn->c_xmit_sg == rm->data.op_nents)
 451                                                 printk(KERN_ERR "conn %p rm %p sg %p ret %d\n", conn, rm, sg, ret);
 452                                         BUG_ON(ret != 0 &&
 453                                                conn->c_xmit_sg == rm->data.op_nents);
 454                                 }
 455                         }

It is complaining that total sent length is bigger that we want to send.

rds_ib_xmit() is wrong for the second entry for the same rds_message returning
wrong value.

The sg and off passed by rds_send_xmit() to rds_ib_xmit() is based on
scatterlist.offset/length, but the rds_ib_xmit() action is based on
scatterlist.dma_address/dma_length. In case dma_length is larger than length
there is problem. for the 2nd and later entries of rds_ib_xmit() for same
rds_message, at least one of the following two is wrong:

1) the scatterlist to start with, the chosen one can be far beyond the correct
   one.
2) the offset to start with within the scatterlist.

Fix is to add op_dmasg and op_dmaoff fields to rm_data_op structure indicating
the scatterlist and offset within the it to start with for rds_ib_xmit()
respectively. The op_dmasg and op_dmaoff fields are initialized to zero
when doing dma mapping for the first time of the message and are changed
when filling send slots.

The same applies to rds_iw_xmit() too.

upstream commit: d655a9fbc8a51ac8d92db7ff5a599aab17dce3ca

Orabug: 21324078

Signed-off-by: Wengang Wang <wen.gang.wang@oracle.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>