Revert "RDS: don't commit to queue till transport connection is up"

Revert "RDS: don't commit to queue till transport connection is up"

This reverts commit 238a807df5e57afb4b1e13ba87015093e3212247.

This change was introduced to address the RDS internal sendQ
occupancy for the messages targeted to dead/non-existing nodes. It was
discovered as part of a customer issue where remote node was shut down
and RDS attempted to repeatedly establish a connection without success.
Same issue can be exploited by sending messages to non-existing node
too since RDS forms connection as part of sendmsg if it doesn't
exist already.

While at that time sending EAGAIN instead of adding messages to
sendQ when remote connection not up, looked straightforward, it
has undesired effect on application to keep spinning even though
there is space to write on socket buffer. And application has
no notion of underneath connections, so RDS needs to handle this
problem internally and transparently. Application will automatically
move to POLL OUT once its own socket buffer is full and will
avoid the CPU tight spinning.

To address draining the internal sendQ messages targeted to
dead nodes or non-existing nodes, one possible way is to
retire/destroy those connections, after some large timeout. That
will also drop those messages from sendQ. This change will be
addressed separately.

Orabug: 27606911

Signed-off-by: Santosh Shilimkar <santosh.shilimkar@oracle.com>
Reviewed-by: Wei Lin Guay <wei.lin.guay@oracle.com>