MTD-Utils: fix handling of ioctl return value in nand-utils

MTD-Utils: fix handling of ioctl return value in nand-utils

Older kernel do not implement the MTDFILEMODE ioctl. In this case
nandwrite and nanddump should have used MEMGETOOBSEL in combination
with MEMSETOOBSEL. Unfortunately the return value of the unsucessfull
ioctl is not -ENOTTY, but -1 and errno contains ENOTTY. This change
fixes this issue. I have not tested all cornercases. Would be
good if someone could do more careful testing than I did, or maybe
reviewing is sufficient in this case.

Signed-off-by: Frank Haverkamp <haver@vnet.ibm.com>
Signed-off-by: Josh Boyer <jwboyer@gmail.com>