Driver core clears the driver data to NULL after device_release
or on probe failure, so just remove it from here.
Driver core change:
"device-core: Ensure drvdata = NULL when no driver is bound"
(sha1: 
0998d0631001288a5974afc0b2a5f568bcdecb4d)
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
 {
        struct amba_kmi_port *kmi = amba_get_drvdata(dev);
 
-       amba_set_drvdata(dev, NULL);
-
        serio_unregister_port(kmi->io);
        clk_put(kmi->clk);
        iounmap(kmi->base);