Most likely it means that the open didn’t fail.
When Perl opens a file, it checks whether or not the file is a TTY (so that it can answer the -T $fh filetest operator) by issuing the TCGETS ioctl against it. If the file is a regular file and not a tty, the ioctl fails and sets errno to ENOTTY (string value: “Inappropriate ioctl for device”). As ysth says, the most common reason for seeing an unexpected value in $! is checking it when it’s not valid — that is, anywhere other than immediately after a syscall failed, so testing the result codes of your operations is critically important.
If open actually did return false for you, and you found ENOTTY in $! then I would consider this a small bug (giving a useless value of $!) but I would also be very curious as to how it happened. Code and/or truss output would be nifty.