Passing a hexadecimal value into a module in Verilog

Verilog treats all bare numeric literals as decimal. A and D are not legal decimal values.

For hexadecimal literals, you need to specify the literal type using 'h:

key_schedule i1('h0A); // works
key_schedule i1('h1D); // works

Refer to the IEEE Std (1800-2009, for example), section “Numbers”.

The following code compiles for me without errors on 2 different simulators (Incisive and VCS):

module tb;
    key_schedule i1(5'h1A);
    key_schedule i2('h1A);
endmodule

module key_schedule (input [4:0] in);
    always @(in) $display(in);
endmodule