Can I configure WordPress to use postfix without a plugin?

WordPress uses the wp_mail() function to send mail. It says on the Codex article there that:

For this function to work, the settings SMTP and smtp_port (default: 25) need to be set in your php.ini file.

Also be sure to check that your contact form is sending the required parameters to the wp_mail() function. The required parameters are included on the page in the link above.