You should never ever modify WordPress Core files. WordPress has an Plugin API (http://codex.wordpress.org/Plugin_API) that allows you to modify WordPress to your liking without changing Core code.
In your case, the solution is to check the contents of the custom meta box fields before they are saved via the save_post action.
If the content is not valid, you can display a message on the post screen using the admin_notices action.