I won’t go into the code specifics right now, because I am not sure if you need me to. You essentially need to modify the SWFUpload JavaScript settings array to set the file_upload_limit to 1. Unfortunately I don’t believe SWFUpload allows you to change that settings variable after it has been inited, because it has already launched the Flash embed with the settings.
So, you would have to hook into the flash_uploader filter to disable it setting up the “normal” Flash uploader, then hook into the pre-upload-ui action and set up the Flash uploader using the same variables as WordPress does, but att the file_upload_limit setting. (Look at Line 1446 of media.php to see what I mean).
If you did it that way, you would also have to disable the HTML upload form so user’s can use that to bypass the Flash uploader.
Instead, you could (easier, but maybe not so usable to the user) hook into load-async-upload.php action (see Line 202 of admin.php) to reject the upload request if the current user had already uploaded an image for the given post. That would require the user selecting the image in the Flash Uploader, then it throwing an error once it has uploaded saying “You can not upload another file”