I found an alternative approach.
There is a unique class in body tag for each post type. e.g for portfolio post type i can use the CSS code something like mentioned below to hide the extra option.
.post-type-portfolio #post-format-audio, /* for radio button itself */
.post-type-portfolio .post-format-audio /* for option label */
{
display: none;
}
Hope it helps anybody else.