As per comment trying to do this in one action will probably cause unholy pagination issues.
It might be easier to limit main query to normal posts, query posts with formats separately (depending on what you have in normal posts by date) and mix two sets together on display.