are there any downsides to increasing the number of most recent items in a feed

You can increase the number of items safely. Feed creation is pretty fast. It is just a simple loop through the latest posts, no scripts are loaded, no pagination, widgets or other expensive operations.

For your subscribers, more items are better, because they can explore your content faster. Make sure to use a full text feed, excerpts are rarely useful.