Depends on your plugin. I prefer to use plugins with the minimal functionality that meets my needs. Less configuration options, less documentation to read, and hopefully less bugs.
You also need to think on the upgrade process. If you make a change in a big plugin I should look into upgrading it even if the change is in a functionality I don’t use, and upgrades are always risky or take time to test.
Therefor, if you can break up your functionality into several plugins, that is the way to go IMO. But if it is not possible for what you do, then before you think about distributing addons for your plugin, think how are they going to be installed and upgraded. Unless you have a good plan for managing the addons I suggest you distribute them with the main plugin and then you can directly include them without the need for GLOB_ONLYDIR.