There are roughly following factors to consider:
- Does it ship with WP core
- How easy it is to integrate with WP
- How easy it is to use in public WP extension
Let’s go over these for both.
Bootstrap
Bootstrap doesn’t ship with WordPress core, so your implementation will need to provide it.
While markup of the Bootstrap is rather pleasant by itself, it requires degree of customization, which many parts of WordPress are horrible at. Typically in WordPress development you start with markup that WP gives you and go from there. Starting with external idea of markup and aligning WP output to it is always an extra level of challenge.
Extreme popularity of Bootstrap makes it likely to encounter multiple copies of its styles and/or scripts when using public extensions. So far there hadn’t been enough attention paid to it by developers. Properly shipping Bootstrap or parts of it in code for distribution requires carefully scoping them to your functionality. Again — extra level of development challenge.
jQuery UI
jQuery UI ships with WordPress core, which makes it easy to share and unnecessary to ship.
It is interesting to note, however, that it seems to be quite underused in actual development. WordPress has historically been PHP–centric. Despite quite forceful push for JS–driven functionality, JS still has bit of an under-known (and a lot of horribly undocumented) aspect to it in WP development.
Overall
There is no clear winner here. Both dependencies have their own implementation challenges and while jQuery UI wins in bundling it loses (in my perception) in popularity and ecosystem (Bootstrap has grown not just itself, but wide array of third party extensions and solutions, depending on it).
Personally I use Bootstrap, but my extensions are mostly personal and/or quite niche.