Put simply: not inherently. If you have a widget that does NOTHING at all…like, literally you add it and there’s no code for the widget outside of registering it, you’ll see SUCH a small performance hit from adding it that you would need to be looking at microtime() numbers to notice anything.
That said, as @cmegown hit on a little, it depends what the widgets do. If you have a widget that calculates pi or solves differential equations or something heinous like that (I can’t really think of a good example), then you’re gonna start looking at more and more of a performance hit.
Real world, where you’re going to see issues is widgets that make HTTP requests, be those feeds, stylesheets, or calls to cURL. The best advice I can give you is to know what your widgets are doing, and make sure they’re doing it right. Typically if you do that, everything else falls into place.