bytes objects do not have a __format__ method of their own, so the default from object is used:
>>> bytes.__format__ is object.__format__
True
>>> '{:20}'.format(object())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
It just means that you cannot use anything other than straight up, unformatted unaligned formatting on these. Explicitly convert to a string object (as you did by decoding bytes to str) to get format spec support.
You can make the conversion explicit by using the !s string conversion:
>>> '{!s:20s}'.format(b"Hi")
"b'Hi' "
>>> '{!s:20s}'.format(object())
'<object object at 0x1100b9080>'
object.__format__ explicitly rejects format strings to avoid implicit string conversions, specifically because formatting instructions are type specific.