One-to-many post relationships that are displayed by category (using posts-to-posts plugin)

I’m not that familiar with the posts-to-posts Plugin, but your main problem may be that the category page doesn’t know which building you chose in the step before if you don’t transmit it in a parameter or save it in a session or cookie.
Maybe it would be better to output the connected items on the buildings page, like this:

 Building (description etc)
 - category 1
       -item 1
 - category 2
         -item 2
         -item 3
         ...