What is this format called (nested shortcodes)?

I would use the phrase

Nested Shortcodes

to describe it.

You can find many questions using that phrase here on WPSE.

The Codex also uses it, for example here:

http://codex.wordpress.org/Shortcode_API#Nested_Shortcodes

Since the shortcodes in your example are related to the layout, we could perhaps call it:

Nested Layout Shortcodes

to be more specific.

If you have another phrase in mind, or a better suggestion, please share it with us.