This creates the multidimensional array in advance and then assigns it to options. The multidimensional array is created by assigning an array to each element of the main array. I have reused the $feat_pages_terms variable, but you can also create a separate variable using array() and assign array elements to that, I am including those lines as commented lines. When using this option, $key is not needed as new elements are simply being appended.
function shortcode_ui_feat_pages()
{
$feat_pages_terms = get_terms(array('taxonomy' => 'featured_page_category', 'hide_empty' => false));
//$multiarr = array();
foreach($feat_pages_terms as $key => $feat_page_term) :
$feat_pages_terms[$key] = array('value' => $feat_page_term->name, 'label' => $feat_page_term->name);
//$multiarr[] = array('value' => $feat_page_term->name, 'label' => $feat_page_term->name);
endforeach;
shortcode_ui_register_for_shortcode(
'featured-pages',
array(
'label' => 'Featured Pages Shortcode',
'listItemImage' => 'dashicons-editor-table',
'attrs' => array(
array(
'label' => 'Category',
'attr' => 'category',
'type' => 'select',
'options' =>$feat_pages_terms/*$multiarr*/,
'description' => 'Insert the category full name (i.e. Home page)',
),
...