TinyMCE 4.7.11 custom styles using styleselect – what does the ‘attributes’ argument do?

I’m not sure this is even the intended use for the argument

Yes, it is. Use the attributes option to add HTML attributes (e.g. class, title, etc.) to the generated or target element (e.g. span).

So based on your example:

array(  
    'title'         => 'Link Resource',
    'selector'      => 'a', 
    'classes'       => 'smart-link-external smart-link-resource',   
    'wrapper'       => false,
),

When you select a link (i.e. an a element) in the TinyMCE editor, and then apply the Link Resource style/format to that element/text, TinyMCE will add smart-link-external smart-link-resource to the element’s/link’s class attribute:

<!-- Before you applied "Link Resource" -->
<a href="https://wordpress.org/">WordPress</a>

<!-- After you applied that style/format -->
<a class="smart-link-external smart-link-resource"
  href="http://google.com">WordPress</a>

I was hoping to use the attributes argument to add things like
rel=”external”.

So if you change your code to:

array(  
    'title'         => 'Link Resource',
    'selector'      => 'a', 
    'classes'       => 'smart-link-external smart-link-resource',   
    'wrapper'       => false,
    // The change: Add the `attributes` array.
    'attributes'    => array(
        'title' => 'My Link Resource',
        'rel'   => 'external',
    ),
),

The link’s markup would change to:

<a class="smart-link-external smart-link-resource"
  title="My Link Resource" href="https://wordpress.org/" rel="external">WordPress</a>

So as you can see, the link/element now has these two new attributes: title and rel.

[DEMO] Try this Pen: Select the text/link “TinyMCE” in the heading in the TinyMCE editor, then choose “Link Resource” from the “Formats” drop-down menu in the editor’s second toolbar row. Then go to “View → Source code” or “Tools → Source code” via the first toolbar row.

The corresponding JS/TinyMCE code:

{
    title: 'Link Resource',
    selector: 'a',
    attributes: {
        title: 'My Link Resource',
        rel: 'external',
        'class': 'smart-link-external smart-link-resource'
    },
    wrapper: false
}