Allow roles below admin to add subscribers only

This first step to making this happen is add the capability to create_users to a given role. You do this be calling get_role, then using the add_cap method. This only needs to be done once. Here’s an example that does it on plugin activation.

<?php
register_activation_hook( __FILE__, 'wpse42003_activation' );
function wpse42003_activation()
{
    foreach( array( 'editor', 'your_custome_role' ) as $r )
    {
        $role = get_role( $r );
        if( $role )
            $role->add_cap( 'create_users' );
    }
}

If you do that, you probably want to make sure to undo it on plugin deactivation with remove_cap.

<?php
register_deactivation_hook( __FILE__, 'wpse42003_deactivation' );
function wpse42003_deactivation()
{
    foreach( array( 'editor', 'your_custome_role' ) as $r )
    {
        $role = get_role( $r );
        if( $role )
            $role->remove_cap( 'create_users' );
    }
}

Now the problem of making sure those users can only register subscribers. On the user-new.php page, the roles drop down get generated by a function called wp_dropdown_roles, which in turn calls the function get_editable_roles to fetch the roles the current user can edit/create.

Fortunately for you, there’s a filter in there that will allow us to modify this bit. First we’ll copy all the $roles array keys, then loop through and unset any role other than subscriber from $roles — only if the current user has the role editor or your custom role.

<?php
add_filter( 'editable_roles', 'wpse42003_filter_roles' );
function wpse42003_filter_roles( $roles )
{
    $user = wp_get_current_user();
    if( in_array( 'editor', $user->roles ) || in_array( 'your_custom_role', $user->roles ) )
    {
        $tmp = array_keys( $roles );
        foreach( $tmp as $r )
        {
            if( 'subscriber' == $r ) continue;
            unset( $roles[$r] );
        }
    }
    return $roles;
}

The bonus part of this is that get_editable_roles is called before a user is added or updated — your editors won’t be able to add users by insert extra options in the form with JS or something similar.

note: change your_custom_role in the examples above to your role’s name.

All that as a plugin.