Sort get_users by custom field

I have a custom USER taxonomy called ‘label’ which also has a custom field of ‘sort_order’ – I am trying to return a list of users that have the label taxonomy and then sort this list by the sort_order field.

So far I have…..

<?php
$users = get_users(
array(
    'meta_key' => 'sort_order',
    'fields' => 'all_with_meta'
)
);

function wpse98580_sort_order( $a, $b )
{  
    if ( $a->sort_order === $b->sort_order ) {  
    return 0;  
} elseif ( $a->sort_order > $b->sort_order ) {
    return -1;
}
return 1;  
}  
usort( $users, 'wpse98580_sort_order' );

/* Iterate over the sorted array */

foreach( $users as $user )
{
echo '<h4>' . $user_info-> user_firstname . ' ' . $user_info-> user_lastname . '</h4>';
}
?>

This doesn’t work, I am assuming it is becasue the ‘sort_order’ meta-key is actually a custom field of the taxonomy so it can’t retrieve it. Usually I use the following code to access the sort_order…

$product_terms = wp_get_object_terms($user->ID, 'label');
                    if(!empty($product_terms)){
                    if(!is_wp_error( $product_terms )){
                        foreach($product_terms as $term){

                        $t_ID = $term->term_id;
                        $label_custom_fields = get_option("taxonomy_term_$t_ID");
                        echo $label_custom_fields['sort_order'];

                        }
                        }
                    }

Can anyone help?

By using this you can get your result in ascending order for sort_order custom field

$users = get_users(
   array(
   'meta_key'=> 'sort_order',
   'orderby' => 'meta_value_num',
   'order'   => 'ASC'
   )
);

foreach( $users as $user )
{
   echo '<h4>' . $user_info-> user_firstname . ' ' . $user_info-> user_lastname . '</h4>';
}