Jumpy table when navigating via the pagination

After adding some custom rendering using the "rowCallback": function (only way I could think to do this) my table is jumpy when navigating using the pagination, I think it is down to the buttons rendering in.

The page is live here: https://vccalc.vapingcommunity.co.uk/flavours

I am using an AJAX call to determine the button rendered, so Add to Stash or Remove from Stash is shown depending on the AJAX result.

"rowCallback": function (row, data) {
checkInStash(exid, data.flavour_name, data.flavour_company_name).done(function(data){ 
    if(data == true) {
        $(row).addClass( 'instash' );
        $(row).find('td:eq(4)').html('<button class="btn btn-primary flavour-stash">Remove From Stash</button>');
    } else {
        $(row).removeClass( 'instash' );
        $(row).find('td:eq(4)').html('<button class="btn btn-primary flavour-stash">Add To Stash</button>');
    }
});
},

This is the AJAX success function:

success: function(data){
    table.row( 4 ).invalidate().draw('page');
},