ColVis - Prevent Refresh On Serverside Table Unles Re-Sorted or Page Changed?

I'm using ColVis to show/hide columns, and it works great. The only issue is that I'm using serverside pagination, as there are thousands of rows, and I don't really find it necessary to reload the ajax every time a column is hidden or shown, since the data and column are there, just hidden.

Is there some way I can prevent ColVis from reloading the table data unless the page is changed or one of the columns is reordered?

Thanks!