I think this is because of gdal-data that is being held back. I think you need to force the upgrade with 'apt install gdal-data'. Some libs will be removed, but I did not see anything of importance. After doing that, there were no removals for me.
But be careful! Check carefully what would be removed before proceeding. I cannot guarantee it is the right thing to do, just that it seemed to work for me.