Yes, you'll purge first, but here's a problem for you. What you should have done was enable the repos where you found the older package versions, (was it stable or testing?) and disable the unstable repo. You say you "downloaded" but what if those three packages have dependencies, and those dependencies are located in repos that you do not have enabled?
So, you will have to enable the repos where you found the 19.6-1 packages, and disable the unstable repos, and then apt update
and then apt install
and install the three packages that you downloaded, FROM THE REPO, not using the downloaded packages.
This could work -- good luck! And of course we still don't know whether the xserver packages are even the source your slow KDE packages, so .........