There are 2 ways to delete and reconfigure a TFS collection on your desktop (or local machine).
Using Visual Studio.
Using Team Explorer, you can first click on Connect. If you are already connected to a collection, you will not be able to delete it. You will have to Disconnect by clicking on the Context Menu.
Once disconnected, you can click on Remove. This method works most of the time. However if you have many such collections configured, then you can use the registry mechanism.
Also, please note that Team Explorer does not a collection unless and until it has a project associated with it. So to delete that you will have to use Registry Mechanism
Team Explorer stores the configured collections in registry folder
You can delete one of the sub-folder in Instances to delete the collection reference from Visual Studio
In the above snapshot, you can see 3 instances (collections) configured; however, in Team Explorer only 2 were visible as one of the collections does not have any project associated.
If you want to reconfigure your collection, you can again click on Connect in Team Explorer and follow through the wizard.