As Dana mentioned, unloading and then detaching is not necessary. If you want the xref gone, just detach it.
As for the xref not detaching, I have had this happen to me before and the solution was to repath the xref and then detach it. I think if Autocad loses the association to the file, it gets confused and doesn't know what to do, so resetting the file path has worked for me.