Limitations of Trace Invariants to the Inverse Spectral Problem