You might look for locally milled quiviut -- it's got to be hand combed to remove the guard hair, and maybe they stuff you've seen hasn't bothered? The quiviut I've got is like petting butter, it's so soft.

Merino is a breed of sheep that has extremely fine wool, even as adults -- soft enough to wear next to your skin, if the sheep are actually withing the breed standard.

Cashmere is more fragile that wool, since it doesn't have the 'bounce' of wool -- no other animal fiber has 'memory' like wool, so it's generally a trade-off between drape and shape in other animal fibers. You might like a wool-silk blend, now that I think about it.