Superfluid ³He, with unconventional spin-triplet p-wave pairing, provides a model system for topological superconductors, which have attracted significant interest through potential applications in topologically protected quantum computing. In topological insulators and quantum Hall systems, the surface/edge states, arising from bulk-surface correspondence and the momentum space topology of the band structure, are robust. Here we demonstrate that in topological superfluids and superconductors the surface Andreev bound states, which depend on the momentum space topology of the emergent order parameter, are fragile with respect to the details of surface scattering. We confine superfluid ³He within a cavity of height D comparable to the Cooper pair diameter ξ₀. We precisely determine the superfluid transition temperature T_c and the suppression of the superfluid energy gap, for different scattering conditions tuned in situ, and compare to the predictions of quasiclassical theory. We discover that surface magnetic scattering leads to unexpectedly large suppression of T_c, corresponding to an increased density of low energy bound states.