Remove link button on subpanels

Hello all,

Is there an easy way to hide the "Remove link" button on subpanels to specific users?

I tried with roles but it doesn't work.

Thanks