I think that I found a better fix by looking at what GCC does with the -mcpu argument, see aarch64_rewrite_mcpu and the aarch64-cores.def file. I'll look into making a new patch.