Could you put the library itself into a separate tar rather than patch?