If you remove the binaries, then people can't build modules, which is a big bummer. So if there's some workaround, I suppose that'd be ideal.
Alternatively, you can remove the binaries, but then make another package all together that ships the entire kernel source alongside Modules.symvers and .config, like the headers package, so that people can use that for building modules.
Either way is fine. The current work around is something nasty involving untarring the source into the headers folder, to add the missing sources, and then running `make scripts` on the target, to rebuild those binaries. That works, but it's very burdensome and a fairly ugly hack.