Do I have to buy a set for 5.1 surround audio, or can I just buy 5 speakers and one subwoofer separately to potentially save money and link them to my AVR?
You can definitely just use any 5 speakers. But typically you would want at least the left, right, and center speaker to be from the same brand and even line to make the sound signature more cohesive. Good idea to have the surrounds from the same line too, but not necessary. And the sub could be from any decent manufacturer.
Potentially even the centre channel may be a bad idea - these are frequently lacklustre and do not share the same frequency response as the rest of the speakers in the series
Good video: https://youtu.be/GZrdsxrcpBw
Otherwise yes, ideally you want speakers to have a similar tonal profile
You can use a different speaker for ever channel. This is not recommended, but you can do it.
They do not have to match, just be compatible with the AVR as far as I’m aware. I just recently bought a 5.1.4 setup and could swap out any speaker if I wanted to