Ok, I think I found why this is happening. It's stupid, really really stupid.
I have a firefox youtube extension and one option that I ticked when I installed it was "set max quality automatically" and I didn't notice it until now. I'd say this is pretty normal behaviour when watching 4k videos on youtube, isn't it? (clocks don't max out for the whole duration of the video, just spike around)
As for the clocks spiking when I'm not even using firefox they do for about 2 seconds, which again I guess it's normal?
The main reason why I was worried is because when I do play some older games clock/memory don't max out, so yeah, that's about it.
I apologize for wasting everyone's time.