Ok, doesn’t change the fact that clocks weren’t available to the layman until a couple centuries ago.
We did not evolve to be sensitive to the passage of time.
It’s a thing whether you like it or not. Your preference for punctuality is a learned behavior, not something that everyone else has.
Ok, then it’s a skill that not everyone else values the same way you do. It doesn’t make you better than anyone. Frankly, it appears to stress you out, and probably lots of people around you too.