Indeed, “light” is not all that important in this game. What’s important is that there exists some unique preferred velocity through spacetime. It just so happens that light moves at that speed when it’s traveling through empty space—but the existence of a speed limit is what matters, not that light is able to go that fast.