Information is Physical?

I've heard a few times recently the phrase "Information only exists in a physical state". It come from the quantum computing world where they claim quantum changes the game when it comes to representing information.

As one who has spent his career studying theoretical information that has never and never will exist in a physical state, how can we reckon with such a statement? For starters let's consider the set of all primes--how does that infinite set exist in our finite world? 

Information is physical but not directly, but rather as its description. We can discuss a computational process or more generally a mathematical model that captures the set of all primes and we can and have store that description physically.

Let's consider a single prime, the recently discovered Mersenne prime \(2^{136279841}-1\). Note how we must describe the number in a very compressed format, certainly not as a collection of \(2^{136279841}-1\) ping pong balls or even \(2^{136279841}-1\) atoms, far more than the roughly \(2^{365}\) atoms in the observable universe.

In a similar fashion, a large language model stores information through its weights--not a direct encoding of the sentences it can generate. 

Now let's think of quantum computing. The quantum algorithm is always classically defined. All the information in quantum states has a classical description. An entangled quantum state may require an exponentially large explicit description, but the algorithm generating it provides a short classical physical description. So if we allow information to only physically represented by its description then it's hard to argue that quantum is somehow special. There are differences to how quantum works but when we try to simplify the message, it can confuse people into thinking quantum is more powerful than it really is.