In reality, Charlie was unfaithful. He took his heart to an addiction to deal with what he should have been dealing with directly with Leigh. He had a double life. He would try to please her on the outside, but inside take his deepest needs and desires to his fantasy life. He was split, and this was keeping the couple from resolving their problems.
