One key chronicle imputes immorality, claiming Agnes had affairs with Heraclius, later the Patriarch of Jerusalem, and Aimery de Lusignan, but modern historians question whether she could have had a scandalous reputation at the time of the divorce.
The biggest problem with this explanation is it’s impossible on chronological grounds: both Aimery de Lusignan and Héraclius were definitely or probably still in France when the marriage between Agnès and Amaury was annulled in 1163.

