In essence, the SafeVarargs annotation constitutes a promise by the author of a method that it is typesafe. In exchange for this promise, the compiler agrees not to warn the users of the method that calls may be unsafe. It is critical that you do not annotate a method with @SafeVarargs unless it actually is safe.
Sounds risky to leave this to human error. Part of the whole benefit of a type system is not having to worry about human error