“Many AI researchers today claim that their systems are cognitively inspired (in particular inspired by the popular System 1/System 2 distinction introduced by Daniel Kahneman in his dual-process theory) just because their decision-making mechanisms couple both fast routines and slow decision-making strategies.

This is a clear example (one of the many in the field) of the misconceptions that have been raised by the shallow ascription of labels coming from the cognitive vocabulary to the behavior and/or design of such systems.

Unfortunately, it is not sufficient to just implement “fast” and “slow” mechanisms in an artificial system to claim any kind of cognitive inspiration or of cognitive plausibility.
To make one of these claims, in fact, one should build and integrate algorithms in a way that is much more constrained with respect to such a generic and shallow description of how an intelligent system (natural or artificial) works (note: the book Daniel Kahneman - Thinking, Fast and Slow (2011) was written for a popular audience and therefore contains obvious oversimplifications of the dual-process theory of reasoning. Unfortunately, many people in AI have considered the book as a scientific publication ignoring the actual scientific papers laying down the theory).

For example, one should consider "how” such fast or slow mechanisms are built, how they interact between them (both within the System 1/System 2 components and between them), how they evolve over time (e.g. System 2 mechanisms can be “automatized” and become System 1 routines) etc.

In Cognitive Design for Artificial Minds, the distinction between these “shallow” and “constrained” systems is made clear by introducing the “functional” and “structural” design approaches and by exploring the different explanatory roles that such design perspectives put in place.”