We study the dynamics of nuclear many-body system undergoing along macroscopic collective path and intrinsic nucleonic excitations. The collective motion is treated classically within the cranking approach while the chaotic intrinsic dynamics is described quantum mechanically with the help of the random matrix theory. We investigate different regimes of the quantum mechanical energy diffusion of the intrinsic nucleonic subsystem as a function of the energy and time correlations of the transition matrix elements. The relation between the properties of the intrinsic energy diffusion and the corresponding time properties of collective dissipation is also discussed.