Deep learning-based autoregressive models have garnered a lot of popularity in recent times, primarily owing to their success in several fields of the physical sciences, e.g., fluid dynamics, weather and climate dynamics, ocean dynamics, etc. Despite their success, there are several fundamental challenges in developing such models, especially for high-dimensional multi-scale nonlinear dynamical systems. One of them being the lack of stability of these models especially when integrated for long time scales. In this talk, we would derive a fundamental bias in deep neural networks that leads to instability of autoregressive models, discuss principled structures that can be enforced to mitigate such bias, analyze them in through the lenses of linear stability analysis and random matrix theory, and finally implement them for realistic high-resolution climate modeling at scale.