Uncertainty is ubiquitous in complex dynamical systems. System uncertainties can lead to severe degradation of the performance of advanced optimization-based estimation and control methods, which are crucial for supporting safety and reliability of the system, reducing variability in the system operation, and enhancing the system efficiency. In this talk, we will present a framework for integrated optimal control and learning of stochastic systems based on the notion of stochastic dynamic programming. Various techniques for probabilistic uncertainty propagation and chance constraint approximation are then discussed to obtain tractable formulations for optimization-based estimation and control of stochastic systems, which are amenable to online implementations.