There is a compelling need for the regression capability of mapping the initial field and applied conditions to the evolved field, e.g., given current flow field and fluid properties predicting next-step flow field. Such a capability can provide a maximum to full substitute of a physics-based model, enabling fast simulation of various field evolvements. We propose a conceptually simple, lightweight, but powerful multi-input convolutional network (ConvNet), yNet, that merges multi-input signals by manipulating high-level encodings of field/image input. yNet can significantly reduce the model size compared with its ConvNet counterpart (e.g., to only one-tenth for main architecture of 38-layer depth) and is as much as six orders of magnitude faster than a physics-based model. yNet is applied for data-driven modeling of fluid dynamics, porosity evolution in sintering, stress field development, and grain growth. It consistently shows great extrapolative prediction beyond training datasets in terms of temporal ranges, spatial domains, and geometrical shapes.