In this paper, we intend to lift the veil on masonry construction process in Japan, through the analysis of the early 17th century folding screen called Chikujō-zu byōbu. The great pictorial richness of the screen allows us to reveal many aspects and stages of the masonry construction process of a castle in early modern Japan (1573-1867): stone transportation methods, folk traditions related to stone hauling, implementation of stone materials (especially large ones), and social hierarchy on the construction site. Their analysis and comparison with other paintings and actual remains of early modern Japanese castles will reveal one of the building culture in Japan. Indeed, this paper shows the influence of the social context on the building process and over all on the building result itself. Thus, the great Japanese castles known today as major heritage sites must be understood as the materialization of the early modern social context.