Construction workers are commonly subjected to ergonomic risks. Accurate ergonomic assessment is needed to reduce ergonomic risks. However, the diverse and dynamic nature of construction sites makes it difficult to collect workers posture data for ergonomic assessment without intrusiveness. Therefore, this paper proposed a joint-level vision-based ergonomic assessment tool for construction workers (JVEC) to provide automatic and detailed ergonomic assessments of construction workers based on construction videos. JVEC extracts construction workers’ skeleton data from videos with advanced deep learning methods, then Rapid Entire Body Assessment (REBA) is used to conduct the joint-level ergonomic assessment. This approach was demonstrated and tested with a laboratory experiment and an on-site experiment, which indicated the accuracy of the ergonomic risk scores (70%–96%) and its feasibility for use on construction sites. This research contributes to an accurate and nonintrusive ergonomic assessment method for construction workers. In addition, this research for the first time introduces two-dimensional (2D) video–based three-dimensional (3D) pose estimation algorithms to the construction industry, which may benefit research on construction health, safety, and productivity by providing long-term and accurate behavior data.