Automatic recognition of construction workers’ activities contributes to improving productivity and reducing the potential risk of injury. Kinematics sensors have been proved feasible and efficient to recognize construction activities. However, most of the sensors need to be tightly tied to workers’ bodies, which might result in uncomfortableness and workers’ reluctance to wear the sensors. To solve the problem, this paper proposes a less physically intrusive construction activities recognition method with a single in-pocket smartphone. The smartphone was placed in the pocket in a natural and non-fixed manner, with its built-in accelerometer and gyroscope collecting motion data. Machine learning-based classifiers were trained to recognize construction activities. An experiment simulating rebar activities was designed to verify the effectiveness of the proposed method. The experiment results showed that the proposed method could identify rebar activities (with an accuracy over 94%) in a non-intrusive manner.