In the milling process, chatter, which results in poor surface quality, dimensional errors, and reduced cutter and machine life, is one of the main limitations on performance. Consequently, a reliable, real-time detection method is desired to recognize chatter while it is developing. This study develops a novel method of online chatter identification for milling processes. In this method, optimized variational mode decomposition (OVMD) is used to decompose cutting force measurements, and the sub-components containing chatter information are extracted using a simulated annealing (SA) algorithm. The approximate entropy and the sample entropy are used to detect the onset of chatter. To evaluate the effectiveness of the proposed method, milling operations were performed and force measurements were collected for five types of operating conditions. The results show that the proposed method is suitable for detecting both continuous and intermittent chatter. Rather than establishing an absolute threshold for chatter detection, the onset of chatter is identified from relative changes in the entropy with time that occur under the various cutting conditions. The proposed method is shown to have greater sensitivity and stability than empirical mode decomposition (EMD).