The development of autonomous vehicle systems has progressed rapidly in recent years. One challenge that persists is the capability of the autonomous system to respond to human drivers. Human behavior is an integral part of driving; thus, driver behavior determines changing lanes and speed adjustments. However, human behavior is unpredictable and immeasurable. Some traffic accidents are caused due to the erratic behavior of the driver. Although, traffic laws, such as in Indonesia, regulate the use of lanes concerning the vehicle’s speed. The drivers’ behavior in the lane is more likely to be influenced by the regulation. This paper proposes a novel method of predicting drivers’ behavior by utilizing the concept of fuzzy Hidden Markov Model (fuzzy HMM). HMM has been proven reliable in predicting human behavior by observing measurable states to determine unmeasurable hidden states. The use of fuzzy logic is to mimic the way that humans perceive the speeds of other vehicles. The fuzzy logic determines the relative observed state of other vehicles according to the measured velocity of an ego vehicle and the observed state of observed vehicles. Observation data is obtained by equipping an ego vehicle with an action camera. The observed data, in the form of a video, is then discretized every 2 seconds. The resulting sequence of images is processed to determine several variables: speed and state of the observed vehicles (lane position and speed) and the time instance of the observation. The fuzzy HMM is generated based on observational data. A predictor created using fuzzy HMM equipped with a training and prediction algorithm successfully predicts the behavior of other drivers on the road.