In this study, a vision-based localization and mapping system for navigating an autonomous agricultural vehicle was developed by using our customized ORB-SLAM2. In order to solve a low measurement accuracy in agricultural field scenes, a blank area was applied as a non-processing area for an input image to control the region of interest of the ORB-SLAM2. The performance of our system was tested using our targeted three types of agricultural fields, namely a farm road, a tilled field, and a tea field. In the experiments, our system could more accurate and robustly measure the shape and scale of the trajectories of targeted routes in the fields, while a LiDAR based SLAM of LOAM SLAM could not measure the trajectories completely in the vast flat fields. In localization, the system could consistently estimate the self-positions along the targeted routes. Furthermore, as an application of our system for an agricultural working scene, the system was applied for a Japanese plum harvesting scene in a plum orchard where GNSS signal could not be received stably. In the experiments, the trajectory of the working route was consistently measured to build a navigation map and the plum trees along the route were detected by using a deep learning model of YOLOv3 to draw them on the map. Through all of the experiments, the results demonstrated the high potential of our system to navigate an autonomous robot vehicle without using GNSS in the targeted agricultural field types.