In recent years, driverless vehicle technology receives more attention because of its excellent performance on safety and efficiency. On the other hand, driverless vehicle calls for high-precision environmental perception and expert-like control strategies, which needs both lots of costly sensors and complex algorithms, and makes it difficult to achieve. Machine learning provides a new theoretical basis to solve this problem with big data, while most of data has not been calibrated yet. To solve these problems partly, a machine learning model based on a temporal neural network is described in this paper to achieve “end-to-end” self-driving from uncalibrated monocular images to control signals. The proposed approach is designed for adaptive cruise control situation. The approach is implemented in a simulation platform which has the control signal data from “expert.” According to the experiment in simulation platform, it shows that the proposed approach achieves “end-to-end” self-driving and has good performance on the prediction of desired acceleration.