42dot actively uses machine learning technologies to develop autonomous driving systems. Machine learning development requires high-level algorithms, large amounts of data, complex computing operations, and support of software and hardware to perform them. On top of this, 42dot tailors various technologies to develop and operate a machine learning&data Platform. Even now, 42dot's machine learning developers are using the above-mentioned platform for effective development.
Data Pipeline
Data is very important in machine learning development. A large amount of data is required to improve the performance of a machine learning model. It is also important to curate a dataset that will boost its performance. This data preparation process is a complex and time-consuming task. 42dot hence develops and operates an automated data pipeline, which smoothly processes all data-related task to support in-house machine learning developers.
Data pipeline works in the following processes:
1) Data collected from our autonomous vehicles is transmitted to the cloud and data center through the edge cluster of each site.
2) Data sent to the cloud passes through transformation and indexing for retrieval before being stored in a database.
3) The data sent to the data center is stored in the database after passing the weakness detection pipeline for the data curation.
4) The saved data is either used in the development process for the debugging and re-simulator. Also, a data console accelerates data curation processes.
5) Additional privacy protection measures such as personal information encryption and de-identification takes place.
6) The data after labeling is stored in the shared storage connected to the GPU cluster. The machine learning developers can work by accessing the annotated datasets using the Kubeflow-based ML workspace.
ML Workspace
Training of machine learning models requires a lot of computation that depends on many GPUs. The machine learning developers simultaneously request multiple GPU resources; therefore, the efficient utilization of the GPUs is important. In fact, we could become more productive as our ML workspace allows faster model training despite the limited resources.
It is necessary to prepare a development environment, for instance, installing various libraries. For such requirements, 42dot provides an ML workspace based on Kubeflow.
In the case of a GPU cluster, we configure a Kubernetes cluster with multiple GPU servers and connect it to a shared storage that allows direct access using a high-speed network. We then build a Kubeflow-based system in the cluster to support easy access of the resources to the in-house developers. Precisely, the developers can request virtual server instances with desired specifications, such as CPU, memory, GPU, etc., and return the resources after completing their tasks.
Such dedicated virtual server instances can be accessed through a web browser, where we provide the most popular features for machine learning development.
ML Pipeline
To make a product-grade machine learning model, we should repeat the following processes: training, evaluation, model transformation, integration testing, and deployment. To speed up the development processes and focus on the main tasks, we automate repetitive tasks as much as possible so that developers can offload routine jobs.
First, we leverage MLFlow to support tracking interim training of histories and model registration. All training processes are automatically recorded in MLFlow, installed in the cloud, and the results of training are stored in the model registry. Authorized users can search for the best model and its configurations including hyper-parameters to reproduce the model. Also, the user can monitor the training process and result throughout the API.
Second, we build the ML pipelines. When training outcomes are registered in the Model Registry, we automatically perform the following pipelines: evaluation, model transformation, and testing. Dozens of pipelines are in operation using Kubeflow and are performed automatically more than hundreds of times a day, accelerating the development of autonomous driving systems.
・ ・ ・
In this post, we have introduced the ML data platform used by the 42dot machine learning development team. The ML data platform will continue its efforts to accelerate the journey of bringing autonomous driving systems closer to our everyday lives.
Eddie Yang | Data & ML (Tech Lead)
I am in charge of developing Machine Learning & Data platform for autonomous vehicles.