Large model pre-training engineer

Job Responsibilities

• Responsible for the design, implementation, and optimization of a pre-training system for ultra-large language models (LLM) and multimodal models.

• Design and maintain a distributed training architecture, including data parallelism, model parallelism, pipeline parallelism, etc. training strategies.

• Construct and optimize high-performance training workflows, covering data preparation, data loading, Checkpoint, error recovery mechanism, and training monitoring.

• Collaborate with MLOps / Infrastructure teams to optimize GPU cluster resource utilization and training stability.

• Perform training performance analysis and tuning, reduce communication overhead, improve throughput and model convergence efficiency.

• Support model version management, experimental reproducibility, and training process standardization.

• Track and evaluate the latest large-scale model training techniques (such as ZeRO, FSDP, MoE, FlashAttention, etc.), and implement them in actual systems.