Data Engineer (Financial Data Cleaning and Model Fine-Tuning Direction)

<p>Duties include:</p><ul><li><p>Responsible for processing a large amount of financial raw data in a server environment, using Python for table merge, join, etc., to ensure efficient data integration</p></li><li><p>Perform data cleaning, including handling missing values, expanding json fields, identifying and correcting duplicates and outliers, and improving data quality</p></li><li><p>According to the semantic needs of the model prompt, build and maintain a data dictionary, and implement prompt-level data processing and null value filtering</p></li><li><p>Based on prompt, call open-source large model API (such as LLM), generate high-quality data, and combine rejection sampling and other methods for secondary processing to ensure that the data can be used for downstream model fine-tuning</p></li></ul><p></p><p>Job Requirements</p><ul><li><p>Bachelor's degree or above, computer science or related fields are preferred</p></li><li><p>Proficient in Python and its data processing related libraries (such as pandas, numpy), with large-scale data processing experience</p></li><li><p>Have data cleaning, feature engineering, ETL, etc. related experience, and be able to independently complete the whole data preprocessing process</p></li><li><p>Familiar with json, data dictionary, prompt engineering, etc., data structures and related processing methods</p></li><li><p>Applicants with experience in calling large model APIs (such as OpenAI, LLaMA, etc.) and generating and filtering data are preferred.</p></li><li><p>Have good business understanding, communication and team collaboration skills, and be able to withstand a certain amount of work pressure</p></li></ul>