ETL數據工程師 ETL Data Engineer

工作內容： ETL 設計與開發： 負責企業數據倉庫（DWS）的 ETL 流程設計、開發與優化，確保數據從源系統到目標系統的高效且安全傳輸。 使用 ETL 工具（如 Informatica、DataStage、Talend、Apache Nifi、SSIS 等）建構數據管道，執行數據清洗、轉換與加載。 撰寫高效能 SQL 語句，設計數據庫存儲過程，處理大規模數據集並優化查詢效率。 數據整合與處理： 設計與開發數據整合流程，處理來自關聯式數據庫（如 MySQL、Oracle）、大數據平台（如 Hadoop、Spark）及 API 等多來源數據。 優化數據質量（如缺失值、重複值處理）與數據一致性，確保提供高品質數據供業務使用。 性能優化與監控： 深入優化 SQL 查詢效能，包括索引優化、分區策略、執行計劃分析等。 持續提升 ETL 作業效能，縮短執行時間並提高數據處理效率。 監控 ETL 作業執行情況，及時診斷並解決作業失敗、性能瓶頸等問題。 數據安全法： 確保 ETL 流程符合企業數據安全標準與隱私保護規範（如 GDPR、ISO27001）。 執行數據加密、脫敏等安全策略，確保敏感數據的保護。 文件撰寫與團隊協作： 撰寫與維護 ETL 設計文件、數據流程圖，確保專案具可追溯性與可維護性。 與數據分析師、BI 開發人員、數據科學家等團隊密切合作，支援數據產品開發。 任職要求： 學歷與經驗： 大學以上學歷，計算機科學、軟體工程、資訊管理等相關科系。 具備 3 年以上 ETL 開發經驗，曾參與數據倉庫或大數據相關專案者優先考慮。 Job Responsibilities: ETL Design and Development: Responsible for the design, development, and optimization of ETL processes for the enterprise data warehouse (DWS), ensuring efficient and secure data transfer from source systems to target systems. Build data pipelines using ETL tools (such as Informatica, DataStage, Talend, Apache Nifi, SSIS, etc.), performing data cleansing, transformation, and loading. Write high-performance SQL queries, design database stored procedures, handle large-scale data sets, and optimize query efficiency. Data Integration and Processing: Design and develop data integration processes to handle multi-source data from relational databases (such as MySQL, Oracle), big data platforms (such as Hadoop, Spark), and APIs. Optimize data quality (such as handling missing values, duplicates) and ensure data consistency, ensuring high-quality data for business use. Performance Optimization and Monitoring: Deeply optimize SQL query performance, including index optimization, partitioning strategies, and execution plan analysis. Continuously improve ETL job performance, reduce runtime, and enhance data processing efficiency. Monitor ETL job execution, diagnose and resolve job failures, performance bottlenecks, and other issues in a timely manner. Data Security and Compliance: Ensure that ETL processes comply with the company's data security standards and privacy protection regulations (such as GDPR, ISO27001). Implement data encryption, anonymization, and other security strategies to ensure the protection of sensitive data. Documentation and Team Collaboration: Write and maintain ETL design documents, data flow diagrams, ensuring traceability and maintainability of the project. Collaborate closely with data analysts, BI developers, data scientists, and other teams to support data product development. Qualifications: Education and Experience: Bachelor’s degree or higher in Computer Science, Software Engineering, Information Management, or related fields. 3+ years of ETL development experience, with preference given to those with experience in data warehouse or big data projects.