Job Title: Senior Data Engineer
Location: Atlanta, GA
Duration: 8 Months
Bill Rate: $73/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description :
· Design production-grade PySpark batch/streaming pipelines; read S3 Parquet/Delta datasets, implement complex transformations and high-volume deduplication.
· Tune Spark using partitioning, caching, broadcast joins, shuffle/skew optimization and execution-plan analysis; balance speed, reliability and cloud cost.
· Engineer Snowflake warehouses, schemas and models; implement SnowSQL, Snowpipe, Snowpipe Streaming, Streams, Tasks, Dynamic Tables and Materialized Views.
· Create Snowflake-managed or externally managed Apache Iceberg tables for interoperable lakehouse solutions; implement incremental ingestion and CDC patterns.
· Optimize Snowflake with query profiling, clustering, micro-partition pruning, warehouse sizing, caching and materialization strategies.
· Build Snowpark Python DataFrame transformations, UDFs and stored procedures; evaluate Snowpark Container Services for advanced processing.
· Enable Snowflake Cortex Analyst (semantic models/views, natural-language-to-SQL), Cortex Search (semantic retrieval/RAG) and Cortex AI Functions (summarization, classification, extraction).
· Explore Cortex Agents, Snowflake Intelligence and Snowflake Copilot; integrate LLM workflows, embeddings, APIs, prompt evaluation and governed AI access as appropriate.
· Build AWS-native solutions using S3, Glue, EMR, Lambda, Step Functions and Redshift; manage IAM, CloudWatch, orchestration and operational monitoring.
· Modernize DataStage/Informatica ETL; ensure data quality, validation, reconciliation, lineage, security, audit readiness and complete technical/runbook documentation.
· Collaborate with architects, analysts, TPMs and business teams to deliver maintainable, secure and cost-effective data products.
Required Qualifications
· 6+ years of enterprise data engineering; strong hands-on PySpark, Spark SQL, Python and advanced SQL (CTEs, window functions, joins, subqueries, tuning).
· Production experience reading large S3 Parquet/Delta datasets, implementing deduplication, and optimizing high-volume PySpark processing.
· Deep Snowflake experience: data modeling, SnowSQL, Snowpipe, Streams, Tasks, RBAC, Iceberg Tables, Dynamic Tables, Materialized Views and performance/cost tuning.
· Working exposure to Snowflake Cortex AI (Cortex Analyst, Search or AI Functions), semantic analytics and Snowpark Python; understanding of GenAI, LLMs, embeddings and RAG.
· Hands-on AWS S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch and Redshift; Airflow, Step Functions or comparable orchestration.
· Knowledge of data warehousing, ETL/ELT, data lakes and lakehouse patterns; strong communication, problem-solving and stakeholder collaboration.
Preferred Qualifications
· Hands-on Cortex Analyst semantic models/views, Cortex Search/RAG, Cortex Agents, Snowflake Intelligence, Copilot, Snowpark Container Services, Streamlit in Snowflake or Native Apps.
· IBM DataStage/Informatica modernization; Git, Jenkins, GitHub Actions, Terraform and CI/CD; Great Expectations/dbt tests; Kafka/Kinesis; Databricks; Control-M and shell scripting.
· Financial services, regulated mortgage or Fannie Mae/Freddie Mac/GSE programs; Sn