Hadoop Jobs

We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, private repositories, internal servers, or ZIP archives. Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack Approximate LOC/module cou...

I need a cloud-native ETL pipeline built end-to-end on AWS, coded in PySpark and designed for production reliability. The pipeline will ingest data from three sources—databases, APIs, and file systems—then standardise and load it into an analytics-ready destination. Source files arrive in a mix of CSV, JSON, and Parquet, so the job must include automatic format detection, schema inference, and efficient column-wise writes. Beyond raw transformation, I want solid engineering practices: parameter-driven jobs, modular Spark code, unit tests, logging, alerting, and retry logic. Leveraging AWS native services such as Glue, EMR, Lambda, and S3 is expected, but I’m open to other AWS components if they shorten development time or lower cost. Candidates must have expertise in da...

I’m setting up an intensive, expert-level course for a team that already designs and operates complex AWS workloads. The objective is to deepen their Data Engineering skills on AWS, so we need a trainer who can move straight past fundamentals and into real-world architecture, scaling strategies, and cost-efficient design. Core focus: Data Engineering with AWS, specifically • Data Storage Solutions • Data Processing & ETL • Big Data Analytics Your role is to craft and deliver a hands-on program that blends concise theory, architectural white-boarding, and live labs. Glue, EMR, Redshift, Lake Formation, Kinesis, Lambda, Step Functions, and related services should feature prominently, but I’m open to your proven toolset and workflows. Deliverables 1...

Automated Terraform infrastructure deployments, achieving a 97% security score in CSPM, ensuring cloud infrastructure met enterprise-grade security and compliance standards Hardened cloud infrastructure to CIS Benchmark v3/v5 and PCI-DSS v4 standards, directly contributing to enterprise compliance certification and reducing security vulnerabilities across all environments. Built Ansible playbooks automating software installation and configuration across multiple servers simultaneously, cutting provisioning time by 20% and eliminating manual configuration errors. Architected and managed production AWS infrastructure across 15+ services, including EMR, WAF, GuardDuty, and Global Accelerator, supporting a highly available and secure multi-environment setup. Managed hybrid deployments spanning...

Hiring: GCP DevOps Engineer (2 Positions) Experience: 5+ Years Location: Remote Contract Duration: 3 Months Salary: ₹40,000 – ₹50,000/month --- Key Responsibilities: Infrastructure & Platform Engineering: - Design and manage cloud infrastructure (GCP preferred / AWS / Azure) using Terraform - Build and manage Kubernetes clusters (GKE/EKS/AKS) with multi-tenancy - Implement auto-scaling (HPA/VPA) and optimize resource usage - Ensure high availability, disaster recovery, and backup strategies CI/CD & Release Engineering: - Design and maintain CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins, ArgoCD) - Implement GitOps workflows for automated deployments - Enable zero-downtime deployments (blue-green, canary) - Automate database migrations, rollbacks, and environment s...

Hiring: ML Engineer – Test & Learn Platform Experience: 3+ Years Location: Remote (1–2 visits to Bangalore required) Salary: ₹40,000 – ₹50,000/month --- Role Overview: We are looking for an ML Engineer to build and scale experimentation and causal inference systems. You will work on statistical engines, APIs, and cloud-based pipelines to enable data-driven decision-making. --- Key Responsibilities: - Develop ML/statistical models (DID, Synthetic Control, A/B Testing) in Python - Build and integrate FastAPI-based services - Design large-scale data pipelines using PySpark, Delta Lake, and Azure Data Lake - Optimize Spark jobs (memory, partitioning, performance tuning) - Work with Databricks for job orchestration and data workflows - Containerize and deploy applica...

Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables * Com...

Hope you are doing well. We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, private repositories, internal servers, or ZIP archives. Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack A...

# Lung Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables...

I’m looking for an experienced consultant who can take the lead in architecting and standing up a full-scale data lakehouse, with the immediate goal of seamless data integration across my current and future sources. What I need from you: • A clear, vendor-agnostic architecture blueprint that balances performance, cost, and governance. • Hands-on implementation of the core storage layer (Delta/Parquet or comparable), compute engine (Spark or equivalent), and ingestion pipelines. • Robust security and data-quality controls baked in from day one, including role-based access, lineage, and monitoring. • Documentation and a concise knowledge-transfer session so my internal team can extend and maintain the platform confidently. Acceptance criteria: 1. A reproducible i...

# Lung Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables...

This is a focused 1-day task. The LLM pipeline already exists in Python — I need it containerized with Docker and running live on AWS by end of day. No research, no exploration — come ready to execute. Deliverables: 1. Dockerfile & Docker Compose Write a production-grade Dockerfile for the LLM inference pipeline. Multi-stage if needed, minimal image size, env vars for config and secrets. 2. AWS deployment Deploy the container to AWS (ECS Fargate preferred, or EC2 if GPU required). Expose a working endpoint. Basic IAM role and security group setup. 3. Smoke test + handoff Confirm the endpoint is live and responding. Provide a short Bash script or README so I can redeploy independently. No undocumented magic. YOU'RE A FIT IF: ✔️ You've deployed a containerized ...

Dear Data Engineers, We are currently looking for mutliple Data Engineers for one of the leading tech companies in Europe. The position is 100% remote and long-term. QUICK FACTS: - Freelancing/Contracting - Start: ASAP - 100% remote - Duration: long-term 12+ months - Capacity: full-time, part-time also possible - Language: English (no german needed!!) - German business hours REQUIREMENTS: - Experience in data engineering, ideally in Microsoft Azure environments. - Knowledge of Databricks, Apache Spark, and Medallion Architecture. - Experience with Delta Lake / Delta Tables, including concepts such as Row-Level Security. - Ability to work with Spark DataFrames, Spark SQL, and UDFs. - Good Python skills for data processing, transformation, and pipeline development. Are you interested? I...

Hope you are doing well. We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, private repositories, internal servers, or ZIP archives. Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack A...

We are looking for a freelance Subject Matter Expert (SME) who has successfully cleared the Microsoft Certified: Azure Databricks Data Engineer Associate (beta) certification to support the development of high-quality certification preparation content. Project Scope The selected SME will be responsible for: Creating practice questions aligned with the certification objectives and exam blueprint Designing scenario-based and real-world exam-style questions Providing detailed explanations and answer rationales Ensuring technical accuracy and relevance to the latest certification updates Reviewing and refining content based on feedback, if required Requirements Must have successfully cleared the Microsoft Certified: Azure Databricks Data Engineer Associate (Beta) certification Strong hands-...