Why This Job Matters

Your role as a Big Data Engineer in the Cyber Platform team is integral to the build and configuration of our threat hunting and intelligence solutions, used to maintain and protect BTs customers and core network. These platforms are actively used to detect and stop potential threats to our customers and BT as an organisation.  

Join our high performing, efficient and adaptable software team to be at the forefront of Big Data as we look to build the next generation big data platform to identify cyber threats and provide our customers the data they require to protect BT.  

‍

What you’ll be doing

You will be responsible for managing and maintaining our Cloudera-based big data infrastructure along with developing data consumption and processing pipelines.

You’ll work closely with cross-functional teams to ensure the availability, security, and performance of our data platform.  

The Big Data Developer is responsible for designing, developing, and maintaining large-scale data processing systems.  This role involves working with big data technologies to manage and analyse vast amounts of data, ensuring efficient data storage, retrieval, and processing.  

The Big Data Developer will collaborate with data scientists, analysts, and other IT professionals to build robust data solutions that support business objectives.

‍

Skills Needed

Data Pipeline Development:

• Design, develop, and implement scalable and efficient data pipelines.

• Integrate data from various sources, including databases, APIs, and streaming platforms.

• Ensure accurate and efficient data processing and transformation.

Big Data Platform Management:

• Manage and optimize big data platforms (e.g., Hadoop, Spark).

• Monitor and troubleshoot performance issues within big data environments.

• Implement data storage solutions.

Design:

• Create technical components based on evolving business requirements.

Communication:

• Ability to convey complex details clearly to both technical and non-technical audiences.

Focus on Quality:

• Support agile delivery processes, collaborating with QA and other teams.

‍

Experience Expected

Mandatory:

• In-depth understanding of open-source big data tools: o Apache Hadoop: Distributed storage and processing framework for large-scale data taskso Apache Spark: In-memory compute for ETL, machine learning, and data science workloads. o Apache Kafka: Scalable, fault-tolerant messaging system

‍• Java Development: o Write and maintain Java applications for data pipelines.o Troubleshoot Java-related issues within the ACP Platform.

• DevOps: o Write Puppet modules for deployment management.o Familiarity with GIT and CI/CD pipelines.

• Operating Systems: o Proficiency in Linux and shell scripting.

Preferred:

• AVRO Serialization

• Apache NiFi

• Knowledge of Cloudera’s open-source tools and ecosystem.

• Familiarity with the Elastic (ELK) stack: o Collaborate with data engineering teams to integrate Cloudera with ELK.o Manage Elasticsearch, Logstash, and Kibana for log analysis, monitoring, and visualization.

‍