Software engineer – big data architect

Software Engineer Big Data Architect Job Description Illuminate Technologies team is hiring software engineers who can produce efficient & functional web-based software services to solve complex analytics problems.

We are developing data-driven solutions for petabyte scale, high velocity, big data analytics in a modern DevOps environment.

We are looking for an experienced big data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up.

If you are an exceptional developer and who loves to push the boundaries to solve complex customer problems using innovative solutions, then we would like to talk to you.

Responsibilities
· Design and implement big data pipelines, real-time services and apply great problem-solving skills, technical ability and confidence to run spikes and resolve architectural choices
· Implement big data solutions throughout the DevOps cycle, developing & testing modular, reusable, efficient and scalable code
· Organize, transform and optimize data architecture for query of large, complex, structured and unstructured data sets
· Develop data pipelines for optimal extraction, transformation, and loading of data from a wide variety of data sources using modern, big data technologies
· Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics
· Build processes supporting data transformation, data structures, metadata, dependency and workload management Core Skill Set
· 3-5 years of relevant experience designing and building data-driven solutions for big data projects
· Advanced working SQL knowledge and experience working with SQL on object stores, relational databases, query authoring as well as working familiarity with a variety of databases
· Experience with NoSQL/unstructured data and object stores including KV Stores, Search Engine technologies such as Elasticsearch, Object storage technologies such as S3
· Build processes supporting data transformation, data structures, metadata, dependency and workload management
· Working knowledge of ETL, message queuing, stream processing, and data lakes
· Experience with big data frameworks and environments including Hadoop, Spark and Kafka in a scale-out environment
· Experience with stream-processing systems such as Kafka, Spark Structured Streaming, Flink
· Strong experience with Python, Java, Javascript, and/or related modern development and scripting languages
· Development experience using public cloud services, preferably AWS
· Experience working within a Linux computing environment and use of command line tools including knowledge of shell/Python scripting for automating common tasks
· Ability to work in a team in an DevOps setting, familiarity with JIRA, Git based development and clear understanding Agile, CI/CD and related technologies (Docker, K8S) Desirable Experience
· Some knowledge of statistics and Machine Learning technologies on cloud-based platform using Hadoop, Spark, H2O or Tensorflow
· Interest integrating/deploying AI/ML models for real-time predictions Education
· Bachelor’s degree in computer science, mathematics, physics, statistics or related technical degree
· Masters (optional) Security
· This position may require access to protected US Government information.

The Applicant must be willing and able to process for and obtain a US Government security clearance.

· Applicants must be a US Person and currently authorized to work in the U.S.

on a full-time basis.

Employment-based visa sponsorship, including H-1B sponsorship or F1 status, is not available for this position.

You must not now or in the future require sponsorship for employment visa status.

from Berkeley Job Site https://ift.tt/2rPhZRV
via IFTTT