Scientific Networking Research Associate

Caltech is a world-renowned science and engineering institute that marshals some of the world's brightest minds and most innovative tools to address fundamental scientific questions. We thrive on finding and cultivating talented people who are passionate about what they do. Join us and be a part of the diverse Caltech community.

The Compact Muon Solenoid (CMS) is one of four experiments at CERN's Large Hadron Collider (LHC) in Geneva, Switzerland. The CMS collaboration, numbering over 3,000 physicists and engineers, makes use of a worldwide computing grid for the processing and analysis of the data taken by the experiment. The Caltech group is one of the leading US institutes in CMS, with an excellent track record and reputation both in particle physics research and in state-of-the-art research and development for scientific computing and networking. Caltech hosts one of seven US-based "Tier2" High Performance computing facilities in CMS, which are part of this grid. The Caltech HEP group also manages testbed clusters as part of testing new technologies and network/computing R&D projects.

The successful candidate will join the Caltech High Energy Physics group and the worldwide CMS community, and will work on networking R&D projects in association with Caltech HEP, Fermilab, ESNet, Internet2, the CENICregional network, CERN and campus network engineers in state of the art network and distributed system developments to evaluate and implement new networking technologies to ensure fast and reliable data transfers to/from the computing clusters and new data access methods. The successful candidate will participate in state-of-the-art developments, working with other CMS sites and the global network community, aimed at making the experiment's data distribution and analysis more efficient through the use of advanced network monitoring, caching and high throughput data transport tools. The candidate will participate in the ongoing development of a next generation system that interfaces intelligent networks to the data management systems of CMS and the other LHC experiments, in order to meet the experiments' rising needs while also accommodating the network traffic of other programs and the worldwide academic and research community. As part of a small but skilled team, he/she will also assist in maintaining and advancing Caltech HEP's Software Defined Networking (SDN) Testbed computing facility dedicated to network R&D developments.

We are looking for an eager self-motivated person who can quickly learn to deal with both hardware and software cluster issues as well as grid issues as needed, who will enjoy working with a dedicated team of scientists and engineers engaged in frontier physics research, along with cutting edge technologies and systems needed to support this program.

This is anticipated to be a six-month temporary assignment.

• Hardware installation, OS installation, software installation and updates, monitoring, security set up, and troubleshooting.
• Maintaining SDN Kubernetes cluster with storage at Caltech.
• Implement and test networking protocols, automation tools and simulation environments.
• OS installation, software installation and updates, monitoring, security set up, and troubleshooting.
• Participate (usually by videoconference) in biweekly CMS computing meetings where the status of the Caltech CMS Tier2 computing resources are discussed.
• Work with members of the HEP group and keep abreast of emerging computing, networking, storage technologies and evaluate their suitability for the computing clusters.
• Actively participate in the design and implementation of an integrated infrastructure supporting research and development of state of the art network and server infrastructure and techniques.
• Participate in regular technical and strategy meetings in association with these developments.
• Work with the Open Science Grid (OSG) and local campus security teams to implement reasonable security safeguards and respond to any security threats.
• Collaborating with ESNET on deploying SENSE/Rucio Testbed on US-CMS Tier1/Tie2 sites.

• Bachelor of Science degree with related work experience.
• Solid understanding of TCP/IP, OSI layers and networking protocols and networking fundamentals.
• Knowledge and hands on experience in computer networking, including switches and routers, routing and switching protocols, including BGP and OSPF., network interfaces and distributed system and network.
• Analyze network performance, identify bottlenecks, and develop solutions for scalability and resilience which includes working knowledge of tools like iperf3, tcpdump, Wireshark, netcat and perfSONAR
• Knowledge and hands-on experience with open-source projects. Experience in Unix/Linux operating systems and hands-on experience with operating, monitoring, and managing large-scale computing, networking, and storage infrastructures.
• Sound knowledge on GitHub/GitLab.
• Excellent oral and written communication skills including developing, writing, and reviewing technicaldocuments.
• Excellent team player, ability to work independently and as part of a team.
• Ability to work on multiple tasks simultaneously and set priorities.
• Ability to work across institutions and communicate effectively with both operational and research teams
• Some travel in-state and nationally may be required.

• Master's degree with at least 2 years of work experience.Operational experience of production network operations.Ability to learn any network operating system including but not limited to Arista, Dell, Cisco, P4, and Sonic, and to adapt to new network tools, methods and devices as needed.
• Experience with management and automation tools such as Ansible, Proxmox etc.
• Knowledge on virtualization such as network, storage.
• Solid knowledge and hands-on experience on software-defined networking (SDN)
• Knowledge and experience with container networking as Kubernetes and Docker.
• Hands-on experience with network monitoring systems (e.g., Grafana, Prometheus), packet analysis, and performance tuning
• Proficiency in shell scripting and Python.
• Familiarity with R&D networking environments and wide-area performance challenges.

• Resume.