Data Pipeline Migration From AWS to GCP

Industries:
Migrated SETI Institute’s data pipelines from AWS to GCP, enhancing stability, performance, and code quality while streamlining research operations.
about SETI Institute

SETI Institute started small, with just one project — NASA’s SETI Program and two employees. Over the years, other research disciplines have been added to the Institute’s portfolio, all unified by their relevance to the search for, and understanding for, life beyond Earth. Today, the Institute has approximately 100 scientists and specialists in administration, education, and outreach.

SETI’s Citizen Scientist Program

The SETI Institute has partnered with Unistellar, a company aiming to revolutionize popular astronomy through the Enhanced Vision Telescope (eVscope). This innovative, mass-market digital telescope is compact enough to fit in a backpack.

The SETI Institute and Unistellar are engaging in a unique citizen science collaboration. With a global network of over 8,000 users, SETI Institute researchers can quickly mobilize teams for observations, leading to significant scientific discoveries and contributions.

Franck Marchis, a senior planetary astronomer at the SETI Institute and chief scientific officer at Unistellar, and Thomas Esposito, research assistant focused on the architecture and evolution of exoplanetary systems, lead the scientific collaboration between the institute and Unistellar.

The following observations have been included:

Asteroids

Successful observation of the Trojan asteroid Eurybates, target of the NASA Lucy mission, as it crossed Europe, deriving for the first time its complex shape.

Exoplanets

An ongoing exoplanet detection program to help confirm exoplanet candidates identified by TESS by observing possible exoplanet transits from Earth

Planetary Defense

DART mission observations on Reunion Island and in Kenya successfully captured the impact, including the formation of clouds and material after impact, a brightening, a plume and later the formation of a faint tail of ejected material.

Comets

A new program to better understand the changing activity of comets as they approached the inner part of the solar system.

Transient Events

A new program to support surveys like ZTF and the upcoming Vera Rubin telescope, and monitor cosmic cataclysms, like super-novae.

Techtinium's Collaboration With The SETI Institute

We were tasked with several critical responsibilities to support this initiative:

  • Pipeline Migration: Migrate all the data pipelines from AWS to GCP.
  • Code Refactoring: Refactor and rewrite code where necessary.
  • Environment Setup: Establish separate environments for testing and production.
  • Pipeline Stability: Enhance pipeline stability.

Techtinium's Collaboration With The SETI Institute

  • Version Control: Implemented Git for efficient version control and improved debugging with the logger library.

  • Version Upgrades: Upgraded the Python server and resolved all dependency conflicts during the update process.

  • Environment Tracking: Utilized pipenv as their virtualenv management tool to track all Python libraries, ensuring the reproducibility of code.

  • Astrometry Software: Upgraded the astrometry software to perform WCS transformations, and stacking, of telescope data.

  • Performance Comparison: Conducted root cause analysis to compare AWS and GCP pipeline performance, finding GCP pipelines to be faster.

  • Cloud Logging: Implemented file logging accessible in GCP, enabling easy searchability of cloud logs.

  • Code Quality: Addressed issues with the existing code, including numerous background processes. Improved logging with different log levels, for improved debugging.

The technology that we use to support SETI

Amazon Web Services
Python
django
Google Cloud
FastAPI
Slack
Amazon CloudWatch
Cloud Storage
Google Big Table
Google Big Query

Looking for a Robust Cloud Infrastructure?

case studies

See More Case Studies

ASTRO LLM: AI-Powered Citizen Scientist Assistant

ASTRO LLM is an AI-powered assistant built by Techtinium for a space science research organization, designed to support citizen scientists with instant, accurate answers to astronomy and Unistellar telescope queries. Powered by a RAG-based system, it delivers level-specific responses, from beginner-friendly explanations to advanced, equation-based insights, while reducing scientist workload and enabling scalable participation in space research.

Learn more
Contact us

Partner with Us for Comprehensive IT

We’re happy to answer any questions you may have and help you determine which of our services best fit your needs.

Your benefits:
What happens next?
1

We schedule a call at your convenience 

2

We do a discovery & consulting meeting 

3

We prepare a proposal 

Schedule a Free Consultation