Moving to Meltano for code-first ELT

Why teams migrate to Meltano

Meltano provides an open-source DataOps platform that uses the Singer data format for source integrations. The 17-person team works remotely and recently raised $8.2 million to extend its seed round to $12.4 million. This funding, led by Venrock, supports the company’s return to its original vision of an end-to-end platform. The company was originally built by GitLab to improve its own data lifecycle platform. Meltano provides 300+ connectors out of the box, while competitors like Airbyte and Fivetran provide over 700 connectors. Many teams migrate from Fivetran because its Monthly Active Rows pricing model creates unpredictable costs when volumes increase. Other teams move from Airbyte because they want to avoid the infrastructure overhead and connector maintenance that comes with self-hosting. I recommend Meltano for technical teams that want to manage data pipelines through a CLI-driven workflow. You probably already know that managing infrastructure consumes significant engineering hours. Meltano helps prevent this by integrating tools like dbt, Airflow, and Great Expectations into a single control plane.

Setting up your first project

Setup requires a system running Linux, macOS, or Windows with Python 3.10, 3.11, 3.12, 3.13, or 3.14. You should use a dedicated Python virtual environment for your projects.

First, install the pipx package manager using python3 -m pip install --user pipx. Run pipx ensurepath to update your path. You can then install the package with pipx install meltano. To create a project, navigate to your working directory, run mkdir meltano-projects, and then cd meltano-projects. Run meltano init my-meltano-project to generate a meltano.yml file. This file includes three environments: dev, staging, and prod. To add a Google Sheets extractor, use meltano add tap-google-sheets. You can also add a utility like meltano add utility dbt-snowflake. If a connector is not available in MeltanoHub, use meltano add --custom <tap name> to add a custom Singer tap. You can verify your installation by running meltano invoke <plugin> --help.

Comparing orchestration and workflows

Meltano integrates with Airflow for workflow orchestration and dbt for transformations. It also uses Great Expectations for data quality and Superset for visualization. For teams that need to load data into a warehouse, you can add a loader like target-postgres or target-snowflake. A typical pipeline uses meltano run tap-postgres target-snowflake dbt-snowflake to execute the sequence. Because Meltano relies on third-party open source projects, it adds an intermediate layer that simplifies the differences between best-in-class components like Airflow, dbt, and Great Expectations for modern data teams during their daily orchestration. While Fivetran handles schema changes automatically, Meltano requires you to manage these changes via Git. Airbyte provides a UI for non-technical users, but Meltano’s CLI-only interface requires significant engineering expertise. I find that using the meltano invoke command helps prevent deployment failures. Airbyte offers 700+ connectors and a 25,000-member community, but its SaaS tier limits sync frequency to once per hour. Fivetran is a managed solution that uses Monthly Active Rows as its primary metric. How will your team handle the lack of a native user interface?