Building a Unicorn – How We Helped Develop a Top Open Source Data Integration Platform with 300+ Connectors

Using Python to attract the data engineering community

Although the core of Airbyte is written in Java, Python was a crucial language to work with for several reasons. First, Python is perfect for writing code quickly. But most importantly, Airbyte aims to cover as many platforms as possible, and this can only happen if there's an active community contributing new connectors to the system. Since Airbyte's target audience consists of data engineers and scientists who mostly use Python, it made sense to allow them to create their own connectors more easily in their preferred language.

Flyaps is a Python-focused company, and that's why we were the perfect choice for this project.

Prioritizing developer experience to enable the creation of connectors

The hardest part of making Airbyte was that it's a flexible tool that lots of people can adjust and modify. To keep it intuitive for developers, we had to monitor how people were using the platform and make sure they could add new independent connectors without any trouble.

To make it easy for developers to integrate their data sources, Airbyte provides detailed documentation and tutorials, as well as a big community of more than ten thousand members who can help. Besides, Airbyte has weekly sessions where it teaches users how to use the platform to the maximum. But all of that wasn't enough to reach our client's goal – create the largest catalog of ELT connectors to data warehouses and databases.

To ensure developers enjoy building connectors on the platform, Flyaps prioritized the developer experience. We used various metrics, such as how long it took users to create and integrate connectors, user feedback, and the willingness to recommend Airbyte, to determine if the platform was intuitive enough or if we needed to make changes to the user interface or architecture. We kept making improvements as we went along to ensure that users could develop and integrate connectors on their own without any issues.

We know developer experience is important, so we've created a platform that is easy to use and has a global community that actively contributes to it – and recommends Airbyte to others.

Integrating the certification system to reduce the quality risks of community-built ELT connectors

Because Airbyte is an open-source platform, there was a risk that new connectors added by users wouldn’t meet the platform’s standards and the overall quality of the platform would suffer. To mitigate this risk, we implemented quality control measures. Before adding connectors to the system, we evaluate them to ensure they meet our standards. We also make this evaluation process visible to users, letting them improve their connectors on their own.

To keep Airbyte efficient and reliable, we implemented a certification system that checks if a connector meets the platform’s requirements or if it has to be improved. There are several criteria for a quality connector:

1. It corresponds to the Airbyte protocol;
2. It executes four basic commands: describe yourself (what it accepts as output and input), synchronize, save your state, give me your state;
3. It has the correct configuration.

To make the process of quality control transparent, we created an interactive tutorial: it’s like test-driven development, but on a higher level – with end-to-end tests. Tests check if the connector is able to execute the needed commands correctly, and if not, it tells the user where it should be improved. After testing, the connector receives a grade that indicates the overall level of its reliability. Then our team verifies it once again and adds a connector to the Airbyte catalog.

With this quality control, we encourage users to contribute to Airbyte because they know their connectors will be upgraded with each new iteration and potentially by other users who add new features. To keep people engaged, the Airbyte team even had Hacktoberfest where they paid users who could build a reliable connector. We made sure to check all the connectors users developed to ensure they met our standards before adding them to the platform.

An intuitive low-code platform that fits everyone's needs

To achieve the goal of building a large community and hundreds of connectors, Airbyte needed to be intuitive for every user. It was a challenge to create a clear and simple interface that would enable developers to move data from anywhere, for any use case.

There are so many different API standards, databases, and streaming platforms out there, each with its own rules for extracting data! It's definitely not an easy task to build one interface that looks intuitive to everyone, whether they're building GraphQL or REST API. To tackle this challenge, we had to get creative and come up with some insights to make Airbyte as user-friendly as possible.

We had brainstorming and discussions with devs, architects, UI/UX designers, and data analysts from the client’s team. Everyone shared their vision for how the interface should look using their experience with different databases – and with it, we managed to find a universal option that is simple to use and work with.

We created a low-code platform for building ELT connectors. It allows data analysts to use pre-built components, templates, and drag-and-drop interfaces to create connectors quickly and easily, without having to write extensive lines of code.

An ultimate solution to the data integration problem

By working closely with our clients and making the interface easy for everyone, we made Airbyte a top ELT platform with more than ten thousand community members.