Building a Unicorn – How We Helped Develop a Top Open Source Data Integration Platform with 300+ Connectors
About the client
About the client
Airbyte is an open-source data integration engine that syncs data from applications, APIs, and databases to analytical data destinations like data warehouses and data lakes. It offers a catalog of 300+ pre-built, no-code connectors to connect apps, which is the largest in the industry and is doubling every year, thanks to its open-source community.
When we first started our collaboration with Airbyte, they were a startup with a limited budget and an ambitious goal to become one of the top ELT platforms with the largest catalog of connectors. After a year and a half of working together, we helped the startup successfully accomplish all of its objectives and even exceed them. Airbyte secured two rounds of investment in just a year and cultivated a sizable community of developers who actively support existing connectors and create new ones.
What was special about this project and how did we manage to let anyone build their source connectors using a single interface? Read on to discover.
Airbyte needed a dedicated team of experienced Python engineers to help build a platform that makes it super easy for developers to create connectors and move data from various sources.
Airbyte managed to become a leading tool for data integration. So what differentiates it from other platforms? First of all, it’s open source which has made it popular among developers and data engineers who can create their own out-of-the-box connectors on the platform. Plus, Airbyte's wide coverage of platforms and high-quality connector functionality have been major factors in its success.
To make Airbyte a leading ELT platform, we needed to work in several directions:
To help our client reach their ambitious goal of creating the leading solution for building ELT data pipelines, we made it our top priority to make the platform simple to use so that new connectors can be developed in minutes.
Although the core of Airbyte is written in Java, Python was a crucial language to work with for several reasons. First, Python is perfect for writing code quickly. But most importantly, Airbyte aims to cover as many platforms as possible, and this can only happen if there's an active community contributing new connectors to the system. Since Airbyte's target audience consists of data engineers and scientists who mostly use Python, it made sense to allow them to create their own connectors more easily in their preferred language.
Flyaps is a Python-focused company, and that's why we were the perfect choice for this project.
The hardest part of making Airbyte was that it's a flexible tool that lots of people can adjust and modify. To keep it intuitive for developers, we had to monitor how people were using the platform and make sure they could add new independent connectors without any trouble.
To make it easy for developers to integrate their data sources, Airbyte provides detailed documentation and tutorials, as well as a big community of more than ten thousand members who can help. Besides, Airbyte has weekly sessions where it teaches users how to use the platform to the maximum. But all of that wasn't enough to reach our client's goal – create the largest catalog of ELT connectors to data warehouses and databases.
To ensure developers enjoy building connectors on the platform, Flyaps prioritized the developer experience. We used various metrics, such as how long it took users to create and integrate connectors, user feedback, and the willingness to recommend Airbyte, to determine if the platform was intuitive enough or if we needed to make changes to the user interface or architecture. We kept making improvements as we went along to ensure that users could develop and integrate connectors on their own without any issues.
We know developer experience is important, so we've created a platform that is easy to use and has a global community that actively contributes to it – and recommends Airbyte to others.
Because Airbyte is an open-source platform, there was a risk that new connectors added by users wouldn’t meet the platform’s standards and the overall quality of the platform would suffer. To mitigate this risk, we implemented quality control measures. Before adding connectors to the system, we evaluate them to ensure they meet our standards. We also make this evaluation process visible to users, letting them improve their connectors on their own.
To keep Airbyte efficient and reliable, we implemented a certification system that checks if a connector meets the platform’s requirements or if it has to be improved. There are several criteria for a quality connector:
To make the process of quality control transparent, we created an interactive tutorial: it’s like test-driven development, but on a higher level – with end-to-end tests. Tests check if the connector is able to execute the needed commands correctly, and if not, it tells the user where it should be improved. After testing, the connector receives a grade that indicates the overall level of its reliability. Then our team verifies it once again and adds a connector to the Airbyte catalog.
With this quality control, we encourage users to contribute to Airbyte because they know their connectors will be upgraded with each new iteration and potentially by other users who add new features. To keep people engaged, the Airbyte team even had Hacktoberfest where they paid users who could build a reliable connector. We made sure to check all the connectors users developed to ensure they met our standards before adding them to the platform.
To achieve the goal of building a large community and hundreds of connectors, Airbyte needed to be intuitive for every user. It was a challenge to create a clear and simple interface that would enable developers to move data from anywhere, for any use case.
There are so many different API standards, databases, and streaming platforms out there, each with its own rules for extracting data! It's definitely not an easy task to build one interface that looks intuitive to everyone, whether they're building GraphQL or REST API. To tackle this challenge, we had to get creative and come up with some insights to make Airbyte as user-friendly as possible.
We had brainstorming and discussions with devs, architects, UI/UX designers, and data analysts from the client’s team. Everyone shared their vision for how the interface should look using their experience with different databases – and with it, we managed to find a universal option that is simple to use and work with.
We created a low-code platform for building ELT connectors. It allows data analysts to use pre-built components, templates, and drag-and-drop interfaces to create connectors quickly and easily, without having to write extensive lines of code.
By working closely with our clients and making the interface easy for everyone, we made Airbyte a top ELT platform with more than ten thousand community members.
Airbyte exceeded everyone's expectations and became a global data integration platform with broad possibilities and 300+ pre-built connectors.
The platform is loved by users for its intuitive interface, reliable connectors, and opportunities Airbyte provides for data analysis.
In just under two years, Airbyte has progressed to a $1.5 billion valuation and $181.2 million in funding.
Flyaps delivered over 30 integrations for our team and built a comprehensive test suite for us to use with our users. They were also in charge of building developer tooling for our data integration platform. Their engineers went above and beyond to help us ship our product. We enjoyed working together, and appreciate how adaptive they were to our needs and feedback.
Flyaps is a close-knit team of highly experienced developers specializing in Python and React. We build products that exceed the expectations of our clients – and are loved by their users.Let’s discuss your project