Is GitHub Copilot Worth It? Expert Opinions and Real-World Application
The question of whether AI tools will replace developers has been around for a while now. GitHub Copilot has become a reason for the most significant concerns in this regard, becoming a viral tool for many people involved in coding. Just within the first five months of its official launch, 20,000 have adopted this new tech. There are hundreds of discussions on platforms like X (former Twitter) or Reddit, where people can’t wait to share their experiences with GitHub Copilot or ask others about it.
As a software development team with 11+ years in the business, we at Flyaps are very attentive to new tools that can improve the quality of our work and reduce development time and costs for our clients. So we decided to explore whether GitHub Copilot lives up to the hype. Should it become a must-have tool for every single developer? Keep reading to find out.
Why has GitHub Copilot become more popular than other AI copilots?
GitHub Copilot is a cloud-based tool that analyzes the context of your code and suggests relevant code snippets, functions, or even entire lines of code as you type. It’s also a chat, where you can find solutions to your programming issues.
The tool was first based on a version of the GPT-3 model fine-tuned for coding tasks. Now, GitHub Copilot has become one of the early adopters of GPT-4, the most powerful OpenAI model (at least, until GPT-5 hasn’t yet been released). We'll be mentioning GPT-4 a lot here, as it is the basis for GitHub Copilot, but we will not dwell on all of the specifics of the model.
If you are interested in the features of GPT-3 and GPT-4 in comparison to other LLMs you will find it in our article about 7 best LLM models for your business.
Natalia Palamarchuk
GitHub was already a popular platform among developers when GPT-3 was released and showed that programming languages can be generated as well as natural ones. It partially explains why their AI copilot has gotten a lot of attention among other code-generative tools specifically. Still, there are also other things worth praising:
Easy to install
You can get access to GitHub Copilot through most of the popular integrated development environments (IDEs) such as Visual Studio Code (VS Code), Neovim, and IDEs developed by JetBrains (such as IntelliJ IDEA, PyCharm, WebStorm).
For example, to install the AI assistant through VS Code you will only need two things: a GitHub account and installed VS Code.
Enhanced productivity, satisfaction and well-being of developers
GitHub considers a number of ways that can affect developers’ productivity. Therefore, they use the SPACE productivity framework, which includes satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. By looking at more than just speed, GitHub tries to understand the broader impact of tools like Copilot on developers' well-being and work satisfaction. Their own survey results show that 60-75% of users reported feeling more fulfilled with their job, less frustrated when coding, and able to focus on more pleasant work when using Copilot. This shows that tools like Copilot make coding more enjoyable and less mentally taxing.
Still, while developer satisfaction is essential, speed and efficiency are also critical. GitHub's experiment showed that developers using Copilot completed tasks significantly faster than those who didn't – 55% faster on average. Specifically, developers using Copilot took 1 hour and 11 minutes to complete a task, while those without Copilot took 2 hours and 41 minutes. In addition, the completion rate was higher for the Copilot group (78% compared to 70%).
Remarkable predictive ability
GitHub Copilot uses the context surrounding your code to generate “fill in the blank” completions, which look much like an autocomplete. It is possible mostly because of the GPT-4 model Copilot is based on. This LLM is really good at recognizing patterns in code and predicting what the user might need next. This model excels at identifying and generating relevant code based on the given context. Therefore, by analyzing the surrounding code, GPT-4 allows Copilot to predict and suggest code that fits well with what you are currently writing, reducing the amount of manual coding.
Pattern recognition proficiency
Identifying patterns between words and the structure of language is another strength of the GPT series. Because the model on which Copilot is based is trained on the vast amount of public code repositories on GitHub, the AI assistant understands common coding patterns, idiomatic expressions and effective coding practices across multiple languages.
Continuous research and improvement
GitHub is aware that AI-powered code completion tools like Copilot are still a work in progress. They keep experimenting to understand Copilot better, including its effects on code quality and developer practices. By staying engaged with the research community and gathering feedback from users, GitHub aims to improve Copilot and find out more about its impact on developer productivity.
As we said, AI-driven code assistant is still a relatively new concept so naturally such tools have their flaws, GitHub Copilot included. Let’s take a look at some of them.
Limitations of GitHub Copilot
Most of these drawbacks are common to AI code assistants in general, but Copilot also has specific challenges, like the one outlined in the first point of the following list.
Offers different versions of the GPT models depending on the location
“Depending on where you use GitHub Copilot, you may have access to either GPT-4 or GPT-4 Turbo. This could be due to many reasons, including infrastructure capabilities or regional demand. Users in regions with GPT-4 Turbo will experience faster code completion thanks to the model's enhanced capabilities. GPT-4 Turbo is also more cost-effective as the token price is lower than GPT-4 offers.”
— Oleh, developer at Flyaps.
Different GPT versions can cause problems in collaborative projects where team members are in different places. For instance, developers with different versions of Copilot might get different suggestions, which could cause problems with coding styles, efficiency, and the codebase. This makes it harder to use Copilot with a global team.
Less effective for complex and nuanced programming tasks
"GitHub Copilot is good at solving small coding tasks, helping to fix problems in the code, and optimizing the code. But it's not effective at solving big, complex tasks."
— Ruslan, front-end developer at Flyaps, about his experience with GitHub Copilot.
Think of Copilot as a kind of automation tool. It can be very good at filling in typical patterns, but nothing is as effective as a person's creative mind when it comes to solving problems.
For example, when tackling complex problems or designing innovative solutions, the creative mind and expertise of a developer are irreplaceable. Copilot can help by making suggestions and reducing repetitive coding tasks, but the nuanced understanding required for complex tasks is often beyond its current capabilities. It may suggest syntactically correct but contextually inappropriate solutions that require careful review and adjustment by the developer.
Distraction and overwhelm
One of the biggest disadvantages of using Copilot is that it can be distracting. Copilot can suggest many different things, which can be overwhelming for some developers. This can lead to a loss of concentration, ultimately slowing down the coding process.
Licensing and ownership issues
Because GitHub Copilot has been trained on a large dataset of code from public repositories, it may generate code that is similar to existing works. The generated code could be considered a derivative work over which the original copyright holder has control. For example, if Copilot suggests code that mirrors copyrighted software, the copyright owner could sue the developer who uses that code.
Developers must ensure that they do not inadvertently infringe someone else's intellectual property. This may involve obtaining permission from copyright holders or modifying the generated code sufficiently to avoid similarity.
The legal landscape surrounding Copilot also includes concerns about using code snippets from open-source projects. Although open-source licenses often allow reuse, they come with specific terms and conditions. The misuse of open-source code, as suggested by Copilot, can also lead to legal complications.
As concerns about the legal and ethical side of code AI assistants have become more common, GitHub has also started to take some steps in response. For example, this is one of the most common comments in discussions about GitHub Copilot on various platforms. This particular one comes from the GitHub community:
The answer a community manager gave, albeit briefly, was that while the probability of generating the exact code from the training set is low, there's still a chance that some suggestions will closely match existing code, especially if there's insufficient context or if the solution is very common. To mitigate this, GitHub provides a filter that suppresses suggestions that match public code on GitHub. Users and administrators can enable this filter to avoid potential licensing issues. But despite these measures, developers should still perform rigorous testing, IP scanning and security checks on all Copilot-generated code to ensure compliance and suitability before use.
Performance issues and compatibility
Copilot users might experience performance issues or compatibility problems with certain extensions or versions of VS Code. While auto-updates and frequent releases aim to address these issues, they can still cause disruptions.
Now that we have an understanding of the pros and cons of GitHub Copilot, it's time to talk about the real-world experience of its use.
A real-world example of GitHub Copilot usage
There are a lot of talks about different AI copilots these days, but the most valuable of them are those that are written by experts who work in big enterprises. Just recently a former Facebook executive said that coding with a copilot is like coding with a demigod that amplifies your abilities and anticipates your every move. The media started quoting his tweet immediately.
Therefore, to describe the experience with GitHub Copilot, we take the words of Bharat Ruparel, a cloud architect who works with Azure Cloud and AI for Microsoft. He’s been using the tool for half a year now, so let’s look at how it was for him.
Bharat was working on deploying and debugging an Azure OpenAI embedding model in two different Azure regions. His goal was to create a simple Python program to generate vector embeddings for the word “cat.” Initially, he set up his deployment in the East US region and successfully ran his program, which produced the expected output.
However, when he tried to deploy the same model in the North-Central US region, he faced unexpected issues. Even though he updated the settings, the terminal was still referencing the previous values, causing the program to use the old deployment.
At this point, Bharat turned to GitHub Copilot for help. Using its chat interface, he asked for an explanation and received a clear answer: turned out, he needed to use the `override` argument to ensure his new environment variables were recognized. Following this advice fixed the issue, and his program finally ran correctly using the new deployment.
Bharat concluded that it’s better not to rely on GitHub Copilot (or any other copilot for this matter) as a substitute for a developer’s skills. It is an assistant and better to be used only when a specialist understands the fundamental issue but has doubts about how to solve it.
As our expert, Ruslan, rightly pointed out: “Sometimes GitHub Copilot is perfectly accurate, sometimes it gives you absolute nonsense. So it is important to validate it every time.”
So, where is the place of GitHub Copilot in the future of development?
"GitHub Copilot does represent the future of coding, being like a helpful teammate who can suggest little insights, fix some problems, and write some code."
— Ruslan, front-end developer at Flyaps
In other words, it's a great support tool, but only in the hands of skilled developers who don't rely on it too much.
Copilot can handle many of the repetitive and time-consuming aspects of coding. However, organizations still need talented developers who can understand complex requirements, design robust systems and think critically and creatively about software architecture.
Looking for talented AI-focused developers who use best practices and innovative tools to deliver the best results? Drop us a line! Not only do we use the best tools, we also build the best AI-driven tools for others (in a range of fields such as HR and recruiting, urban planning or logistics). Let’s discuss how we can help bring your project to life.
Flyaps
