Devin: A Hands-On Review of the First AI Software Engineer

If you recall, a product named Devin was launched, claiming to be the first AI software engineer. It boldly stated it could, at the very least, replace junior software engineers. The hype was significant, Cognition AI received substantial funding, but the product itself remained elusive for a long time, fueling widespread speculation about its capabilities.

Now that Devin is publicly available, what's the reality? How capable is it? In this article, we'll conduct a hands-on demo to find out. We will also explore whether Devin lives up to its initial hype and discuss if it's truly capable of taking over the role of a junior software developer.

First Impressions and Setup

Upon visiting Devin's website, the claim is front and center: "The first AI software engineer." It promises to crush your backlog with a personal AI engineering team, automatically creating GitHub pull requests and handling tasks from a simple instruction. Let's find out if this holds true.

Interestingly, Cognition has also acquired Windmill, another AI coding product. This means Devin is now empowered with Windmill's capabilities, effectively merging the two tools.

Getting started requires logging in and connecting a GitHub account. Devin presents two options for GitHub integration: one for repository admins and another for solo developers.

It's crucial to understand that Devin operates on its own virtual machine, not your local machine. As the launch materials explained, Devin has its own environment with a command line, terminal, and IDE. The setup process involves authorizing your GitHub account on Devin's machine.

The process is straightforward: 1. You are given a link and a one-time code. 2. Open the link on a device where you are logged into GitHub. 3. Enter the code to authorize the device.

After a two-factor authentication step, you grant Devin the necessary permissions, and the connection is established.

A quick note: Devin is a paid service. For this review, I started with a $20 credit to explore its functionality, especially since I already use several other AI tools like Cursor and am curious to see how Devin compares.

Configuring the Project

After payment, the next step is to select a repository. I created a new public repository on GitHub named php-todo with a README.md file.

Initially, the new repository didn't appear in Devin's list. After skipping that step, I authorized the Devin integration, but I limited its access to only the php-todo repository for security.

To make the repository available to Devin, you must "index" it and then "add to machine." This process clones the repository into Devin's virtual machine, which runs on Ubuntu 22.04. The repository is cloned into a specific path on Devin's machine (/home/devin/agent-repos/php-todo), and you're warned not to move or delete this directory.

The setup wizard then guides you through configuring the project, including commands for linting, testing, and running the application locally. For now, I skipped most of these, wanting to see how Devin handles a minimal setup.

Building a To-Do App from Scratch

With the setup complete, Devin read the README.md file and noted that the repository was minimal. My first prompt was direct:

"You have to build a to-do app using PHP."

Devin understood immediately and laid out a plan: "Okay, I understand. This is the repository where we need to build a PHP to-do application from scratch."

Its first logical step was to install PHP, as it wasn't present on the base machine. It ran the necessary commands, and I could see the output from its terminal directly in the UI. It then proceeded to install Composer to manage dependencies, even creating a composer.json file when it couldn't find one.

After the setup, Devin has its own interactive browser. To run the app locally, you need to provide the command. For a simple PHP app, this is:

php -S localhost:8000

I provided this URL to Devin, and the development server started. Now, it was time to give the main instruction.

"Start building the PHP to-do list app."

Devin's thought process was transparent: "I will start building the PHP to-do list app. Now, let me create the basic structure and files needed for a functional to-do application. First, let me create the main index.php file, which will be our entry point."

It created the file and, after a few moments, the basic application was ready. I opened Devin's browser and tested it. It was a functional, albeit very plain, to-do list. You could add and complete tasks.

Improving the User Interface

The initial result was functional but visually unappealing. I gave Devin a new challenge:

"The UI doesn't look good. You can do better, man. Come on."

Devin took the feedback seriously and began editing the index.php file to add CSS and improve the layout. This is where it started to feel like collaborating with a real developer. The most impressive part is its ability to directly create a pull request on GitHub.

After a few moments of processing, Devin presented the updated application. The transformation was remarkable. The new UI was clean and modern, with a welcoming message and counters for total, completed, and remaining tasks. It had taken my vague feedback and produced a genuinely impressive result.

From Code to Commit: The Automated Pull Request

Now for the final test: pushing the code to GitHub. I gave the command:

"Create a pull request on GitHub with the changes now. LGTM."

Devin immediately started running git commands: - It ran git status to see the changes. - It created a new feature branch with git checkout -b feature/add-todo-app. - It added the modified index.php, composer.json, composer.lock, and a newly created .gitignore file to the staging area. - It created a commit with a descriptive message. - It pushed the new branch to the remote repository.

Finally, using the gh CLI, it created a pull request with a detailed summary of the changes.

A notification appeared on GitHub almost instantly. The pull request was there, complete with all the changes Devin had made. I could review the code—the new CSS, the updated PHP logic, and the project configuration files—just as I would with a human colleague. If everything looked good, I could merge it.

The Verdict: Hype vs. Reality

So, what's the final impression? Devin is an amazing product. The effort required to build it and the real-world software problems it aims to solve are commendable.

The standout features are: - A Dedicated VM: Unlike tools that modify your local machine, Devin works in its own isolated environment, which is a significant advantage. - Integrated Browser: The ability to see the running application inside Devin's own browser is incredibly useful for web development. - Automated Pull Requests: This is the killer feature. It truly simulates the workflow of a developer completing a task and submitting it for review.

However, the initial hype often creates fear and unrealistic expectations. This tool was built by real software developers. The question remains: will it replace them?

In its current state, AI is a powerful tool, not a replacement for human expertise. The developers who learn to leverage AI will be the ones who succeed. Understanding how AI and machine learning work behind the scenes provides a massive advantage.

If you find yourself in a comfort zone, performing repetitive tasks that can be easily automated, then yes, AI might replace that part of your job. But for complex problem-solving, architectural design, and creating novel solutions, human ingenuity is still indispensable.

The argument that these tools can build entire companies on their own is flawed. If that were true, why haven't we seen a flood of AI-built startups? The answer is that building a company involves far more than just writing code.

It's childish to think, "I won't learn Python because AI will replace me." In fact, you need to learn Python more deeply than ever to write the kind of code that AI cannot write. If AI could write any code imaginable, why would companies like OpenAI, Google, and Meta still be hiring thousands of software engineers?

The nature of the job will change. Perhaps where 25 engineers were needed, 15 will suffice with AI's help. That number may decrease over time, but it's unlikely to become zero until we reach Artificial General Intelligence (AGI), a scenario that remains in the realm of science fiction for now.

Your best move in today's world is to work hard and keep learning. AI is just a tool. Use it to maximize your potential, whether in learning or building. Devin is an amazing tool, and I'm excited to explore its capabilities further with more complex applications.