Review: Zencoder has a vision for AI coding

Zencoder is an AI coding assistant with interfaces to Visual Studio Code and JetBrains IDEs. It is designed to help developers ship products faster, much like GitHub Copilot, Amazon Q Developer, and Tabnine. Zencoder also offers AI coding agents, much like Solver, GitHub Copilot Workspace, and Amazon Q Developer Agent. Tabnine also offers agents, but doesn’t differentiate them from its core product with a separate name—much like Zencoder.

Zencoder offers “Repo Grokking” to analyze and comprehend your entire repository, which customizes and improves its code generation. That’s similar to Tabnine Enterprise, and to most of the other products I’ve mentioned. In the case of Amazon Q Developer, the comparable feature seems to be Amazon Q customizations. Repo Grokking does not mean model fine-tuning, which is an advanced customization feature of Tabnine Enterprise. Zencoder looks at whole repositories, generates embeddings, and stores those in a vector database for retrieval-augmented generation (RAG), and also generates a graph representation of the repo.

[ Editor’s note: Zencoder has made three significant updates to their product since InfoWorld tested it. The company says Zencoder now supports more than 70 programming languages, it allows users to select from GPT, Claude 3.5 Sonnet, and custom models, and its coding agent can reason and modify multiple files. ]

If you have the impression that AI-powered code generation that offers agents is becoming a crowded field, you’re not wrong. There are still differences in the products, however, and it’s worth diving into how Zencoder works and how it compares with the other AI coding assistants.

Zencoder key capabilities

Code generation depends on the AI model used, the context that the model can use for reference, and the prompt presented by the user. Zencoder’s ability to gather and index context from an entire code repository (Repo Grokking, discussed below) helps to improve its code generation quality. Similar considerations apply to the Zencoder chat assistant, its code completion, its docstring generation, its unit test generation, and its code repair. Zencoder claims “industry-leading compilation rates.”

Zencoder uses a pipeline (see diagram and screenshot below) rather than just a large language model (LLM) to perform its code generation and related functions.

Zencoder preprocesses your code to provide context to its large language model. In addition, it parses and analyzes your code to improve its correctness and quality. 
Zencoder

At the left, you can see the explanation Zencoder gave for the code on the right without scanning the repository. In addition to the code on the right, you can also see the Zencoder context menu.
IDG

Zencoder IDE integrations and programming language support

Zencoder currently integrates with Visual Studio Code and JetBrains IDEs. It supports a “broad array” of programming languages, including but not limited to Python, Java, JavaScript, TypeScript, C#, and Kotlin. According to the company, full support of C++ and Go is coming soon.

Repo Grokking

Repo Grokking is essentially automatic. When you open a workspace that is managed by Git, Zencoder recognizes it as a code base and reads through the code. It creates embeddings and stores them in a vector database in the cloud for use in RAG queries. It also creates the graph that represents the structure of the code base. It does not copy the workspace, to preserve the privacy of your code.

As the repo updates, Zencoder updates its vectors and graph. In the course of my review, Zencoder got the capability of indexing in repos without Git; unfortunately, I had completed most of my testing by that point.

Once Zencoder has comprehended a code base, it can use that knowledge to help with code generation, completion, and repair, and with any chat requests that have “code base knowledge” enabled.

At the upper left, you can see Zencoder’s explanation of the entire Haystack repository after scanning it. At the lower left, you can see that the context provided for this query included the entire code base. At the right, you can see the README for the Haystack project. The Zencoder explanation includes detail not present in the README file.
IDG

Agentic Repair

Zencoder uses its understanding of your code base to help you with repairing code, generating complete unit tests, and resolving issues in real time. At this point, Zencoder only claims that its multi-step repair agents can fix simple bugs. Some competitors, for example Solver and GitHub Copilot Workspace, can go farther and fix complex bugs at the whole-repo level. Some competitors can go farther and generate integration tests in addition to unit tests.

In the settings dialog that comes up in the left-hand tab if you use the Zencoder: Custom Agents command (see screenshot below), you can see the two predefined agents as well as the interface for defining custom agents.

Zencoder currently offers two built-in agents, for unit test generation and coding, and an interface for defining custom agents. Custom agents are essentially named, saved instruction prompts for the large language model, aimed at solving specific tasks.
IDG

Zencoder compared to competitors

Zencoder (the company) helpfully prepared comparison charts for Zencoder (the product) and eight competitors. Not surprisingly, since I never really trust what one vendor has to say about another’s product even when they attempt to be fair and balanced, I think that all of these comparisons were designed to tout Zencoder. Some have inaccuracies that understate the rival’s capabilities, and several seem to be about older, less capable versions of the competition. For example, the Copilot side of the screenshot below is newer than the version Zencoder used to write its comparison.

Zencoder omitted Solver from their comparisons. Given how new Solver is, that’s understandable. Nevertheless, Solver is even more promising than Zencoder, since it not only reads whole repositories, it can modify whole repos. You can read my review of Solver here.

At the left, you can see Zencoder’s explanation of a C++ sorting function; at the right, you can see GitHub Copilot’s explanation of the same code using the o1-preview model. At the top you can see the Zencoder functions available in VS Code in the Zencoder v0.5.1 plugin. Overall, I like the explanation on the right better, but I’m partial to a documentation style that mixes words and code.
IDG

An ambitious attempt

Zencoder is, in my opinion, an ambitious attempt to enter the code generation and repair market, but it’s still wet behind the ears. Using a pipeline with error correction to avoid hallucinations, instead of straight code generation with an LLM, sounds promising, but it isn’t ipso facto better than a model that generates correct code right out of the box and suppresses hallucinations internally. Similarly, “grokking” whole repositories should provide better context than just looking at whatever pages are open in the editor, but the time it takes to digest large repos is significant, and it looks like Zencoder redoes that process from scratch whenever you refresh the repo.

I’m seeing quite a bit of convergence in this market. Zencoder’s two innovations are grokking a whole repo for context and using an error-corrected inference pipeline. Solver and agents from Amazon and GitHub can modify whole repos. It’ll be interesting to see what happens as all these efforts cherry-pick each other’s techniques.

I’ll keep Zencoder around and try it occasionally as it develops, but I wouldn’t pay money for it at this point.

Bottom Line

Zencoder is, in my opinion, an ambitious attempt to enter the code generation and repair market, but it’s still wet behind the ears.

Pros

            Does code generation and repair in seven programming languages

            Can read and index entire repositories for context

            Has an error-corrected inference pipeline to cut down on hallucinations

            Claims that its code will always compile, for supported languages

Cons

            Cannot modify more than one file at a time, at least currently

            Limited list of supported languages

            Doesn’t offer a choice of models

Cost

Free plan, up to 10 seats. Business plan, up to 200 seats, $19 per user per month after free trial. Enterprise plan, $39 per user per month. See https://zencoder.ai/pricing.

Platforms

Visual Studio Code and JetBrains IDEs on Windows, macOS, and Linux.