Tuesday, November 17, 2026
Follow on LinkedIn

Jeffrey Zang Is Rethinking How AI Agents Read a Computer

Through OSX-Query, he is testing whether structured operating-system data can make computer-use agents faster, lighter, and practical enough to run in the background.

A computer-use agent must first understand the machine before it can act. Many current systems rely on screenshots and visual models to interpret what appears on the screen.

Jeffrey Zang, the 19-year-old founder of Proletic, has taken a different approach by designing a harness that represents computer state as compact text.

The system is called OSX-Query. It draws from accessibility trees, window metadata, and the user-interface hierarchy rather than depending on a screenshot-and-click process for every interaction.

Zang designed it so smaller, text-only models can operate a computer without requiring a heavyweight vision model for each action.

“I wanted to test whether an agent could use structured computer state as compact text,” Zang said. “The goal is to make computer-use agents faster and more context-efficient.”

That question places Zang inside one of the most active areas of artificial intelligence. Computer-use agents are designed to click, type, and operate applications rather than remaining inside a chat window.

The promise is straightforward, but the engineering problem begins before the agent performs any action. It needs a usable representation of what is happening across the machine.

Zang’s contribution focuses on that perception layer. OSX-Query gives the model a text-based account of the computer’s current state.

His aim is to reduce the amount of context required compared with an approach that repeatedly processes screenshots.

“If an agent is going to run in the background, it has to be cheap and fast enough to remain available,” he said. “That becomes harder when every small action depends on a large visual model.”

Zang began exploring the broader problem through Opus, a computer-use agent he built about a year earlier.

Opus executes real mouse and keyboard inputs in the background through a combination of the macOS Accessibility API, AppleScript, JavaScript injection, and Bash.

He took the project from its initial build to launch in three days. Opus attracted more than 500 beta users, drew over 200,000 views, and generated inbound interest from venture capital firms.

“Opus went from zero to launch in three days and reached real users,” Zang said. “OSX-Query pushed me to think more closely about speed and context efficiency.”

OSX-Query emerged from that next stage of work. Zang describes it as an agent harness, meaning the engineering layer that helps a model interact with the environment around it.

In this case, the harness is designed to give a text-only model enough information to drive computer actions.

The architecture represents a specific bet about the future of computer-use agents. Instead of assuming every system will depend on increasingly large models, Zang is exploring whether carefully structured information can allow smaller models to perform useful work.

Model size can affect where and how often an agent can operate. Zang wants computer-use systems to become practical enough for routine background use rather than being reserved for tasks that justify heavier processing.

“I am trying to make agents cheap, fast, and accessible enough to run everywhere,” he said. “That means designing for smaller, text-only models.”

His work also reflects the constraints of building without the resources available to major AI laboratories.

Those organizations are pursuing computer use with far more funding, compute, and engineering resources than Zang has available.

He is approaching the same category while studying computer science at the University of Waterloo and building his own company.

Zang does not present that gap as evidence that his method will prevail. It influences the questions he chooses to pursue.

With fewer resources, efficiency cannot be postponed until after the system works. It has to shape the architecture from the beginning.

That pressure has appeared throughout his work. Zang has repeatedly built quickly, released projects to users, and used the response to decide what deserved further development.

Opus tested whether people would try a background computer-use agent.

OSX-Query examines a narrower technical question about how the agent receives information from the machine.

Rather than creating another interface for users, OSX-Query works beneath the visible application and changes the information available to the agent.

It is intended to support the operating-system-level direction Zang is pursuing through Proletic.

Proletic is developing a desktop system that can surface useful next actions during everyday computer use.

The company remains in stealth, and the product is still being built. OSX-Query addresses part of the technical foundation that could make continuous background assistance more practical.

The technical problem can be easy to underestimate. A computer screen may appear simple to a person, but an agent needs information it can process before it can select and complete an action.

The format of that information affects what kind of model is required and how much context the system must handle.

Zang believes smaller, text-only models could make computer use faster and more accessible. OSX-Query is his attempt to test that belief through a working harness rather than an abstract proposal.

His earlier work gives the project a concrete foundation. Opus demonstrated that he could build an agent capable of real mouse and keyboard actions and release it to hundreds of users.

OSX-Query now examines whether the system beneath that behavior can be made lighter.

“The question is not only whether an agent can use a computer,” Zang said. “It is whether the system can do it efficiently enough to become part of everyday computing.”

That is the problem Zang has chosen to pursue. He is asking whether a compact representation of computer state can reduce dependence on repeated visual processing and allow smaller models to operate more effectively.

OSX-Query remains an evolving technical project built around that view. Its importance lies in the architecture Zang designed and the alternative it presents within computer-use AI.

For agents to operate continuously, they will need efficient ways to understand the machines they are expected to use.

Zang is testing one possible route by changing what the model receives before it clicks, types, or takes the next action.

Kavichselvan
Kavichselvan
Kavichselvan is a Cybersecurity Enthusiast and Journalist covering Cyber Attacks, Threats, Breaches, Vulnerabilities and other happenings in the cyber world.
0-Hour Cyber Security Alerts!
Get the latest Cyber security News sent directly to your inbox.

Cyber Security Guide

Latest Cyber News

Expert Talks