Attention Similarity Bias is Worse in Vision
70% of what a ViT token gets back from attention is information it already has. I measure this across patch sizes, show it compounds through layers, and test a projection that removes it.
Hi! I'm an ML Principal at Salesforce. Previously founding ML engineer at Convergence where I created our first web agents and developed our OS web agent proxy lite. Before that, ML engineer at Cohere where I started the Command project and led the creation of our SOTA reward models.
Posts
How will AI Agents use computers?
Why I think Coding Agents are the future of agentic computer use.
Pyxel Playground: A Web-Based Game Editor
A browser-based editor for making retro games with Pyxel, a Python retro game engine.
An Avatar game with realistic physics
Finding the physics of water bending.
A dataset of ran code
Learning from state changes in 1 million Python programs.