Converting programs to visual representation by reading the compiled binary

Gathering visual context the hard way (but it always works).

Problem

When we started, the idea of gathering the visual representation of a mobile application was considered by most to be impossible (or too hard to be worth doing). When we did come up with a solution, we found that it only worked for a tiny fraction of apps. By then we realized that although it would involve way too much coding, we could capture the data we needed from the compiled code as it was being executed by the processor. This technique extracts the visual context from any app.

Solution

The compiled code is analyzed and converted into a visual representation by looking at the binary commands that paint the screen and then using that data to recreate a visual representation of the app that can be rendered in a browser.

Impact

The beauty of this method is that it works on any code no matter how it was created; Objective-C, Swift, Unity, LiveCode, C#, Java, etc. Everything has to ultimately convert to machine language.

This is the generic method for converting code into visual context. The easier method, when possible, is by intercepting screen draws.

Status

US Patent Application Number 61928918
Filed 20-JAN-2015