All product videos

How does an AI learn to operate software on its own?

SharePoint administrators and site owners can watch how Cognillo's patent teaches an AI to operate software the way a person does. See the AI learn to find buttons, text boxes and drop-downs on a screen, link each control to an action, prove its accuracy before acting, take a plain-English request and return a confirmation with a screenshot. The explainer covers U.S. Patent No. 10,782,966, issued September 22, 2020.

This video is a simplified illustration of embodiments described in the patent. The patent's claims define the scope of the invention. Narrated with an AI-generated voice.

Video chapters

  1. 0:00 Intro
  2. 0:14 The old way: hand-written scripts
  3. 0:26 Step 1: Learn to see
  4. 0:47 Step 2: Link controls to actions
  5. 1:02 Step 3: Prove it, the test loop
  6. 1:33 Step 4: Understand people
  7. 1:50 Step 5: Do the work
  8. 2:04 Learning from documentation
  9. 2:14 More from the patent
  10. 2:32 Timeline
  11. 2:45 Summary

Full transcript

0:00 Almost all software is built for people looking at a screen. This is a Cognillo invention that teaches an AI to use software the way people do: by looking, clicking, and typing.

0:14 Traditionally, automating software meant hand-written scripts, tied to exact spots on the screen. When a button moves in an update, the script breaks, and a programmer has to fix it.

0:26 The patent describes a different approach. First, the AI learns to see. It is given many screenshots of software interfaces, and using image recognition, it learns to find the controls: buttons, text boxes, checkboxes, and drop-downs. It can read the labels beside them, and it pinpoints the position of each one.

0:47 Next, it links every control to an action. Click this button. Type into this text box. Tick this checkbox. Choose from this drop-down. Position plus action become a set of execution instructions.

1:02 Then comes the step at the heart of the patent: it has to prove what it learned. The AI performs each action at the position it identified, captures an image of the result, and checks whether the intended thing really happened. That gives it an accuracy score. If the score falls short of the required threshold, it takes in more screenshots, learns again, and tests again, repeating until it passes. Only then are its instructions identified as live, and ready for real use.

1:33 The AI also understands people. You can ask in plain language, by chat, email, or voice. Please add Bob Smith to the accounting system. Natural language processing works out the intent, and converts the request into the exact steps to run.

1:50 Using its verified instructions, it operates the real application, clicking and typing just as a person would. When the task is done, it sends a confirmation, with a screenshot of the result.

2:04 It can learn from documentation too. User manuals, training guides, and how-to articles show it how a piece of software works, step by step.

2:14 The patent describes more. When an interface changes, the system can adapt, and alert you if a task is no longer possible. You decide how much human review is required before it acts. And it works across applications, without anyone hand-coding the automation.

2:32 The idea was first filed in July 2017, and the patent was issued in September 2020, more than two years before GPT-4 was released.

2:45 A self-learning AI that sees the screen, proves its accuracy, and operates software for you. Invented at Cognillo.

Related tasks

To see it running on your own tenant, email hello@cognillo.com