Why traditional methods fall short in automating B2B software UIs

Many B2B software applications rely on complex user interfaces (UIs) that can be challenging to automate. Traditional approaches that rely solely on neural language models (NLMs) struggle in this area. While NLMs excel at understanding APIs, UIs present a different beast. To control a B2B software UI, an NLM would need to convert the interface into text, images, or sequences for processing. This approach suffers from several drawbacks:

• Loss of Valuable Data: Converting UIs to simpler formats throws away crucial information about the structure and layout of the interface.
• Complexity Overload: Real-world B2B software UIs can be intricate and overwhelming even for the most powerful NLMs.
• Ambiguous Instructions: Describing actions in natural language can be unclear and prone to misinterpretation.
• Limited Practicality: Existing NLM-based automation methods often lack real-world application or require extensive user input to define actions.

Symbolic algorithms offer a viable alternative

The field of web and robotic process automation (RPA) offers a promising alternative with symbolic algorithms. These algorithms have a proven track record of success in automating specific tasks within B2B software. They are fast, efficient, and provide clear explanations of their actions. However, symbolic algorithms struggle with generalizability - they often require a custom-designed approach for each unique task.

The Future: A hybrid approach with neuro-symbolic integration

Both NLM and symbolic approaches have limitations when applied to B2B software UI automation. Researchers are exploring a hybrid approach that combines the strengths of both, offering several advantages:

• Richer UI Models: We can define complex UI structures beyond simple tokens, making them compatible with both symbolic algorithms and neural networks. This allows for a more nuanced understanding of the interface.
• Explainability and Efficiency: Symbolic algorithms provide explainable and efficient automation, while neural networks can leverage their language, vision, and zero-shot reasoning capabilities to adapt to unforeseen situations.
• Continuous Improvement: As more data becomes available, both the symbolic and neural components of the system can learn and improve over time.

Building the new model: Learning by Demonstration for B2B software

A new model called Learning by Demonstration with Abstraction Modules (LAM) takes a unique approach to B2B software UI automation. LAM focuses on learning by observing a human interact with the software. By observing these demonstrations, LAM aims to replicate the process even if the UI changes slightly. This approach offers several benefits:

• Transparency: Unlike black-box models, LAM's "recipes" are transparent and can be inspected by humans to understand the automation logic.
• Direct Execution: Once a task is demonstrated, LAM can directly execute the routine without needing further observation or adaptation. This translates to faster and more efficient automation.
• Deep Understanding: Over time, LAM builds a "conceptual blueprint" of the underlying functionalities within the B2B software. This allows LAM to adapt to minor UI changes and generalize its automation capabilities.

LAM as a Bridge for B2B Workflows

LAM acts as a bridge, allowing businesses to automate complex workflows within their B2B software applications through demonstrations. This can significantly improve efficiency and reduce human error in repetitive tasks.

This revised text focuses on the challenges of automating B2B software UIs and presents LAM as a promising solution that leverages a combination of NLM and symbolic approaches.

‍