Automating the Enterprise with Foundation Models

Abstract.

Automating enterprise workflows could unlock $4 trillion/year in productivity gains. Despite being of interest to the data management community for decades, the ultimate vision of end-to-end workflow automation has remained elusive. Current solutions rely on process mining and robotic process automation (RPA), in which a bot is hard-coded to follow a set of predefined rules for completing a workflow. Through case studies of a hospital and large B2B enterprise, we find that the adoption of RPA has been inhibited by high set-up costs (12-18 months), unreliable execution (60% initial accuracy), and burdensome maintenance (requiring multiple FTEs). Multimodal foundation models (FMs) such as GPT-4 offer a promising new approach for end-to-end workflow automation given their generalized reasoning and planning abilities. To study these capabilities we propose ECLAIR, a system to automate enterprise workflows with minimal human supervision. We conduct initial experiments showing that multimodal FMs can address the limitations of traditional RPA with (1) near-human-level understanding of workflows (93% accuracy on a workflow understanding task) and (2) instant set-up with minimal technical barrier (based solely on a natural language description of a workflow, ECLAIR achieves end-to-end completion rates of 40%). We identify human-AI collaboration, validation, and self-improvement as open challenges, and suggest ways they can be solved with data management techniques. Code is available at: https://github.com/HazyResearch/eclair-agents

1. Introduction

Figure 1. Differences between ECLAIR and traditional RPA. ECLAIR uses FMs to learn expertise via video demonstrations (left), navigate GUIs given written documentation (center), and audit completed workflows (right).

Digital workflows are the core of our modern economy, with 92% of jobs now requiring digital skills (Bergson-Shilcock and Taylor, 2023). Many workflows can and should be automated — across industries, workers average 3 hrs/day doing repetitive digital workflows tangential to their core jobs (Anywhere, 2020), a phenomenon referred to as ”death by 1,000 clicks” (Schulte and Fry, 2019).

For example, consider a large B2B enterprise with tens of thousands of customers (full case study in Section 3.2). A key accounting workflow is ingesting customer contracts into a centralized database. After receiving a contract via email, an analyst must manually extract relevant data and enter it into an enterprise resource planning system (e.g. NetSuite, SAP, etc.) before they can conduct downstream analyses. Doing this at enterprise scale requires hundreds of individuals, each potentially taking 40 minutes per contract.

As illustrated above, most enterprise workflows involve data integration, ingestion, and transformation, and the market for process management is projected to reach $65 billion by 2032 (Fahland et al., 2024). Decades of data management work has gone into understanding and managing these workflows (Casati and Shan, 2000; Sayal et al., 2002; Zeng et al., 2001; Hull et al., 2013; Georgakopoulos et al., 1995; Jennings et al., 1998), yet the ultimate vision of end-to-end automation — from understanding to execution to monitoring — has remained elusive.

The current best-in-class solution is Robotic Process Automation (RPA), in which workflows identified via process mining get manually encoded into a fixed set of rules for a program to follow (Leno et al., 2021; Moreira et al., 2023; Muthusamy et al., 2023). Narrowly scoped RPA deployments can have ROIs of 30-200% and 2x the speed of workflows (Lhuer, 2016; da Costa et al., 2023). However, more widespread adoption of RPA has been limited by three key failure modes (Sarilo-Kankaanranta and Frank, 2021) which surfaced in interviews with technology leaders at a hospital and B2B enterprise (full case studies in Section 3):

• High set-up costs: The desired workflow must be demonstrated to the RPA bot. This is costly, as it requires a trained specialist to map workflows, write automation scripts, and integrate with IT infrastructure (Moreira et al., 2023; Perdana et al., 2023; Hull et al., 2013). A leading RPA vendor estimates that 3-6 months of experience is needed to become proficient in RPA (UIPath, 2022). In our B2B case study, it took 12 months to go from project kickoff to deployment.

• Brittle execution: The RPA bot must execute the workflow. Since RPA relies on hard-coded rules, bots cannot adapt to slight variations in input (e.g., a button changing location on a screen, or a form field being renamed) (Ye et al., 2023; Wewerka and Reichert, 2020; Chakraborti et al., 2020). This leads to a “death by a thousand cuts” as the space of possibilities is essentially unbounded. In our B2B case study, the RPA bot was initially only 60% accurate and took 6 months of improvement to reach 95%.

• Burdensome maintenance: RPA deployments often require human oversight to validate outputs and fix edge cases. In our B2B case study, the bot required continual monitoring by 2 full-time equivalents (FTEs).

The common cause behind these shortcomings is the difficulty of encoding ”tacit” (i.e. difficult to define) human workflow expertise into a rule-based system like RPA (Muthusamy et al., 2023; Autor, 2014; Brynjolfsson et al., 2023). Automating most workflows requires planning a sequence of actions and executing them on a graphical user interface (GUI). This requires (a) visual understanding, e.g. identifying a button on a screen; (b) real-time decision making, e.g. knowing to scroll to locate a form field that has shifted location; and (c) common sense to error correct, e.g. hitting escape when an irrelevant pop-up appears.

Multimodal foundation models (FMs) such as GPT-4 (OpenAI, 2023) have demonstrated visual understanding (Bavishi et al., 2023; Wang et al., 2023b; Zhang et al., 2023) and generalized reasoning abilities (Yao et al., 2022; Wei et al., 2022; Ahn et al., 2022; Wang et al., 2023c) for automating simple digital workflows (Zheng et al., 2024; Gur et al., 2023; Yang et al., 2023a; Yan et al., 2023; Zhang et al., 2024; Wu et al., 2024; Hong et al., 2023a). This offers the possibility of sidestepping the failure modes of traditional RPA, just as deep learning eclipsed rule-based approaches over the past decade in machine learning. We thus ask the question: Can multimodal foundation models automate enterprise workflows?

We take a first natural step in studying the opportunities and challenges of applying multimodal FMs across all three stages of traditional RPA by proposing ECLAIR – “Enterprise sCaLe AI for woRkflows”. As shown in Figure 1, our system is defined as follows:

1. Demonstrate: ECLAIR uses multimodal FMs to learn from human workflow expertise by watching video demonstrations and reading written documentation. This lowers set-up costs and technical barriers to entry. Initial experiments show that ECLAIR can identify every step of a workflow based on screenshots from a demonstration with 93% accuracy.

2. Execute: ECLAIR observes the state of the GUI and plans actions by leveraging the reasoning and visual understanding abilities of FMs (Yao et al., 2022; Bavishi et al., 2023; OpenAI, 2023). Based solely on written documentation of a workflow, ECLAIR improves end-to-end completion rates over an existing GPT-4 baseline from 0% to 40% on a sample of 30 web navigation tasks (Zhou et al., 2023). However, this is still far from the accuracy needed for enterprise settings, and we identify opportunities to close this gap.

3. Validate: ECLAIR utilizes FMs to self-monitor and error correct. This reduces the need for human oversight. When classifying whether a workflow was successfully completed, ECLAIR achieves a precision of 90% and recall of 84%.

Our initial evaluations also identify several patterned failure modes for future research. In Execute, ECLAIR has difficulty decomposing higher-level steps into discrete actions (e.g. breaking ”Search XXX” into the sequence of ”click”, ”type XXX”, ”press enter”) and grounding actions to specific GUI elements (e.g. differentiating two buttons with the same label). In Validate, ECLAIR’s lack of heuristics for navigating GUIs make step-level validation challenging (e.g. checking that a text field is first focused before typing).

While there is still progress to make, we are excited by the potential of ECLAIR to automate entirely new categories of workflows that require real-time decision-making, interaction with GUIs, and ”tacit” domain knowledge (Autor, 2014), as outlined in Figure 2. McKinsey estimates this could double the amount of knowledge work that can be automated (Chui et al., 2023).

The rest of the paper is structured as follows. In Section 2, we discuss related work on process mining, RPA, and applying FMs to workflow automation. In Section 3, we provide case studies of a hospital and large B2B enterprise which highlight the limitations of RPA. In Section 4, we outline how ECLAIR can address these shortcomings. We conclude in Section 5 with a discussion of future work and opportunities for the data management community.

Our contributions are: (1) two case studies highlighting the limitations of process mining / RPA, (2) a framework, ECLAIR, for achieving end-to-end enterprise workflow automation with multimodal FMs, (3) evaluations of ECLAIR on 30 workflows involving enterprise web applications, and (4) proposals for applying data management techniques to workflow automation.

2. Background

We survey related work on RPA, process mining, data management tools for workflow automation, and foundation models (FMs), then detail the problem setting that ECLAIR seeks to solve.

Figure 2. ECLAIR can automate entirely new categories of workflows, such as those that contain hard-to-describe steps, require complex decision making, or are knowledge intensive. Listed examples are real-world hospital workflows (see Section 3.1).

2.1. Related Work

Significant effort has gone into developing tools for understanding and automating workflows.

Foundation Models (FMs) are deep learning models trained on large datasets which can be adapted to a broad range of downstream tasks (Bommasani et al., 2021). They have demonstrated robust world knowledge (Safavi and Koutra, 2021; Jiang et al., 2020), reasoning (Yao et al., 2022; Wei et al., 2022), and planning abilities (Ahn et al., 2022; Shen et al., 2023; Di Palo et al., 2023; Wang et al., 2023c), and have achieved state-of-the-art results on data processing tasks such as integration and cleaning  (Narayan et al., 2022; Kayali et al., 2023).

Process Mining is the identification and improvement of workflows based on observational data (Van der Aalst, 2014; Reinkemeyer, 2020; Augusto et al., 2018). Recent works applied FMs to process mining tasks such as Petri net generation, workflow understanding, and process improvement (Fahland et al., 2024; Rizk et al., 2023; Dumas et al., 2023; Vidgof et al., 2023; Berti and Qafari, 2023; Grohs et al., 2023; Muthusamy et al., 2023), but were limited to small case studies and unimodal models.

Robotic Process Automation (RPA) is the leading approach for automating enterprise workflows (Dahabiyeh and Mowafi, 2023; Ivančić et al., 2019). In RPA, a human manually defines a set of rules that a bot then follows to accomplish a specific workflow (Ivančić et al., 2019; Chakraborti et al., 2020; Wewerka and Reichert, 2020). These fixed rulesets make RPA brittle (e.g. failing if a form changes the ordering of its fields) and difficult to maintain (Fernandez and Aman, 2021; Moreira et al., 2023). FMs offer a compelling alternative due to their robust reasoning capabilities (Yao et al., 2022; Ahn et al., 2022) and ability to navigate GUIs (Hong et al., 2023a). Initial FM-based approaches (Deng et al., 2023; Gur et al., 2023; Liu et al., 2023) utilized Large Language Models (LLMs) to act on websites. Since LLMs can only understand text, these works relied on scraping a webpage’s HTML as input to the model. This prevented their application to native desktop and virtualized software. Multimodal FMs address this limitation by attaching a vision model to the base LLM (Zhang et al., 2023), which enables them to directly reason over screenshots of a GUI (Humphreys et al., 2022; Furuta et al., 2023; Shaw et al., 2023; Hong et al., 2023a; Bavishi et al., 2023). Multimodal FMs have already shown promise in navigating websites (Zheng et al., 2024; Yang et al., 2023a; He et al., 2024), mobile apps (Yan et al., 2023), and desktop applications (Wu et al., 2024; Zhang et al., 2024). We aim to design a system that helps bridge the gap between these proof-of-concepts and enterprise-level solutions.

Data Management for Workflow Automation has been studied for nearly two decades, with works ranging from business process management (Zeng et al., 2001; Sayal et al., 2002; Georgakopoulos et al., 1995; Casati and Shan, 2000; Rahman et al., 2015) to workflow automation and understanding (Hull et al., 2013; Jennings et al., 1998). All of this work, however, pre-dates multimodal FMs. As a result, the challenges around which these systems were designed differ substantially from modern systems. Our work aims to build on these prior efforts from the data management community by developing a system that integrates FMs.

2.2. Problem Formulation

We aim to achieve end-to-end automation of enterprise workflows at minimal cost. We have a workflow $w\in\mathcal{W}$ which consists of a sequence of alternating states $s\in\mathcal{S}$ and actions $a\in\mathcal{A}$, such that $w=(s,a,s^{\prime},a^{\prime\prime},...)$. Each workflow $w$ is done by a set of workers during the course of business operations. These workers follow a standard operating procedure (”SOP”), a form of written documentation which outlines all of the steps and actions of the workflow. Once the workflows are executed, an auditing process validates (either manually or programmatically) whether each workflow $w$ was completed successfully. The goal is to automate workflow $w$ by learning from human demonstrations and written documentation (e.g. SOPs). We aim to directly operate on the GUI, as many workflows do not have APIs or must be executed in native desktop/virtualized applications that can only be observed visually (Hong et al., 2023a). This necessitates a vision-based approach (Zheng et al., 2024; Assouel et al., 2023; Zhang et al., 2024).

3. Case Studies

We conducted interviews with business leaders across a number of industries who led RPA projects. We select two organizations — a hospital and large B2B enterprise — to serve as case studies. Both took over a year to deploy RPA pilot projects, and both declined to expand RPA to additional workflows due to its high set-up costs ($100k’s and months of development), unreliable execution (accuracies started around 60% and peaked at 95%), and maintenance requirements. We detail these challenges below.

3.1. Hospital Revenue Cycle Management (RCM)

Given the complexity of healthcare reimbursement, most hospitals have a dedicated Revenue Cycle Management (RCM) department to ensure that timely payment is collected for services delivered (Kilanko, 2023). Example RCM workflows include verifying a patient’s insurance eligibility, obtaining prior authorization, and processing claims (Kilanko, 2023). It is estimated that RCM processes cost roughly 15% of every dollar of revenue gained (Bayley and Levine, 2013). Additionally, 90% of health systems face RCM staffing shortages (R1, 2022). Despite increased interest in automating RCM workflows, most remain highly manual: roughly 94% of claims submissions and 76% of eligibility verifications involve manual labor (Holloway et al., 2018). In conversations with hospital IT leaders who considered automating certain RCM workflows, but only managed a limited deployment, the following weaknesses of RPA surfaced:

(1) High Set-Up Costs. Integration with a hospital’s IT infrastructure is costly, as is retraining staff (Kilanko, 2023). Hospital leadership estimated that it took about 18 months and $10k’s to develop and deploy their RPA bot. Significant back-and-forth with the vendor occurred as each workflow had to be manually mapped and coded into a set of well-defined, ”always true” actions.

(2) Brittle Execution. It was estimated that $\sim$0.2 FTEs worth of effort was saved with the RPA bot, as it could only handle two narrowly-scoped workflows involving one payer and one department. Quarterly updates to the hospital’s electronic health record and constant changes to payers’ websites would break the bot. Eventually, this required the hospital to develop a custom API that the bot could use to increase its reliability.

(3) Burdensome Maintenance. Given staffing constraints, the hospital chose to outsource continued human oversight of the bot to the vendor as a managed service for a fee. Despite this outsourced management, RCM managers still had to manually review outputs to ensure compliance.

3.2. B2B Enterprise Invoice Processing

Large B2B enterprises must ingest a wide range of complex contracts and process the information into systems of record such as NetSuite or SAP. These workflows can vary widely depending on the specific product being invoiced, the end purchaser, and the time the contract was written (Sahu et al., 2020). Given its importance to enterprise financial planning, invoice processing has also attracted attention from RPA vendors. We interviewed the head of revenue operations at a large B2B enterprise which tried to implement RPA for a single invoice processing workflow. After a year of development, 5 FTEs working with the RPA bot were able to successfully accomplish a workflow that previously took 20 FTEs. Despite this apparent success, however, the enterprise chose not to expand RPA to other workflows, as the implementation proved to be too painful:

(1) High Set-Up Costs. Going from initial contract with the RPA vendor ($150k) to production deployment took over 12 months. External consultants (another $100k) and 3 FTEs were required to integrate the bot with existing IT systems. Major challenges were (a) accurately defining the workflow; and (b) the steep learning curve for programming RPA bots, which required learning a proprietary low-code development toolkit.

(2) Brittle Execution. The RPA bot was initially only 60% accurate post-deployment and took 6 months of iteration to reach a final accuracy of 95%. Beyond invoice processing, only $\sim$50% of desired workflows seemed feasible to automate with RPA.

(3) Burdensome Maintenance. The cost of a wrongly processed invoice is high ($10k’s), so 2 FTEs were allocated to continuously monitor the bot to debug issues, add new input formats, and inspect outputs. An outside firm also conducted manual batch reviews.

4. Preliminary Evaluations

In this section, we describe how ECLAIR can leverage multimodal FMs across all three stages of the traditional RPA pipeline.

We provide preliminary experiments assessing the feasibility of this approach. For our evaluations, we subsample 30 workflows from the WebArena benchmark (Zhou et al., 2023). WebArena provides a set of interactive websites in which an AI agent must complete complex workflows specified via natural language. Specifically, we choose 30 workflows from the Gitlab and Adobe Magento environments which their GPT-4 baseline model failed to complete (Gur et al., 2023). We have human annotators record themselves completing each workflow and write a step-by-step guide (”SOP”) on the steps they took.

4.1. Demonstrate

ECLAIR aims to learn from passively collected human demonstrations, with no updates to the underlying FM’s weights. This limits the cost of labeling data, simplifies deployment, and avoids known biases that arise when humans try to articulate their work processes (Li et al., 2019). Our experiments show that GPT-4 can accurately identify the steps of a workflow based on visual observation of a human demonstration, with step-level precision of 0.94 and recall of 0.95.

4.1.1. Can ECLAIR determine the steps of a workflow by viewing raw video demonstrations?

This would enable ECLAIR to substantially improve the effectiveness and scalability of process mining.

Hypothesis: A mulitmodal FM can generate accurate SOPs based on screenshots taken at key frames from a video recording.

Set-Up: We prompt GPT-4 to generate an SOP for a workflow given a human demonstration. We ablate various ways to provide the demonstration: just the workflow description (WD); the workflow description and screenshots of key frames from a video recording of the demonstration (WD+KF); or the workflow description, key frames, and a textual action log of each click and keystroke (WD+KF+ACT). Using a manually-written SOP as reference, a human annotator calculates GPT-4’s precision (”What percent of steps in the GPT-4 SOP are in the true SOP?”), recall (”What percent of steps in the true SOP are in the GPT-4 SOP?”), and correctness (”By following the GPT-4 SOP, can I complete the workflow?”).

Table 1. (Demonstrate) GPT-4 generation of SOPs. Metrics averaged across all 30 workflows.

Method	# of Steps in SOP			Accuracy of SOP
	Missing	Incorrect	Total	Precision	Recall	Correctness
WD	1.57	3.58	13.67	0.75	0.81	0.60
WD+KF	0.67	1.05	10.17	0.89	0.92	0.90
WD+KF+ACT	0.63	0.57	9.63	0.94	0.95	0.93
Ground truth	0	0	8.70	1	1	1

Takeaways: As shown in Table 1, the SOPs generated by GPT-4 using the WD+KF+ACT strategy are judged as sufficiently ”correct” to complete 93% of workflows. Even providing GPT-4 with screenshots alone (WD+KF) achieves 90% correctness. However, WD+KF experiences almost twice as many hallucinations (1.05 incorrect steps per SOP versus only 0.57 when the action trace is included). Note that we do not do any workflow-specific prompt engineering or data labeling to achieve these results. Additionally, we preprocess our video demonstrations into a sequence of key frames using imperfect heuristics (i.e. alignment with clicks and keystrokes). Future work may benefit from using a model that can directly process video (rather than just images).

4.2. Execute

After defining the workflow, ECLAIR must execute a sequence of steps to accomplish the workflow. We divide each step into two phases: (1) action suggestion, i.e. planning what action to take; (2) action grounding, i.e. translating the plan into actual clicks/keystrokes of GUI elements. We find that providing domain knowledge to ECLAIR via SOPs doubles workflow completion rates, while general-purpose models such as GPT-4 lag smaller models fine-tuned on GUIs for action grounding.

4.2.1. Can ECLAIR accurately suggest the next action to take in a workflow?

We investigate whether multimodal FMs can predict the next action in a workflow based on the current state of the GUI and action history. We evaluate if providing the FM with an SOP for the workflow increases overall workflow completion rates.

Hypothesis: Providing high-level natural language guidance to the model via an SOP improves workflow completion rates.

Set-Up: At each step of the workflow, the model takes as input the ground truth history of actions, full SOP, and current GUI, and is expected to generate the next action to take. We measure the accuracy of the model’s suggested next action using a human annotator to evaluate whether it is semantically equivalent to the corresponding ”ground truth” action for the workflow.

Table 2. (Execute) GPT-4 average accuracy on next action suggestion with and without SOP guidance.

SOP	Next Action Suggestion Acc.	Overall Workflow Completion Acc.
✘	0.83	0.17
✔	0.92	0.40

Takeaways: Our results in Table 2 demonstrate that SOPs improve overall workflow completion rates by up to 23 points. While the SOP boosts the accuracy of each step’s action suggestion to 0.92, the model struggles to associate these suggested actions with the appropriate GUI elements, thereby deflating its overall completion rate (recall that the model must ground its actions to successfully complete the workflow). For example, when the action is “Click on the profile button”, but the HTML element for the icon is identified as ”svg” rather than ”button”, the model fails to select this element.

4.2.2. Can ECLAIR accurately ”ground” its actions suggestions to GUI elements?

Once ECLAIR suggests an action to take (i.e. “Click on the ”Submit” button”), it must then map the action into mouse/keyboard commands which specify the pixel location of the GUI element to interact with (Zheng et al., 2024). Multimodal FMs are known to have difficulty with this, so we evaluate several grounding strategies (Zheng et al., 2024).

Hypothesis: Providing bounding boxes for each element in a GUI improves a multimodal FM’s ability to ground elements.

Set-Up: We sample a total of 120 and 302 webpages from the WebUI (Wu et al., 2023b) and Mind2Web (Deng et al., 2023) datasets, respectively. We create a natural language description for one element on each page, then prompt a model to generate a bounding box (BB) for that element given a natural language description of the BB and screenshot of the webpage. We measure ”accuracy” as the percentage of predicted BBs whose center is within the element’s true BB (i.e. If the model clicked on the center of its prediction, would it successfully hit the target element?). We evaluate GPT-4 (OpenAI, 2023) and CogAgent (Hong et al., 2023a) as two state-of-the-art closed/open source multimodal FMs for GUI navigation. While CogAgent directly outputs BBs, GPT-4 does not. Thus, we use ”set-of-marks” prompting for GPT-4, in which we overlay a unique numeric label on top of every element in the webpage screenshot provided to GPT-4, and have it output the number of a labeled element (Yang et al., 2023b). We generate these labels either directly from the webpage’s HTML (”HTML”) or from a YOLONAS object detection model (”YOLO”) finetuned on 7k WebUI webpages (Wu et al., 2023b). The latter simulates the setting where HTML is not available.

Table 3. (Execute) Accuracy on grounding actions to GUI elements. ”S — M — L” is accuracy on small, medium, and large elements. Note: We found that HTML bounding boxes were not accurate in Mind2Web, so it is excluded.

Model	Bbox Source	Mind2Web		WebUI
		S — M — L	Overall	S — M — L	Overall
GPT-4	–	0.01 — 0.03 — 0.16	0.07	0.00 — 0.03 — 0.14	0.05
GPT-4	YOLO	0.38 — 0.68 — 0.80	0.62	0.50 — 0.58 — 0.69	0.58
GPT-4	HTML	–	–	0.56 — 0.58 — 0.67	0.60
CogAgent	–	0.55 — 0.71 — 0.87	0.71	0.71 — 0.64 — 0.75	0.70

Takeaways: The 18-billion parameter CogAgent outperforms GPT-4 on action grounding. This suggests that smaller models purpose-built for GUI navigation can be more effective than larger, general purpose multimodal FMs. However, overall accuracy peaks at 70%, and interacting with smaller-sized elements remains challenging. For GPT-4, using the bounding boxes generated by the YOLONAS model performs similarly to the ”ground truth” HTML boxes. This suggests that detecting elements on a GUI with a vision model is not the bottleneck, but rather choosing which of those detected elements is the desired element to interact with.

4.3. Validate

There are several levels of validation that ECLAIR must provide. At the individual step-level, the agent should validate that its action suggestions are (a) feasible to execute and (b) making progress towards accomplishing the workflow. At the workflow-level, the agent should understand (a) whether the workflow was successfully completed and (b) whether the steps taken to achieve the workflow were sensible. We find that GPT-4 struggles with the former lower-level details but performs well at the latter higher-level reasoning.

4.3.1. Can ECLAIR self-monitor at the individual step-level?

Identifying errors in individual actions would allow ECLAIR to perform error correction in real-time. This would improve the reliability of the Execution stage by enabling the model to avoid undesirable states and backtrack where appropriate.

Hypothesis: A multimodal FM can detect if an action will succeed or fail based on visual observation of changes in screen state.

Set-Up: First, we test the model’s ability to detect if an action failed (e.g. typing had no effect because no text field was first focused). We sample $(s,a,s^{\prime})$ traces from our dataset (positive examples) and generate tuples where $s^{\prime}=s$ (negatives). We prompt GPT-4 to identify if the action $a$ in $(s,a,s^{\prime})$ was successfully executed, given screen shots for $s$ and $s^{\prime}$. We sample three negatives for each positive example. Second, inspired by prior work on data cleaning (Benedikt et al., 2015; Baltopoulos et al., 2011), we create a set of ”integrity constraints” defining whether an action is viable at a particular state. For example, an ”integrity constraint” for clicking a button is that the button is visible and not disabled. We annotate constraints for all actions in our dataset, then prompt GPT-4 with $(c,s)$ pairs where $c$ is the constraint for the action directly after state $s$ (positive examples) and $(c,s^{\prime})$ pairs where $s^{\prime}$ is a random state occurring before $s$ (negatives).

Table 4. (Validate) Performance of GPT-4 on self-validation tasks. Metrics averaged across all 30 workflows.

Eval Type	Precision	Recall	F1
Actuation	0.95	0.85	0.90
Integrity Constraint	0.67	0.36	0.47
Workflow Completion	0.90	0.84	0.87
Workflow Trajectory	0.88	0.83	0.85

Takeaways: The results are shown in rows ”Actuation” and ”Integrity Constraint” in Table 4. GPT-4 has a high precision (0.95) and recall (0.85) when assessing whether an action was successfully executed. However, the low integrity constraint scores indicate that it struggles to identify which actions are viable given the state of the GUI. This could be due to only observing static screenshots, which makes it difficult to discern animations such as a blinking cursor. These results suggest that current multimodal FMs cannot adequately self-monitor for enterprise use cases.

4.3.2. Can multimodal FMs self-monitor at the overall workflow-level?

Understanding whether a workflow was successfully completed or not can help the agent (a) at runtime to know when to stop execution, and (b) post-deployment for self-auditing.

Hypothesis: A multimodal FM can determine if a demonstration correctly achieved a workflow based on visual observation of changes in screen state.

Set-Up: First, we test if the model can determine if it successfully completed a workflow. To evaluate this, we sample full traces of $(s^{1},a^{1},...,a^{n-1},s^{n})$ from our dataset (positive examples) and truncate some by a random number of frames to get $(s^{1},a^{1},...,a^{k-1},s^{k})$ (negatives) where $k<n$. Given the trace and workflow description, we prompt GPT-4 to provide a binary assessment of whether the workflow was successfully completed. Second, we investigate whether the model understands the proper trajectory of actions in a workflow — i.e. it is not sufficient to merely complete the workflow, but the steps taken to complete it must align with its SOP. We sample full traces $(s^{1},a^{1},...,a^{n-1},s^{n})$ from our dataset (positives) and either (a) randomly shuffle or (b) randomly delete frames from this trace (negatives). We provide GPT-4 with the SOP for the workflow, the workflow description, and the trace, and have it output a binary assessment of whether the trace exactly followed the SOP.

Takeaways: The F1 scores of 0.87 and 0.85 in rows ”Workflow Completion” and ”Workflow Trajectory” of Table 4 suggest that GPT-4 can self-monitor higher-level properties of a workflow, but still has significant room for improvement.

5. Discussion

To be deployed, ECLAIR must meet a minimum level of performance. While this is highly workflow-dependent, ECLAIR must be accurate enough such that the cost of correcting its errors is outweighed by the efficiency gains of using it (which might not require 100% accuracy). In our Section 3.2 case study, for example, an accuracy of 95% was sufficient for the RPA bot to be deployed. Defining such success criteria can be done via interviews with stakeholders, financial modeling, and other enterprise planning methods (Leno et al., 2021) We discuss considerations for deploying ECLAIR below.

Error Handling and Monitoring. We envision a multi-tiered system which combines (1) self-validation, (2) programmatic heuristics, and (3) limited human intervention to provide error correction. (1) Per our Section 4.3 experiments, a repository of integrity constraints — which have successfully enhanced the quality of database schemas (Benedikt et al., 2015; Baltopoulos et al., 2011) — could improve the accuracy of FM self-monitoring. Though FMs can exhibit non-deterministic behavior, their reliability can be improved by setting their temperature to 0, repeatedly querying (Shinn et al., 2023) and ensembling predictions (Li et al., 2024), or eliciting confidence scores to surface cases where intervention is necessary (Tian et al., 2023). (2) Programmatic heuristics can also be used to detect failures, e.g. deviation from the average time to execute a workflow. (3) Existing methods for auditing human workflows can be re-applied, such as random spot checks or screenshots of confirmation screens. However, we envision such monitoring to be more limited – namely, to close the knowledge gap between ECLAIR and a particular domain – as the outputs from monitoring can be repurposed into a dataset that can be used to improve ECLAIR (Ouyang et al., 2022).

Human-ECLAIR Collaboration. While the long-term vision for ECLAIR is to require minimal human interaction — i.e. only the Demonstrate phase (Section 4.1) needing human involvement — we acknowledge that human supervision may be necessary for certain workflows. For example, a physician sign-off before prescribing medications or tasks involving user authentication. To accomplish this, the SOP could mark steps where the model transfers control to a human. Alternatively, a whitelist of sensitive actions can be compiled to automatically force transfer of control to a human when triggered, similar to how kernels use interrupts to handle control flow (Mejia-Alvarez et al., 2018). Finally, as mentioned in the prior section, generated human-ECLAIR execution traces can be used to improve ECLAIR performance via fine-tuning or few-shot prompting (Ouyang et al., 2022).

Self-Improvement. As ECLAIR repeatedly executes a workflow, it can observe the effects of its actions on the environment. By documenting these observations, ECLAIR can compile a database of common ”skills” that can later be transferred to different workflows (Fu et al., 2024; Yang et al., 2023a; Wu et al., 2024; Wang et al., 2023a; Park et al., 2023). Applying principles from self-driving databases, which aim to continuously improve performance by implementing sequences of ”actions” (i.e. changes to their configurations) based on utilization patterns (Pavlo et al., 2021, 2017; Ma et al., 2018), could provide a principled approach for such a self-improving workflow automation system.

Multi-Agent Collaboration. Applying multiple agents to the same task can improve accuracy (Li et al., 2024; Liang et al., 2023), as seen in recent work on multi-agent software engineering (Hong et al., 2023b) and chatbot applications(Wu et al., 2023a). Such an approach could be utilized within the ECLAIR framework to create specialized agents for distinct subtasks or digital environments. Prior work on collaborative data processing tools offers a reference for how ECLAIR can be scaled to multi-agent (and multi-human) workflows with shared resources. (Bhardwaj et al., 2015; Liu et al., 2022).

6. Conclusion

We are excited for the potential of multimodal FMs to reimagine how work gets done. By addressing the three main shortcomings of traditional process mining and RPA (high set-up costs, brittle execution, and burdensome maintenance), the realization of ECLAIR can help achieve the promise of enterprise workflow automation.