Run consensus workflows

In this guide we'll walk through how you can use a combination of workflow stages, AI or Python properties, and user assignment in V7 Go to run the following consensus workflows:

• Human vs. Human
• Human vs. Model
• Model vs. Model
• RLHF-enhanced Model Training

🚧

User properties

Human vs. human or human vs. model consensus workflows require the User property type. This feature is available on V7's Enterprise tier and is currently in beta.

If you're an Enterprise user, reach out to your CSM to have it enabled.

Create parallel stages

To create two or more sets of feedback in parallel, add a separate workflow stage for each pass that your data requires.

To do this:

1. Add two or more Single select properties.
2. Set Tool to User input

3. Add an option that will send the file to its own distinct stage

4. Star the option so all new files enter this stage by default

5. Repeat the process with an option that sends completed entities to a stage where entities can be reviewed/compared.

Configure stages

Click on the relevant workflow stage at the bottom of the project page.

Add new properties or select from existing properties in other stages to build out the stage. Select the 3 dots to the right of the rightmost property to make a property read-only or write.

In a workflow where multiple users/models will be providing feedback on the same set of prompts, set up a unique property for each view (for instance, Stage 1 - Question 1, Stage 1 - Question 2 in Stage 1 and Stage 2 - Question 1, Stage 2 - Question 2 in Stage 2).

Assign users

For human vs. human or human vs. model workflows, you can add a User property to a specific stage in order to assign all entities within the stage to a user.

To set up a User property:

1. Create a new User property
2. Set the users that will be assigned to the property

3. (Optional) Star a user to make them the default assignee for all new properties

📘

Worker permissions

If you set up users to have Worker permissions, they will only have access to the data to which they are assigned within a specific stage.

Admin users will have all access to data within a project.

Compare results and mark as "Complete"

Once files have been completed across the parallel stages in a project, create a validation step which checks for agreement.

If you're comparing the results of a property with single or multi-select options, you can use the Python tool to check for agreement between two or more properties. Here's an example of a simple snippet which compares two single-select properties called "Urgency" and uses a single select to mark them with "Agreement" or "Disagreement."

if @Rater 1 Urgency == @Rater 2 Urgency:
    result = ['Agreement']
else:
    result = ['Disagreement']

When comparing free-text responses, use an AI tool in a single-select property to do something similar, prompting it to compare two or more properties, and mark them with "Agreement" or "Disagreement."