Toloka-Kit Python Library

Toloka-kit provides a wide range of practical examples categorized by domain to help you implement various crowdsourcing tasks. You can find specialized notebooks for:

• Computer Vision (CV): Image collection, classification, object detection (bounding boxes), HTR (handwriting) gathering, video collection, and text recognition (OCR).
• Natural Language Processing (NLP): Text classification (e.g., clickbait detection), Question Answering (SQuAD), sentiment analysis, and intent classification.
• Audio Analysis: Audio collection (TTS/ASR), audio classification (gender detection), and audio transcription.
• Ranking: Side-by-side image comparison and preference aggregation.
• Spatial Crowdsourcing: Collecting location-based data (e.g., monitoring physical objects).
• Surveys: Simple survey implementation for data collection.
• Pipelines: Integrating Toloka with ML models using Prefect or building streaming pipelines.
• Relevance: Search and ad relevance rating.
• Benchmarks: Pre-configured setups for standard datasets like CINIC-10 (CV) and IMDB (NLP).
• Metrics: Visualizing data via Jupyter dashboards or sending metrics to Graphite.

Streaming allows you to react to events in real-time. For example, you can automatically create verification tasks as soon as a detection task is submitted. This is done using AssignmentsObserver and a Pipeline.

1. Define a handler (e.g., DetectionSubmittedHandler) that implements __call__ to process AssignmentEvents.
2. Register an AssignmentsObserver for each pool.
3. Run the Pipeline with a Storage backend (like JSONLocalStorage) to ensure progress is saved.

from toloka.streaming import AssignmentsObserver, Pipeline
from toloka.streaming.storage import JSONLocalStorage

# 1. Define handler
class DetectionSubmittedHandler:
    def __init__(self, client, verification_pool_id):
        self.client = client
        self.verification_pool_id = verification_pool_id

    def __call__(self, events: List[AssignmentEvent]) -> None:
        # Logic to create verification tasks from detection results
        ...

# 2. Setup Pipeline
detection_observer = AssignmentsObserver(toloka_client, detection_pool.id)
detection_observer.on_submitted(DetectionSubmittedHandler(toloka_client, verification_pool.id))

storage = JSONLocalStorage('./storage/')
pipeline = Pipeline(storage=storage)
pipeline.register(detection_observer)

# 3. Run
await pipeline.run()

A project defines the task structure and the visual interface. You can use the template_builder (aliased as tb) to construct complex layouts like SideBySideLayoutV1.

Steps to create a project:

1. Define the main_interface using layout components (e.g., ImageViewV1) and controls (e.g., ButtonRadioGroupFieldV1).
2. Add plugins like HotkeysPluginV1 to improve performer efficiency.
3. Define the input_specification and output_specification using field types like UrlSpec() and StringSpec().
4. Wrap the configuration in a TemplateBuilderViewSpec.
5. Assign the task_spec to the project and call toloka_client.create_project(project).

import toloka.client.project.template_builder as tb
import toloka.client as toloka

# 1. Define Interface
main_interface = tb.SideBySideLayoutV1(
    items=[
        tb.ImageViewV1(url=tb.InputData(path='image_left'), full_height=True),
        tb.ImageViewV1(url=tb.InputData(path='image_right'), full_height=True),
    ],
    controls=tb.ButtonRadioGroupFieldV1(
        data=tb.OutputData(path='result'),
        label='Which icon do you like more?',
        options=[
            tb.GroupFieldOption(label='Left', value='LEFT'),
            tb.GroupFieldOption(label='Right', value='RIGHT'),
            tb.GroupFieldOption(label='Loading error', value='ERROR'),
        ]
    )
)

# 2. Add Plugins
hot_keys_plugin = tb.HotkeysPluginV1(
    key_0=tb.SetActionV1(data=tb.OutputData(path='result'), payload='ERROR'),
    key_1=tb.SetActionV1(data=tb.OutputData(path='result'), payload='LEFT'),
    key_2=tb.SetActionV1(data=tb.OutputData(path='result'), payload='RIGHT'),
)

# 3. Build View Spec
project_interface = toloka.project.view_spec.TemplateBuilderViewSpec(
    config=tb.TemplateBuilder(
        view=main_interface,
        plugins=[hot_keys_plugin],
    )
)

# 4. Set Data Specs and Task Spec
input_specification = {
    'image_left': toloka.project.field_spec.UrlSpec(),
    'image_right': toloka.project.field_spec.UrlSpec(),
}
output_specification = {'result': toloka.project.field_spec.StringSpec()}

project = toloka.Project(
    assignments_issuing_type='AUTOMATED',
    public_name='Which icon do you like more?',
    public_description='Look at the icons and decide which one you like more.',
)

project.task_spec = toloka.project.task_spec.TaskSpec(
    input_spec=input_specification,
    output_spec=output_specification,
    view_spec=project_interface,
)

# 5. Create on Toloka
project = toloka_client.create_project(project)

You can add automated actions to restrict performers based on specific conditions. Common collectors include:

• Income: Check performer's income.
• SkippedInRowAssignments: Check if they skip too many tasks in a row.
• MajorityVote: Check if their answers deviate from the majority.
• GoldenSet: Check accuracy against known correct answers (Control tasks).

Actions are typically toloka.actions.RestrictionV2 which can restrict a performer from a scope='PROJECT' for a specific duration.

# Example: Restrict if Golden Set accuracy is too low
pool.quality_control.add_action(
    collector=toloka.collectors.GoldenSet(),
    conditions=[
        toloka.conditions.GoldenSetCorrectAnswersRate < 60.0,
        toloka.conditions.GoldenSetAnswersCount >= 3
    ],
    action=toloka.actions.RestrictionV2(
        scope='PROJECT',
        duration=10,
        duration_unit='DAYS',
        private_comment='Golden set'
    )
)

Tasks are the individual units of work.

1. Regular Tasks: Use toloka.Task(pool_id=pool.id, input_values={...}).
2. Golden Set (Control Tasks): These are tasks where the answer is known. Use known_solutions to provide the ground truth and set infinite_overlap=True so they can be used repeatedly to check quality.
3. Upload: Use toloka_client.create_tasks(list_of_tasks, allow_defaults=True) to upload them to the pool.

# Create Control Tasks (Golden Set)
golden_tasks = [
    toloka.Task(
        pool_id=pool.id,
        input_values={'image': row['url']},
        known_solutions=[
            toloka.task.BaseTask.KnownSolution(output_values={'result': row['label']})
        ],
        infinite_overlap=True,
    )
    for i, row in golden_dataset.iterrows()
]

# Create Regular Tasks
tasks = [
    toloka.Task(pool_id=pool.id, input_values={'image': url})
    for url in task_dataset['url']
]

# Upload
created_tasks = toloka_client.create_tasks(golden_tasks + tasks, allow_defaults=True)