Apify Agent Skills

repository·main·Indexed 25 days ago

https://github.com/apify/agent-skills

Production-grade web scraping and automation capabilities for AI coding agents such as Claude Code, Cursor, and Windsurf. It provides specialized skills including apify-ultimate-scraper for universal scraping, apify-actor-development for creating and deploying Actors in JS/TS or Python, apify-actorization for converting code into Actors, and tools for schema generation and SDK integration.

Tokens
28.2K
Snippets
64
Records
130
Agent score
80%

What's inside apify-agent-skills

  1. Available Apify Agent Skills

    main

    The repository provides several specialized skills for different automation and development tasks:

    SkillDescription
    apify-ultimate-scraperAI-powered universal scraper covering 130+ curated Actors (Instagram, Google Maps, Amazon, etc.) with fallback to the full Apify Store.
    apify-actor-developmentTools to create, debug, and deploy Apify Actors in JavaScript, TypeScript, or Python.
    apify-actorizationConverts existing code into Apify Actors using JS/TS SDK, Python async context manager, or a generic CLI wrapper.
    apify-generate-output-schemaAnalyzes Actor source code to generate dataset_schema.json, output_schema.json, and key_value_store_schema.json.
    apify-sdk-integrationIntegrates Apify into existing apps via apify-client (JS/TS/Python) or the REST API.
    apify-actor-commandsA pack adding slash commands like /create-actor for guided scaffolding.
  2. Generate Actor output schemas

    main

    Use the apify-generate-output-schema skill to create or update Apify Actor output schema files (dataset_schema.json, output_schema.json, and key_value_store_schema.json) by analyzing the Actor's source code. These schemas tell the Apify Console how to display run results.

    Core Principles

    • Analyze code first: Derive schemas from actual data-pushing calls (e.g., Actor.pushData) or existing type definitions (TypeScript interfaces, Python Pydantic models/dataclasses) rather than guessing.
    • Every field is nullable: Set "nullable": true on every field because external APIs and websites are unpredictable.
    • Anonymize examples: Use generic values (e.g., "exampleuser", "Example Channel") instead of real user data.
    • Reuse patterns: Match the existing repository's description style, naming conventions (camelCase vs snake_case), and JSON formatting.
  3. What is Standby mode and when to use it

    main

    Standby mode allows Actors to function as API servers that remain ready in the background to handle HTTP requests.

    Use Standby mode when your Actor needs to handle interactive, real-time HTTP requests, such as:

    • API endpoints
    • Webhook receivers
    • Real-time data lookups
    • MCP servers
    • Scraping APIs serving on-demand single-URL requests

    To enable this, set usesStandbyMode: true in your .actor/actor.json file and implement an HTTP server within your Actor code.

  4. Define the Actor output schema structure

    main

    The Actor output schema specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these definitions to display run results. The schema follows a JSON structure containing actorOutputSchemaVersion, a title, and a properties object where you define your specific outputs.

    {
        "actorOutputSchemaVersion": 1,
        "title": "<OUTPUT-SCHEMA-TITLE>",
        "properties": {
            /* define your outputs here */
        }
    }
  5. Manage state with Request Queue for pausable tasks

    main

    The Request Queue can be used for any task processing (not just web scraping) to enable pausable task execution. For non-URL tasks, use a dummy URL combined with a uniqueKey for deduplication and userData to carry your actual task payload.

    const requestQueue = await Actor.openRequestQueue();
    
    // Add tasks to the queue (works for any processing, not just URLs)
    await requestQueue.addRequest({
        url: 'https://placeholder.local',  // Dummy URL for non-scraping tasks
        uniqueKey: `task-${taskId}`,       // Unique identifier for deduplication
        userData: { itemId: 123, action: 'process' },  // Your custom task data
    });
    
    // Process tasks from the queue (with Crawlee)
    const crawler = new BasicCrawler({
        requestQueue,
        requestHandler: async ({ request }) => {
            const { itemId, action } = request.userData;
            // Process your task using userData
            await processTask(itemId, action);
        },
    });
    await crawler.run();
    
    // Or manually consume without Crawlee:
    let request;
    while (request = await requestQueue.fetchNextRequest()) {
        await processTask(request.userData);
        await requestQueue.markRequestHandled(request);
    }
  6. Convert existing projects into Apify Actors

    main
    Actorization is the process of converting existing software (JavaScript/TypeScript, Python, or CLI tools) into reusable serverless applications called Actors. Actors are packaged as Docker images that accept JSON input and produce structured JSON output. Use this skill when migrating code to Apify, wrapping CLI tools, or adding the Apify SDK to existing projects.
  7. Generate `dataset_schema.json` structure and rules

    main

    The dataset_schema.json file defines the schema for the Actor's dataset and how it is displayed in the Apify Console.

    File Structure

    {
        "actorSpecification": 1,
        "fields": {
            "$schema": "http://json-schema.org/draft-07/schema#",
            "type": "object",
            "properties": {
                // ALL output fields must be listed here
            },
            "required": [],
            "additionalProperties": true
        },
        "views": {
            "overview": {
                "title": "Overview",
                "description": "Most important fields at a glance",
                "transformation": {
                    "fields": [
                        // 8-12 most important field names
                    ]
                },
                "display": {
                    "component": "table",
                    "properties": {
                        // Display config for each overview field
                    }
                }
            }
        }
    }

    Hard Rules (Mandatory)

    • Complete Superset: The fields.properties object must contain every field the Actor can output. The views section only selects a subset for display.
    • Nullability: Every field must have "nullable": true.
    • Required Fields: The "required": [] array must always be empty.
    • Additional Properties: You must set "additionalProperties": true on both the top-level fields object and on every nested object within properties.
    • Type Requirement: Every field with "nullable": true must also have a defined "type" (AJV validation requirement).
    {
        "actorSpecification": 1,
        "fields": {
            "$schema": "http://json-schema.org/draft-07/schema#",
            "type": "object",
            "properties": {
                // ALL output fields here — every field the Actor can produce,
                // not just the ones shown in the overview view
            },
            "required": [],
            "additionalProperties": true
        },
        "views": {
            "overview": {
                "title": "Overview",
                "description": "Most important fields at a glance",
                "transformation": {
                    "fields": [
                        // 8-12 most important field names
                    ]
                },
                "display": {
                    "component": "table",
                    "properties": {
                        // Display config for each overview field
                    }
                }
            }
        }
    }
  8. Difference between Standby mode and Container Web Server

    main

    It is important to distinguish between these two types of web servers:

    • Container web server (ACTOR_WEB_SERVER_URL): Provides a per-run unique URL. It has no load balancing and no auto-scaling. It is primarily useful for viewing live UIs during a specific run.
    • Standby mode (ACTOR_STANDBY_URL): Provides a stable hostname that is load-balanced across runs and auto-scaled based on traffic. This should be used for production APIs.
  9. Apify Actor project structure

    main

    A standard Apify Actor project follows this directory and file structure:

    .actor/
    ├── actor.json           # Actor config: name, version, env vars, runtime
    ├── input_schema.json    # Input validation & Console form definition
    └── output_schema.json   # Output storage and display templates
    src/
    └── main.js/ts/py       # Actor entry point
    storage/                # Local-only storage (NOT synced to Apify Console)
    ├── datasets/           # Output items (JSON objects)
    ├── key_value_stores/   # Files, config, INPUT
    └── request_queues/     # Pending crawl requests
    Dockerfile              # Container image definition
  10. Define a Key-Value Store schema

    main

    The Key-Value Store schema organizes keys into logical groups called collections for easier data management and UI organization. You define these in a JSON file (e.g., .actor/key_value_store_schema.json).

    Each collection can be identified by a specific key or a keyPrefix. Collections can also enforce contentTypes or validate JSON data using a jsonSchema.

    {
        "actorKeyValueStoreSchemaVersion": 1,
        "title": "Key-Value Store Schema",
        "collections": {
            "documents": {
                "title": "Documents",
                "description": "Text documents stored by the Actor",
                "keyPrefix": "document-"
            },
            "images": {
                "title": "Images",
                "description": "Images stored by the Actor",
                "keyPrefix": "image-",
                "contentTypes": ["image/jpeg"]
            }
        }
    }