FSCrawler Documentation

Semantic search improves results by using vector embeddings. It requires Elasticsearch 8.17.0+ and a trial or enterprise license.

• Activation: Enabled by default if requirements are met. Disable via elasticsearch.semantic_search: false.
• Mechanism: When active, a content_semantic field of type semantic_text is created. It uses an inference API (defaulting to the Elser model) to extract semantic information.
• Custom Models: You can change the model by defining your own fscrawler_INDEX_mapping_content_semantic component template with a specific inference_id.

# Example: Using multilingual-e5-small for semantic search
PUT _component_template/fscrawler_fscrawler_mapping_content_semantic
{
 "template": {
   "mappings": {
     "properties": {
       "content": {
         "type": "text",
         "copy_to": "content_semantic"
       },
       "content_semantic": {
         "type": "semantic_text",
         "inference_id": ".multilingual-e5-small-elasticsearch"
       }
     }
   }
 }
}

FSCrawler uses Elasticsearch Component Templates to define mappings and settings.

Template Management Behavior

• Default: FSCrawler checks if templates exist. If they do, it skips management to preserve your custom templates.
• Disable management: Set elasticsearch.push_templates: false to prevent FSCrawler from managing templates.
• Force overwrite: Set elasticsearch.force_push_templates: true to overwrite all existing templates with FSCrawler's defaults.

Customizing Mappings

To use custom analyzers or mappings, create the specific component template before starting FSCrawler. FSCrawler will detect the existing template and skip it, allowing you to inject your own logic (e.g., a custom French analyzer) while it fills in the remaining required templates.

# Example: Define a custom content mapping before starting FSCrawler
PUT _component_template/fscrawler_fscrawler_mapping_content
{
 "template": {
   "mappings": {
     "properties": {
       "content": {
         "type": "text",
         "analyzer": "french"
       }
     }
   }
 }
}

If you want to deduplicate based on the extracted text (content) rather than the binary file, use the Elasticsearch fingerprint ingest processor.

1. Create the Elasticsearch Ingest Pipeline: This pipeline computes a fingerprint of the content field, stores it in a temporary field, sets it as the _id, and then removes the temporary field.

PUT _ingest/pipeline/content-fingerprint-id
{
  "description": "Compute a fingerprint from content and set it as the document _id",
  "processors": [
    {
      "fingerprint": {
        "fields": ["content"],
        "target_field": "_tmp_fingerprint",
        "method": "SHA-256"
      }
    },
    {
      "set": {
        "field": "_id",
        "value": "{{{_tmp_fingerprint}}}"
      }
    },
    {
      "remove": {
        "field": "_tmp_fingerprint",
        "ignore_missing": true
      }
    }
  ]
}

2. Configure the FSCrawler job: Set elasticsearch.pipeline: "content-fingerprint-id" in your job settings.

Important Caveats:

• Overwrite behavior: With bulk_operation: index (default), the last writer wins. With create, the first writer wins.
• Text vs Binary: This method treats files with different binary content but identical extracted text as duplicates. It will overwrite documents that have no extracted text.
• Folder documents: Documents representing folders are not sent through the ingest pipeline and cannot use this method.

FSCrawler generates documents in Elasticsearch with several structured fields depending on your configuration. The main fields include:

• content: The extracted text content.
• content_semantic: A semantic-text copy of the content (enabled when semantic search is configured).
• attachment: A BASE64-encoded binary file (enabled when fs.base64 is set).
• meta.*: Metadata extracted by Apache Tika (e.g., author, title, date, language).
• file.*: File attributes such as filename, extension, filesize, and url.
• path.*: Path information including real, virtual, and root paths.
• attributes.*: Filesystem-level attributes like owner, group, and permissions.
• external: Additional tags provided via external metadata.

{
    "content":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
    "content_semantic":"This is a sample text available in page 1\n\nThis second part of the text is in Page 2\n\n",
    "file":{
       "content_type":"application/vnd.oasis.opendocument.text",
       "created":"2018-07-30T11:35:08.000+0000",
       "extension":"odt",
       "filename":"test.odt",
       "filesize":6236,
       "indexing_date":"2018-07-30T11:35:19.781+0000",
       "last_accessed":"2018-07-30T11:35:08.000+0000",
       "last_modified":"2018-07-30T11:35:08.000+0000",
       "url":"file:///tmp/test.odt"
    },
    "meta":{
       "author":"David Pilato",
       "created":"2016-07-07T16:37:00.000+0000",
       "date":"2016-07-07T16:37:00.000+0000",
       "description":"Comments",
       "keywords":[
          "keyword1",
          "  keyword2"
       ],
       "language":"en",
       "title":"Test Tika title"
    },
    "path":{
       "real":"/tmp/test.odt",
       "root":"7537e4fb47e553f110a1ec312c2537c0",
       "virtual":"/test.odt"
    }
 }

To index only one copy of duplicate files based on their binary content, use an Elasticsearch ingest pipeline to set the _id from the file.checksum field.

1. Enable checksum in FSCrawler settings: Set fs.checksum to a supported algorithm (e.g., SHA-256).

2. Create an Elasticsearch Ingest Pipeline:

PUT _ingest/pipeline/set-id-from-checksum
{
  "description": "Set the _id from file.checksum",
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{file.checksum}}}"
      }
    }
  ]
}

3. Configure the FSCrawler job: Set elasticsearch.pipeline to the name of your pipeline.

Handling Duplicates:

• Last writer wins: Use the default elasticsearch.bulk_operation: index. The last path discovered for a specific content hash will overwrite previous ones.
• First writer wins: Set elasticsearch.bulk_operation: create. Subsequent duplicates will fail with a conflict, which FSCrawler treats as a non-fatal event, preserving the first indexed version.

Note: The checksum is computed from the binary file, not the extracted text.

name: "test"
fs:
  index_content: true
  checksum: "SHA-256"
eslasticsearch:
  pipeline: "set-id-from-checksum"
  bulk_operation: "create"