DOCSEARCHbyAlgolia
Submit Docs

Forem Developer Documentation

website·Indexed 36 days ago

https://developers.forem.com/

Developer guides for Forem, covering installation on Windows, macOS, and Gitpod, as well as technical overviews of its architecture, caching, and embedding-driven semantic search. Includes detailed instructions on testing (Unit, Functional, E2E with Cypress, and Frontend), deployment processes, and self-hosting under the AGPL-3 license.

Tokens
38.2K
Snippets
209
Records
353
Agent score
68%

What's inside Forem

  1. Overview of Forem authentication providers

    Forem handles authentication using the Devise and OmniAuth libraries. It supports several third-party authentication providers, including Facebook, GitHub, Twitter, Apple, and Google.

  2. Overview of Forem Semantic Search

    Forem provides embedding-driven semantic search that matches content based on meaning and intent using high-dimensional vector embeddings. Unlike traditional keyword search, which uses lexical matching and is publicly accessible, semantic search requires authentication and uses cosine distance to determine relevance.

    Technical Implementation:

    • Embedding Model: Google's gemini-embedding-2 (768-dimensional).
    • Vector Database: PostgreSQL with pgvector (v0.8.0+) and HNSW indexes.
    • Metric: Cosine distance.

  3. Overview of Forem Admin Dashboard access

    The Forem administration dashboard is located behind the /admin route. It provides administration tools and reports accessible to users with the admin or super_admin roles. Authorization for these tools is managed via the Rolify gem.

  4. Overview of Forem frontend testing architecture

    Frontend tests in Forem are used to validate JavaScript logic in pack files or Preact components. Tests are located in __tests__ directories adjacent to the components they test. The platform uses Jest as the primary testing library, with expect for unit tests and preact-testing-library for integration tests. If a test requires utility modules, they should be placed in a utilities folder inside the __tests__ directory, as Jest is configured to ignore this folder when identifying test suites.
    app/javascript/article-form
├── __tests__
│   └── utilities/  <-- Place test utilities here
└── articleForm.jsx

  5. Overview of Forem Concepts

    Concepts in Forem allow for the semantic grouping and classification of content using machine learning models (gemini-embedding-2). Unlike traditional tags, concepts use a "semantic anchor" (a name and description) to generate embeddings. Articles are automatically mapped to a concept when their semantic similarity exceeds a specified threshold.

  6. Track code coverage with Codecov

    Forem uses Codecov to track code coverage. Codecov automatically posts comments on Pull Requests indicating the percentage change in coverage compared to the main branch. The recommended target is to maintain a code coverage percentage of 80% or higher.

  7. Understand Forem core conceptual terminology

    Forem uses specific terminology to define its product concepts. Developers should be aware of these terms to avoid conceptual sprawl and ensure consistency across the codebase and UI.

  8. Debug CORS issues locally

    To troubleshoot CORS issues or view detailed information about CORS headers in a local environment, enable the DEBUG_CORS variable in your .env file.
    export DEBUG_CORS=true

  9. Forem technical stack and infrastructure

    Forem utilizes the following technologies and services for its core infrastructure:

    • Web Server: Puma
    • Database & Search: PostgreSQL (primary database and Full Text Search)
    • Caching: Redis (cached data) and Fastly (edge caching)
    • Image Handling: Cloudinary and/or Imgproxy for serving/manipulation; ImageMagick for upload manipulation; Carrierwave, Fog, and AWS S3 for storage/upload
    • Background Processing: Sidekiq and Active Job
    • Error Monitoring: Honeybadger
    • Search & Parsing: Ransack (internal search), Redcarpet and Rouge (Markdown parsing)
    • Frontend: Preact (partial frontend), InstantClick (modified version for navigation acceleration), and Docusaurus (documentation)
    • Authorization & Roles: Pundit (authorization) and Rolify (role management)
    • Email: Sendgrid (transactional) and Mailchimp (marketing)
    • Hosting & Version Control: Heroku (hosting), Git (version control), and GitHub (source code and issue tracking)
    • Utilities: CounterCulture (association counter caches)

  10. Retrieve articles mapped to a concept

    Retrieve a paginated list of articles classified under a specific concept. You can sort the results using the sort parameter:

    • -similarity (default): Most semantically relevant articles first.
    • -published_at: Newest articles first.
    • -score: Most popular/high-engagement articles first.
    # Sort by semantic relevance
curl -X GET "https://dev.to/api/concepts/123/articles?sort=-similarity&page=1&per_page=20" \
     -H "Accept: application/vnd.forem.api-v1+json" \
     -H "api-key: YOUR_API_KEY"

# Sort by popularity
curl -X GET "https://dev.to/api/concepts/123/articles?sort=-score&page=1&per_page=20" \
     -H "Accept: application/vnd.forem.api-v1+json" \
     -H "api-key: YOUR_API_KEY"

  11. Understand the Listing concept

    A 'Listing' is a single unit within the marketplace/classifieds section of a Forem site (found at /listings). Listings utilize both categories for segmentation and tags for organization, similar to how posts function.

  12. Manage concepts based on user roles

    Permissions for the Concepts API are divided by role:

    • Curators / Specific Owners: Can list accessible concepts, retrieve details, fetch mapped articles, and update concept metadata (description, similarity threshold, and score).
    • Super Administrators: Have full system-wide access, including creating new concepts, deleting concepts, and triggering historical backfills (lookbacks) to categorize older articles.