Linguist

repository·main·Indexed 11 days ago

https://github.com/github-linguist/linguist

A Ruby library and CLI tool used by GitHub to detect programming languages in files, ignore binary or vendored files, and generate language breakdown statistics for repositories. It supports analysis via a Ruby API, a command-line interface (github-linguist and git-linguist), and Docker. Users can customize detection and statistics using .gitattributes with attributes such as linguist-language, linguist-detectable, linguist-documentation, and linguist-generated.

Tokens
15.7K
Snippets
64
Records
92
Agent score
96%

What's inside Linguist

  1. Use Math and GFM extensions in MDX

    main

    MDX supports several extensions including Math (LaTeX), GitHub Flavored Markdown (GFM) features like autolinks, footnotes, task lists, and tables, as well as GitHub-specific features like gemoji, mentions, and references.

    // Math
    $$ L = \frac{1}{2} \rho v^2 S C_L $$
    
    // GFM Task List
    * [ ] not done
    * [x] done
    
    // GFM Table
    | Stuff? | stuff! |
    | - | - |
    | asdasda | https://example.com |
    
    // GitHub Mentions & Gemoji
    @username :+1:
  2. How Linguist determines file languages

    main

    Linguist identifies the programming language of a file by applying a series of strategies in a specific order. The process begins by filtering out files that are binary, vendored, generated, documentation, or categorized as data (e.g., SQL) or prose (e.g., Markdown), while respecting any configured overrides.

    If an explicit language override is present via .gitattributes, that language is used immediately. Otherwise, Linguist applies the following strategies in order to identify the language or narrow down candidates:

    1. Vim or Emacs modeline: Checks for language hints in editor configuration lines within the file.
    2. Commonly used filename: Matches the filename against known patterns.
    3. Shell shebang: Inspects the #! line at the start of the file.
    4. File extension: Uses the file's suffix (e.g., .py, .rb).
    5. XML header: Looks for language declarations in XML-style headers.
    6. Man page section: Checks for man page metadata.
    7. Heuristics: Applies rule-based pattern matching.
    8. Naïve Bayesian classification: Uses statistical modeling as a final fallback.
  3. How language analysis is triggered on GitHub.com

    main

    On GitHub.com, language analysis is an asynchronous process:

    • Trigger: A low-priority background job is enqueued whenever changes are pushed to the default branch of a repository.
    • Caching: Results are cached for the lifetime of the repository and are only updated when the repository is updated.
    • Latency: Because analysis runs as a low-priority background job, there may be a delay before the language statistics bar reflects recent changes, especially during periods of high GitHub activity.
  4. Override language definitions using .gitattributes

    main

    Linguist allows you to customize language detection, statistics, and syntax highlighting by adding a .gitattributes file to your project. Use standard git-style path matchers to apply specific Linguist attributes to files or directories.

    Important Note: When testing with a local installation of Linguist, attributes will not take effect until the .gitattributes file is committed to your repository. Paths are calculated relative to the position of the .gitattributes file.

    # Reclassify .rb files as Java
    *.rb linguist-language=Java
    
    # Use hyphens instead of spaces in language names
    *.glyphs linguist-language=OpenStep-Property-List
    
    # Language names are case-insensitive and support aliases
    *.es linguist-language=js
    *.es linguist-language=JS
    *.es linguist-language=JAVASCRIPT
  5. Use MDX extensions for ESM, JSX, and Expressions

    main

    MDX allows you to combine Markdown with JavaScript features. You can use ECMAScript Modules (ESM) to import components and export constants, use JSX tags to render components directly in your content, and use curly braces {} to embed JavaScript expressions or logic.

    import {Chart} from './chart.js'
    export const year = 2018
    
    <Chart year={year} color="#fcb32c" />
    
    Two 🍰 is: {Math.PI * 2}
    
    {(function () {
      const guess = Math.random()
      if (guess > 0.66) return <span style={{color: 'tomato'}}>Look at us.</span>
      return <span>Not me.</span>
    })()}
  6. Exclude documentation using linguist-documentation

    main

    Linguist typically excludes documentation files from language statistics. You can manually mark paths as documentation (to exclude them) or unmark them (to include them) using the linguist-documentation attribute.

    # Apply override to all files in the directory
    project-docs/* linguist-documentation
    
    # Apply override to a specific file
    docs/formatter.rb -linguist-documentation
    
    # Apply override to all files and directories in the directory
    ano-dir/** linguist-documentation
  7. Configure system dependencies for Linguist

    main

    Linguist requires a recent version of Ruby. It is highly recommended to use a version manager like rbenv, rvm, asdf, or Homebrew rather than the default macOS/Xcode Ruby.

    Linguist depends on charlock_holmes (for encoding) and rugged (for libgit2 bindings), which require the following system libraries:

    Required Libraries:

    • `cmake"
    • pkg-config
    • ICU
    • zlib
    • libcurl
    • OpenSSL

    Installation on macOS (Homebrew):

    brew install cmake pkg-config icu4c

    Installation on Ubuntu:

    sudo apt-get install build-essential cmake pkg-config libicu-dev zlib1g-dev libcurl4-openssl-dev libssl-dev ruby-dev
  8. Manage vendored code using linguist-vendored

    main

    To prevent third-party libraries from inflating your project's language statistics, use the linguist-vendored attribute to mark paths as vendored (excluded) or un-vendor them (included).

    # Apply override to all files in the directory
    special-vendored-path/* linguist-vendored
    
    # Apply override to a specific file
    jquery.js -linguist-vendored
    
    # Apply override to all files and directories in the directory
    ano-dir/** linguist-vendored
  9. Mark files as detectable using linguist-detectable

    main

    By default, only languages of type programming or markup are included in language statistics. Languages of other types (like data or prose) are not 'detectable' by default.

    Use linguist-detectable to include or exclude paths from statistics. Note that the language must still be defined in languages.yml for this to work.

    *.kicad_pcb linguist-detectable
    *.sch linguist-detectable
    tools/export_bom.py -linguist-detectable