AIPM: Building Your Knowledge System for Large Language Models

A Mixture-of-Experts (MoE) architecture replaces a single dense layer with a set of $N$ expert functions and a gating mechanism. Instead of a single function $f(x, \theta)$, the MoE approach uses:

1. Experts: $N$ distinct expert functions $f_1, f_2, ..., f_N$, each with its own set of parameters $\theta_1, \theta_2, ..., \theta_N$.
2. Gating Function: A function $G(x)$ that produces a probability distribution over the $N$ experts, determining how much weight to assign to each expert's output for a given input $x$.
3. Combined Output: The final prediction is the weighted sum of the experts' outputs based on the gating distribution.

The development and deployment of Large Language Models (LLMs) intersect with law primarily in two areas: data collection and application deployment.

1. Data: LLMs require massive datasets, often scraped without consent. This raises issues regarding Copyright law (whether training on protected works is a violation) and Privacy law (whether training on public or private data violates individual privacy rights).
2. Applications: Downstream tasks (e.g., chatbots, QA) can be used for intentional harm (spam, phishing, disinformation) or deployed in high-stakes settings (healthcare, lending, education), which may trigger existing fraud, abuse, or industry-specific regulations.

Beyond simple collection, data in industry and research is managed through broader ecosystems:

• Data Governance: The process by which an organization creates, maintains, and secures data quality (e.g., the BigScience project's framework for responsible curation vs. indiscriminate scraping).
• Data Dignity: A concept treating data as collective labor rather than individual property. It suggests that because data has collective value, 'data coalitions' should act as intermediate organizations for collective bargaining between producers and buyers.

Dense Transformer architectures (like GPT-3) apply the same set of parameters to every input token. As these models scale, they face bottlenecks in network bandwidth and training efficiency because they must be split across many machines using data, model, and pipeline parallelism. To continue scaling, 'selective' architectures are used so that each input activates only a small subset of the model's total parameters. Two primary selective architectures are:

1. Mixture-of-Experts (MoE): A set of specialized 'experts' is created, and only a few are activated per input.
2. Retrieval: A store of raw data is maintained; the model retrieves relevant documents from this store to predict the output.

Language models are trained in a task-agnostic way (e.g., on the Pile), but downstream tasks often differ in ways that make simple prompting insufficient:

• Formatting: Tasks may require specific outputs (e.g., binary labels for Natural Language Inference) that differ from next-token prediction or MASK filling.
• Topic Shift: The task may focus on a specialized domain (e.g., medical records) not well-represented in pre-training data.
• Temporal Shift: The task may require knowledge created after the model's training cutoff or knowledge that is not publicly available.

Language models have a "dual use" in the context of content moderation:

• Generation: Malicious actors can use LLMs to generate and amplify toxic content or disinformation.
• Detection: LLMs can be used to detect disinformation and aid in automated content moderation (blocking, deleting, flagging, or hiding content).

Content moderation is fundamentally political and context-dependent; norms vary significantly across different platforms or communities (e.g., different subreddits on Reddit).

Large Language Models (LLMs) can exhibit harms in two primary forms related to group identity:

1. Performance Disparities: Occur when a model performs significantly better for some social groups and worse for others on the same task (e.g., an Automatic Speech Recognition system that has higher error rates for Black speakers than White speakers). This can create a feedback loop where marginalized groups stop using the system, leading to less data and further increasing the disparity.
2. Social Biases and Stereotypes: Social biases are systematic associations between concepts and specific groups (e.g., associating 'science' with 'men'). Stereotypes are a prevalent form of bias that are widely held, oversimplified, and fixed. These are often propagated through language and can lead to performance disparities if the model fails to understand anti-stereotypical data.

Encoder-only models like BERT are designed to produce strong bidirectional contextual embeddings rather than generating text. BERT utilizes two primary objective functions: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).

Key architectural components include:

• [CLS] token: The embedding of this token is used to drive classification tasks.
• [SEP] token: Used to separate two sequences (e.g., a premise and a hypothesis in natural language inference).
• Bidirectional context: Unlike decoder-only models, BERT considers tokens both to the left and right of a target token.

The environmental cost of LLMs is primarily driven by the massive compute required for training and inference. Key impacts include:

• Greenhouse Gas Emissions: Training large models can produce hundreds of metric tons of CO2eq (e.g., DeepMind's Gopher produced ~380 net metric tons CO2eq).
• Water Footprint: Data centers require significant freshwater for cooling, and electricity generation itself is a major water consumer.
• Human Toxicity: Chip manufacturing releases chemicals into the environment that can cause cancer and other health issues.
• Resource Depletion: Manufacturing electronic devices requires minerals like lithium and cobalt, as well as fossil fuels.