DOCSEARCHbyAlgolia
Submit Docs

The Art of Command Line

repository·master·Indexed 13 days ago

https://github.com/jlevy/the-art-of-command-line

A collection of high-level command-line mastery tips, shell scripting best practices, and essential tool usage guides for developers working in Unix-like environments. Covers Bash concepts, system management, process monitoring, text manipulation, and specialized tools for JSON, YAML, and CSV processing.

Tokens
69K
Snippets
234
Records
450
Agent score
93%

What's inside The Art of Command Line

  1. Overview of The Art of Command Line

    master

    The Art of Command Line is a collection of notes and tips for using the command line effectively, aimed at improving flexibility and productivity for engineers. The guide covers a range of topics from basics to advanced one-liners, primarily focused on Linux environments.

    Key Characteristics

    • Scope: Covers essential command-line knowledge for both beginners and experts.
    • Accuracy: Provides specific examples for common use cases.
    • Conciseness: Focuses on high-impact tips that save significant time.

    Target Environment

    • Primary OS: Linux.
    • Shell: Focuses on the interactive Bash shell, though many tips apply to other shells and Bash scripting.
    • Compatibility: Many tips are applicable to other Unices, OS X, or Cygwin.
    • Specialized Sections: Includes specific sections for OS X and Windows.

    Recommended Tools for Learning

    • Package Managers: Use apt-get, yum, dnf, pacman, pip, or brew to install mentioned software.
    • Explainshell: Use explainshell.com to get detailed breakdowns of what specific commands, options, and pipes do.

  3. Core Bash and Command Line Fundamentals

    master

    To use the command line effectively, developers should master several foundational areas:

    • Bash Shell: Understand Bash basics. Use man bash to read the manual. While other shells like zsh or fish exist, Bash is the standard for portability across servers.
    • Text Editors: Become proficient in at least one terminal-based editor. nano is simple for basic tasks, Vim (vi) is powerful and fast for advanced users, and Emacs is used for complex editing tasks.
    • Documentation & Discovery:
      • Use man to read command manuals (e.g., man 1 command for general commands, man 5 for file formats, man 8 for administration).
      • Use apropos <term> to find relevant manual pages.
      • Use help and help -d for Bash built-in commands.
      • Use type <command> to determine if a command is an executable, a shell built-in, or an alias.
      • Use Explainshell to decompose complex commands, options, and pipes.
    • I/O Redirection & Pipes: Master > (overwrite file), >> (append to file), < (input redirection), and | (piping output from one command to another). Understand the difference between stdout and stderr.
    • Globbing & Quoting: Understand file expansion using *, ?, and [...], and the functional differences between single quotes ' and double quotes ".
    • Job Management: Manage background and foreground processes using &, ctrl-z (suspend), ctrl-c (interrupt), jobs, fg (foreground), bg (background), and kill.

  4. Core Bash and Shell concepts to learn

    master

    A foundational understanding of the following concepts is required for effective command line use:

    • Redirection and Pipes: Use > to overwrite a file with output, >> to append to a file, < for input redirection, and | (pipes) to pass the output of one command as input to another. Understand the difference between stdout and stderr.
    • Globbing and Quoting: Learn filename expansion using wildcards like *, ?, and [...]. Understand the difference between double quotes " and single quotes '.
    • Process Management: Manage background and foreground processes using &, ctrl-z (suspend), ctrl-c (interrupt), jobs, fg (foreground), bg (background), and kill.
    • SSH: Learn ssh and passwordless authentication using ssh-agent and ssh-add.

  5. Use subshells to isolate directory changes

    master

    Wrap commands in parentheses (...) to create a subshell. This allows you to change directories or modify the environment temporarily without affecting the parent shell's current working directory.

    # The parent shell stays in the original directory
(cd /some/other/dir && other-command)

  6. Advanced Bash Redirection and Process Substitution

    master

    Use advanced redirection techniques to handle files and command outputs:

    Process Substitution: Use <(command) to treat the output of a command as a file. This is useful for comparing outputs without creating temporary files.

    Redirection:

    • command >logfile 2>&1: Redirect both standard output (stdout) and standard error (stderr) to logfile.
    • command </dev/null: Redirect standard input from /dev/null to prevent commands from hanging if they expect input.

    Here Documents: Use cat <<EOF ... to pass multi-line blocks of text into a command.

    # Compare a local file with the output of a remote command
diff /etc/hosts <(ssh somehost cat /etc/hosts)

  7. Configure environment and locales

    master

    Locale and Performance

    System locale settings (e.g., LANG) affect tool behavior like sorting (collation) and performance. Internationalized routines can be significantly slower. To use traditional byte-based sorting and improve performance for tasks like uniqueness checks, use:

    export LC_ALL=C

    Per-command Environment

    You can set a specific environment for a single command by prefixing it with the variable assignment, e.g.,:

    TZ=Pacific/Fiji date

  8. Handle filenames with spaces safely in Bash

    master

    When dealing with filenames that contain spaces, follow these best practices:

    1. Quote variables: Always wrap Bash variables in double quotes: "$FOO".
    2. Use Null Delimiters: Use -0 (for xargs) or -print0 (for find) to separate filenames with null characters instead of spaces.
      • Example: locate -0 pattern | xargs -0 ls -al
    3. Set IFS: If iterating through files in a loop, set the Internal Field Separator to a newline: IFS=$'\n'.

  9. Essential Bash and File Management Concepts

    master

    Mastering the command line requires understanding several core concepts:

    Input/Output and Pipes

    • Redirection: Use > to overwrite a file with output and >> to append to a file.
    • Streams: Understand stdout (standard output) and stderr (standard error).
    • Pipes: Use | to pass the output of one command as input to another.

    File Manipulation

    • Globbing: Use wildcards like *, ?, and [ ] for filename expansion.
    • Viewing Files: Use less, head, and tail. For real-time monitoring of logs, use tail -f or less +F.
    • Links: Understand the difference between hard links (ln) and symbolic/soft links (ln -s).
    • Permissions and Ownership: Use chown to change ownership and chmod to change permissions.
    • Disk Usage: Use du -hs * for a quick summary of disk space used by files/directories, df for filesystem usage, and ls -i to view i-nodes.

    Task Management

    • Manage background and foreground processes using &, ctrl-z (suspend), ctrl-c (interrupt), jobs, fg, bg, and kill.

  10. Bash Variable Expansions and Brace Expansion

    master

    Bash provides powerful ways to manipulate strings and lists:

    Variable Expansions:

    • ${name:?error message}: Check if a variable exists; exit with error if not.
    • ${name:-default}: Use a default value if the variable is unset or empty.
    • $(( expression )): Arithmetic evaluation (e.g., i=$(( (i+1) % 5 ))).
    • {1..10}: Integer list expansion.
    • ${var%suffix}: Remove suffix.
    • ${var#prefix}: Remove prefix.

    Brace Expansion {...}: Automates combinations of elements. This is performed before other expansions.

    • mv foo.{txt,pdf} some-dir: Moves both foo.txt and foo.pdf.
    • cp somefile{,.bak}: Equivalent to cp somefile somefile.bak.
    • mkdir -p test-{a,b,c}/subtest-{1,2,3}: Creates a complex directory tree.

    Expansion Order: Brace expansion $\rightarrow$ Tilde expansion $\rightarrow$ Parameter/Variable replacement $\rightarrow$ Arithmetic evaluation $\rightarrow$ Command substitution $\rightarrow$ Word splitting $\rightarrow$ Pathname expansion.

  11. Bash Parameter Expansion techniques

    master

    Bash provides powerful ways to manipulate variables without external tools:

    • Check if a variable exists: ${name:?error_messages}. Example: input_file=${1:?usage: $0 input_file} (aborts if $1 is missing).
    • Arithmetic expansion: i=$(( (i + 1) % 5 )).
    • Sequences: {1..10}.
    • String trimming (Suffix): ${var%suffix}.
    • String trimming (Prefix): ${var#prefix}. Example: If var=foo.pdf, then echo ${var%.pdf}.txt outputs foo.txt.
    var=foo.pdf
echo ${var%.pdf}.txt

  12. Fundamental Bash Job and Process Management

    master

    Manage background and foreground processes using these tools:

    • &: Run a command in the background.
    • ctrl-z: Suspend the current foreground process.
    • jobs: List active jobs.
    • fg: Bring a job to the foreground.
    • bg: Resume a suspended job in the background.
    • kill: Terminate a process.