mq Documentation

mq is a command-line tool that processes Markdown using a syntax similar to jq. It’s written in Rust, allowing you to easily slice, filter, map, and transform structured data.

Why mq?

mq makes working with Markdown files as easy as jq makes working with JSON. It’s especially useful for:

• LLM Workflows: Efficiently manipulate and process Markdown used in LLM prompts and outputs
• Documentation Management: Extract, transform, and organize content across multiple documentation files
• Content Analysis: Quickly extract specific sections or patterns from Markdown documents
• Batch Processing: Apply consistent transformations across multiple Markdown files

Features

• Slice and Filter: Extract specific parts of your Markdown documents with ease.
• Map and Transform: Apply transformations to your Markdown content.
• Command-line Interface: Simple and intuitive CLI for quick operations.
• Extensibility: Easily extendable with custom functions.
• Built-in support: Filter and transform content with many built-in functions and selectors.
• REPL Support: Interactive command-line REPL for testing and experimenting.
• IDE Support: VSCode Extension and Language Server Protocol (LSP) support for custom function development.

Getting Started

This section guides you through the installation of mq.

Install

Cargo

cargo install --git https://github.com/harehare/mq.git mq-cli --tag v0.2.8

Binaries

You can download pre-built binaries from the GitHub releases page:

# macOS (Intel)
curl -L https://github.com/harehare/mq/releases/download/v0.2.8/mq-x86_64-apple-darwin -o /usr/local/bin/mq && chmod +x /usr/local/bin/mq

# macOS (Apple Silicon)
curl -L https://github.com/harehare/mq/releases/download/v0.2.8/mq-aarch64-apple-darwin -o /usr/local/bin/mq && chmod +x /usr/local/bin/mq

# Linux x86_64
curl -L https://github.com/harehare/mq/releases/download/v0.2.8/mq-x86_64-unknown-linux-gnu -o /usr/local/bin/mq && chmod +x /usr/local/bin/mq

# Linux arm64
curl -L https://github.com/harehare/mq/releases/download/v0.2.8/mq-aarch64-unknown-linux-gnu -o /usr/local/bin/mq && chmod +x /usr/local/bin/mq

# Windows (PowerShell)
Invoke-WebRequest -Uri https://github.com/harehare/mq/releases/download/v0.2.8/mq-x86_64-pc-windows-msvc.exe -OutFile "$env:USERPROFILE\bin\mq.exe"

Homebrew

# Using Homebrew (macOS and Linux)
$ brew install harehare/tap/mq

Docker

$ docker run --rm ghcr.io/harehare/mq:0.2.8

Visual Studio Code Extension

You can install the VSCode extension from the Visual Studio Marketplace.

GitHub Actions

You can use mq in your GitHub Actions workflows with the Setup mq action:

steps:
  - uses: actions/checkout@v4
  - uses: harehare/setup-mq@v1
  - run: mq '.code' README.md

MCP (Model Context Protocol) server

mq supports an MCP server for integration with LLM applications.

See the MCP documentation for more information.

Python

You can use mq in Python through the markdown-query package:

# Install from PyPI
$ pip install markdown-query

npm

You can use mq in npm through the mq-web package:

$ npm i mq-web

Web crawler

# Using Homebrew (macOS and Linux)
$ brew install harehare/tap/mqcr

Development

Prerequisites

• Rust
• just - a command runner
• wasm-pack (optional, for WebAssembly support)

Setting up the development environment

Clone the repository:

git clone https://github.com/harehare/mq.git
cd mq

Install development dependencies:

# Using cargo
cargo install just wasm-pack

Or if you prefer using asdf:

# Using asdf
asdf install

Common development tasks

Here are some useful commands to help you during development:

# Run the CLI with the provided arguments
just run '.code'

# Run formatting, linting and all tests
just test

# Run formatter and linter
just lint

# Build the project in release mode
just build

# Update documentation
just docs

Check the just --list for more available commands and build options.

Playground

An Online Playground is available, powered by WebAssembly.

Interactive TUI

The Text-based User Interface (TUI) provides an interactive way to explore and query Markdown files directly in your terminal.

$ mq tui file.md

TUI Features

• Interactive Querying: Enter and edit queries in real-time with immediate feedback
• Detail View: Examine the structure of selected markdown nodes in depth
• Navigation: Browse through query results with keyboard shortcuts
• Query History: Access and reuse previous queries

TUI Key Bindings

Web crawler

mq-crawler is a web crawler that fetches HTML content from websites, converts it to Markdown format, and processes it with mq queries. It’s distributed as the mqcr binary.

Key Features

• HTML to Markdown conversion: Automatically converts crawled HTML pages to clean Markdown
• robots.txt compliance: Respects robots.txt rules for ethical web crawling
• mq-lang integration: Processes content with mq-lang queries for filtering and transformation
• Configurable crawling: Customizable delays, domain restrictions, and link discovery
• Flexible output: Save to files or output to stdout

Installation

# Using Homebrew (macOS and Linux)
$ brew install harehare/tap/mqcr

Usage

mqcr [OPTIONS] <URL>

Options

• -o, --output <OUTPUT>: Directory to save markdown files (stdout if not specified)
• -c, --crawl-delay <CRAWL_DELAY>: Delay between requests in seconds (default: 1)
• --robots-path <ROBOTS_PATH>: Custom robots.txt URL
• -m, --mq-query <MQ_QUERY>: mq-lang query for processing content (default: identity())

Examples

# Basic crawling to stdout
mqcr https://example.com

# Save to directory with custom delay
mqcr -o ./output -c 2 https://example.com

# Process with mq-lang query
mqcr -m '.h | select(contains("News"))' https://example.com

Example

under preparation

Hello world

# Hello world
select(or(.[], .code, .h)) | upcase() | add(" Hello World")

Markdown TOC

.h
| let link = to_link(add("#", to_text(self)), to_text(self), "")
| if (is_h1()):
  to_md_list(link, 1)
elif (is_h2()):
  to_md_list(link, 2)
elif (is_h3()):
  to_md_list(link, 3)
elif (is_h4()):
  to_md_list(link, 4)
elif (is_h5()):
  to_md_list(link, 5)
else:
  None

Exclude code

select(not(.code))

Extract js code

.code("js")

Extract table

.[1][]

Extract list

.[1]

Extract MDX

select(is_mdx())

Custom function

def snake_to_camel(x):
  let words = split(x, "_")
  | foreach (word, words):
      let first_char = upcase(first(word))
      | let rest_str = downcase(slice(word, 1, len(word)))
      | s"${first_char}${rest_str}";
  | join("");
| snake_to_camel()

Generate sitemap

def sitemap(item, base_url):
  let path = replace(to_text(item), ".md", ".html")
  | let loc = add(base_url, path)
  | s"<url>
  <loc>${loc}</loc>
  <priority>1.0</priority>
</url>";

CSV to markdown table

$ mq 'nodes | csv2table()' example.csv

Merging Multiple Files

$ mq -S 's"\n${__FILE__}\n"' 'identity()' docs/books/**/**.md

Reference

This is a reference documentation for the mq.

CLI

The mq command-line interface provides tools for querying and manipulating markdown content. Below is the complete reference for all available commands and options.

Usage: mq [OPTIONS] [QUERY OR FILE] [FILES]... [COMMAND]

Commands:
  repl  Start a REPL session for interactive query execution
  lsp   Start a language server for mq
  mcp   Start an MCP server for mq
  tui   Start a TUI for mq
  fmt   Format mq files based on specified formatting options
  docs  Show functions documentation for the query
  help  Print this message or the help of the given subcommand(s)

Arguments:
  [QUERY OR FILE]  
  [FILES]...       

Options:
  -f, --from-file
          load filter from the file
  -I, --input-format <INPUT_FORMAT>
          Set input format [possible values: markdown, mdx, html, text, null]
  -L, --directory <MODULE_DIRECTORIES>
          Search modules from the directory
  -M, --module-names <MODULE_NAMES>
          Load additional modules from specified files
      --args <NAME> <VALUE>
          Sets string that can be referenced at runtime
      --rawfile <NAME> <FILE>
          Sets file contents that can be referenced at runtime
  -F, --output-format <OUTPUT_FORMAT>
          Set output format [default: markdown] [possible values: markdown, html, text, json]
  -U, --update
          Update the input markdown
      --unbuffered
          Unbuffered output
      --list-style <LIST_STYLE>
          Set the list style for markdown output [default: dash] [possible values: dash, plus, star]
      --link-title-style <LINK_TITLE_STYLE>
          Set the link title surround style for markdown output [default: double] [possible values: double, single, paren]
      --link-url-style <LINK_URL_STYLE>
          Set the link URL surround style for markdown links [default: none] [possible values: none, angle]
  -S, --separator <QUERY>
          Specify a query to insert between files as a separator
  -o, --output <FILE>
          Output to the specified file
  -P <PARALLEL_THRESHOLD>
          Number of files to process before switching to parallel processing [default: 10]
  -h, --help
          Print help
  -V, --version
          Print version

Examples:

To filter markdown nodes:
$ mq 'query' file.md

To read query from file:
$ mq -f 'file' file.md

To start a REPL session:
$ mq repl

To format mq file:
$ mq fmt --check file.mq

Types and Values

Values

• 42 (a number)
• "Hello, world!" (a string)
• [1, 2, 3], array(1, 2, 3) (an array)
• {"a": 1, "b": 2, "c": 3}, dict(["a", 1], ["b", 2], ["c", 3]) (a dictionary)
• true, false (a boolean)
• None

Types

Environment Variables

A module handling environment-specific functionality.

• __FILE__: Contains the path to the file currently being processed.

Conditionals and Comparisons

Conditional expressions and comparison operators in mq allow for decision-making based on the evaluation of conditions, enabling dynamic behavior in your queries.

Conditionals

mq supports standard conditional operations through the following functions:

• and(a, b) - Returns true if both a and b are true
• or(a, b) - Returns true if either a or b is true
• not(a) - Returns true if a is false

Examples

# Basic comparisons
and(true, true, true)
# => true
or(true, false, true)
# => true
not(false)
# => true

Comparisons

mq provides comparison functionality through built-in functions.

Basic Comparisons

Standard comparison operators are supported:

• eq(a, b), a == b - Returns true if a equals b
• ne(a, b), a != b - Returns true if a does not equal b
• gt(a, b), a > b - Returns true if a is greater than b
• gte(a, b), a >= b - Returns true if a is greater than or equal to b
• lt(a, b), a < b - Returns true if a is less than b
• lte(a, b), a <= b - Returns true if a is less than or equal to b

Examples

# Basic comparisons
1 == 1
# => true
2 > 1
# => true
"a" <= "b"
# => true

# String comparisons
"hello" == "hello"
# => true
"xyz" > "abc"
# => true

# Numeric comparisons
5.5 >= 5.0
# => true
-1 < 0
# => true

# Logical operations
and(true, false)
# => false
or(true, false)
# => true
not(false)
# => true

# Complex conditions
and(x > 0, x < 10)
# =>  true if 0 < x < 10

Syntax

This section outlines the syntax rules in mq, providing a clear reference for writing valid code.

Comments

Similar to jq, comments starting with # are doc-comments.

# doc-comment
let value = add(2, 3);

Control flow

If Expression

The if expression evaluates a condition and executes code based on the result:

 if (eq(x, 1)):
   "one"
 elif (eq(x, 2)):
   "two"
 else:
   "other"

 if (eq(x, 1)):
   "one"

The if expression can be nested and chained with elif and else clauses. The conditions must evaluate to boolean values.

While Expression

The while loop repeatedly executes code while a condition is true:

let i = 0 |
while (lt(i, 3)):
  let i = add(i, 1) | i;
# => [1, 2, 3]

The while loop in this context returns an array containing all elements processed during the iteration. As the loop executes, it collects each processed value into an array, which is then returned as the final result once the loop condition becomes false.

Key points:

• Creates a new array from loop iterations
• Each loop cycle’s result is added to the array
• Returns the complete array after all iterations
• Similar to map/collect functionality but with while loop control

Until Expression

The until loop repeatedly executes code until a condition becomes true:

let x = 5 |
until(gt(x, 0)):
  let x = sub(x, 1) | x;
# => 0

Until loops are similar to while loops but continue until the condition becomes true instead of while the condition remains true.

Foreach Expression

The foreach loop iterates over elements in an array:

let items = array(1, 2, 3) |
foreach (x, items):
   sub(x, 1);
# => array(0, 1, 2)

Foreach loops are useful for:

• Processing arrays element by element
• Mapping operations across collections
• Filtering and transforming data

Operator

Pipe Operator

A functional operator that allows chaining multiple filter operations together.

Usage

The pipe operator (|) enables sequential processing of filters, where the output of one filter becomes the input of the next filter.

Examples

# Basic pipe usage
42 | add(1) | mul(2)
# => 86

# Multiple transformations
let mul2 = def mul2(x): mul(x, 2);
let gt4 = def gt4(x): gt(x, 4);
array(1, 2, 3) | map(mul2) | filter(gt4)
# => [6]

# Function composition
let double = def _double(x): mul(x, 2);
let add_one = def _add_one(x): add(x, 1);
5 | double(self) | add_one(self)
# => 11

? Operator

The ? operator is a safe navigation operator that provides null-safe operations.

Usage

When applied to a None value, the ? operator prevents errors by returning None instead of raising an exception.

Examples

# Safe access with ? operator
let x = None | x | add(1)?
# => None

# Chaining with ? operator
None | add(1)? | mul(2)?
# => None

# Normal operation when value exists
42 | add(1)?
# => 43

.. Operator

The range operator (..) creates sequences of consecutive values between a start and end point.

Usage

The range operator generates arrays of values from a starting point to an ending point (inclusive). It works with both numeric values and characters.

Examples

# Numeric ranges
1..5
# => [1, 2, 3, 4, 5]

# Character ranges
'a'..'e'
# => ["a", "b", "c", "d", "e"]

# Using ranges with other operations
1..3 | map(fn(x): mul(x, 2);)
# => [2, 4, 6]

# Reverse ranges
5..1
# => [5, 4, 3, 2, 1]

# Single element range
3..3
# => [3]

Def Expression

The def expression defines reusable functions with parameters:

Examples

# Function that doubles input
def double(x):
  mul(x, 2);

# Function with conditional logic
def is_positive(x):
  gt(x, 0);

# Composition of functions
def add_then_double(x, y):
  add(x, y) | double(self);

Environment variables

Environment variables can be referenced using $XXX syntax, where XXX represents the name of the environment variable. For example:

• $PATH - References the PATH environment variable
• $HOME - References the HOME environment variable
• $USER - References the current user’s username

This syntax is commonly used in shell scripts and configuration files to access system-level environment variables.

Fn Expression

Anonymous functions (lambda expressions) allow you to define functions inline without naming them. These functions can be passed as arguments to other functions, assigned to variables, or used directly in expressions.

Syntax

fn(parameters): program;

Examples

# Basic Anonymous Function
nodes | map(fn(x): add(x, "1");)

# Using Anonymous Functions as Callbacks
nodes | .[] | sort_by(fn(x): to_text(x);)

Include

Loads functions from an external file using the syntax include "module_name". The include directive searches for .mq files in the following locations:

• $HOME/.mq - User’s home directory mq folder
• $ORIGIN/../lib/mq - Library directory relative to the source file
• $ORIGIN/../lib - Parent lib directory relative to the source file
• $ORIGIN - Current directory relative to the source file

include "module_name"

Examples

# Include math functions from math.mq
include "math"
# Now we can use functions defined in math.mq
let result = add(2, 3)

Let Expression

The let expression binds a value to an identifier for later use:

# Binds 42 to x
let x = 42
# Uses x in an expression
let y = add(x, 1)
# Binds `add` function to z
let z = def _add(x): add(x, 1); | z(1)

Self

The current value being processed can be referenced as self. When there are insufficient arguments provided in a method call, the current value (self) is automatically passed as the first argument.

Examples

# These expressions are equivalent
"hello" | upcase()
"hello" | upcase(self)

String Interpolation

String Interpolation allow embedding expressions directly inside string literals. In mq, an interpolated string is prefixed with s" and variables can be embedded using ${} syntax.

Syntax

s"text ${ident} more text"

Escaping

You can escape the $ character in a string interpolation by using $$. This allows you to include literal $ symbols in your interpolated strings.

let price = 25
| s"The price is $$${price}"
# => Output: "The price is $25"

Examples

let name = "Alice"
| let age = 30
| s"Hello, my name is ${name} and I am ${age} years old."
# => Output: "Hello, my name is Alice and I am 30 years old."

Nodes

The nodes in mq allows you to access and manipulate all Markdown nodes as a single flat array.

Basic Usage

The nodes filter returns an array of all nodes in a Markdown document:

nodes

Examples

Finding all headings

nodes | select(.h)

Converting all text to uppercase

nodes | map(upcase)

Counting nodes by type

nodes | len()

Builtin selectors and functions

This page provides an introduction to the built-in selectors and functions available in mq. These are predefined components that you can use in your queries without having to define them yourself.

Builtin functions

Builtin selectors