Skip to main content
CommunityTeamEnterprise

Manual

Overview​

kJQ is a filter-based language for processing structured data. A kJQ program is a "filter" that takes an input and produces an output. Filters can be combined using pipes (|) to create powerful data transformation pipelines.

kJQ is based on the lightweight jq language for JSON processing. See the Differences from jq section to learn more about the differences between kJQ and jq.

Core concepts​

The pipeline model​

The fundamental concept in kJQ is the pipeline, where data flows from left to right through a series of transformations:

input | transform1 | transform2 | output

Each stage in the pipeline receives the output of the previous stage as its input.

Filters​

A filter is the fundamental building block of kJQ. Every kJQ expression is a filter that takes an input value and produces an output. Filters can be simple (like extracting a field) or complex (like transforming entire data structures). The power of kJQ comes from combining filters using pipes to create sophisticated data processing pipelines.

Think of filters as functions that transform data: they receive input, process it, and produce output. Even simple values like 42 or "hello" are filters - they ignore their input and always produce the same literal value as output.

The identity filter​

The simplest filter is . (dot), which passes its input unchanged to its output. This is a starting point in pipelines.

Object identifier-index​

Object identifier-index filters extract values from JSON objects using field names. These are the most common way to access object properties and navigate nested data structures.

  • .foo - Extract the value of field "foo" from an object
  • .foo.bar - Chain field access (equivalent to .foo | .bar)
  • ."field-name" - Access fields with special characters or that start with digits
  • .["foo"] - Bracket notation for field access (equivalent to .foo)

Array index​

Array index filters access specific elements within arrays using their position. Arrays are zero-indexed, meaning the first element is at position 0.

  • .[0] - Access the first element of an array (zero-indexed)
  • .[-1] - Access the last element (negative indexing)
  • .[-2] - Access the second-to-last element

Array/String slice​

Array slice filters extract subsequences from arrays or substrings from strings. They use the syntax .[start:end] where start is inclusive and end is exclusive, similar to Python slicing.

  • .[2:5] - Extract elements from index 2 (inclusive) to 5 (exclusive)
  • .[1:] - Extract elements from index 1 to the end
  • .[:3] - Extract elements from the beginning to index 3 (exclusive)
  • .[2:-1] - Extract from index 2 to second-to-last element

Literals and data construction​

  • "hello" - String literal
  • 42 - Number literal
  • true, false, null - Boolean and null literals

Tagged literals​

Tagged literals provide a way to create richer data types:

  • #dt "2025-01-01" - Date literal from ISO 8601 string
  • #dt "2025-01-01T10:30:00Z" - Date with time component
  • #uuid "550e8400-e29b-41d4-a716-446655440000" - UUID literal

Data types​

kJQ supports a rich type system that extends beyond basic JSON types, especially when working with AVRO logical types or Clojure data formats.

Basic JSON types​

  • null - Represents absence of a value
  • boolean - True or false values
  • number - Numeric values (internally handled as appropriate precision)
  • string - Text data
  • array - Ordered collections of values
  • object - Key-value mappings

Extended types​

  • date - ISO 8601 date values (e.g., 2024-12-25T10:30:00Z)
  • double - Double-precision decimal numbers for financial/scientific calculations
  • uuid - Universally unique identifiers (e.g., 550e8400-e29b-41d4-a716-446655440000)
  • keyword - Clojure keywords (symbolic identifiers that evaluate to themselves, e.g. :topic)

Type behavior notes​

  • Numbers are automatically promoted to appropriate precision (long, double) based on deserializer context
  • Dates support ISO 8601 format and can be manipulated with date-specific operations
  • Doubles maintain Double precision for mathematical operations where floating-point errors matter
  • UUIDs are treated as a distinct type with their own validation and formatting rules

Special values and date operations​

Negation​

A kJQ query filter can be negated with the not transform. Negation can be applied to logically combined filters.

e.g. | not

Current time​

  • now - Evaluates to the current timestamp when the kJQ expression executes

Duration expressions​

Duration expressions use ISO 8601 duration format for date arithmetic:

  • pt5m - 5 minutes
  • pt1h - 1 hour
  • pt2d - 2 days
  • pt1w - 1 week

Date arithmetic​

  • now - pt1h - One hour ago
  • now + pt30m - 30 minutes from now
  • .value.someDate | from-date - pt2d - Two days before someDate

Transforms​

Transforms are zero-argument operations that can be easily chained in pipelines. They operate on the current data and return a transformed result.

Type conversion​

  • to-long - Converts value to a long integer
  • to-double - Converts value to a double-precision number
  • to-uuid - Converts a string representation of a UUID to a UUID instance
  • to-string - Converts value to a string representation
  • parse-json - Parses a JSON string into a data structure

Date/Time​

  • from-date - Converts a ISO 8601 date string to a UNIX timestamp

Numeric operations​

  • min - Returns the minimum value from an array
  • max - Returns the maximum value from an array
  • floor - Rounds down to the nearest integer
  • ceil - Rounds up to the nearest integer

String operations​

  • upper-case - Converts string to uppercase
  • lower-case - Converts string to lowercase
  • trim - Removes whitespace from both ends
  • ltrim - Removes whitespace from the left end
  • rtrim - Removes whitespace from the right end

Array operations​

  • reverse - Reverses the order of elements in an array
  • sort - Sorts array elements in ascending order
  • unique - Removes duplicate elements from an array
  • first - Returns the first element of an array
  • last - Returns the last element of an array
  • flatten - Flattens nested arrays into a single level

Object operations​

  • keys - Returns an array of object keys
  • values - Returns an array of object values

Utility​

  • is-empty - Returns true if the value is empty (null, empty string, empty array, empty object)
  • length - Returns the length of strings, arrays, or objects

Functions​

Functions take one or more argument and operate on the current data with that parameter.

String functions​

  • startswith(prefix) - Tests if string starts with the given prefix
  • endswith(suffix) - Tests if string ends with the given suffix
  • contains(substring) - Tests if string contains the given substring
  • inside(string) - Tests if current value is contained within the given string
  • test(regex) - Tests if string matches the given regular expression
  • split(delimiter) - Splits string into array using the given delimiter
  • join(separator) - Joins array elements into string using the given separator

Collection functions​

  • has(key) - Tests if object has the given key or array has element at given index
  • within(collection) - Tests if current value exists within the given collection

Operators​

Comparison operators​

  • == - Equal to
  • != - Not equal to
  • < - Less than
  • <= - Less than or equal to
  • > - Greater than
  • >= - Greater than or equal to

Logical operators​

  • and - Logical AND operation
  • or - Logical OR operation

Math operators​

  • + - Addition
  • - - Subtraction
  • * - Multiplication
  • / - Division
  • % - Modulo (remainder)
  • mod - Modulo operation (alternative syntax)
  • quot - Integer division (quotient)
  • rem - Remainder operation

Alternative operator​

  • // - Provides fallback values when the left side produces false, null, or no results

The alternative operator is commonly used for:

  • Default values: .name // "Unknown"
  • Null coalescing: .user.email // .contact.email // "no-email"
  • Fallback chains: .primary_data // .backup_data // .default_data

Unlike or, which is for boolean logic, // is for value selection.