A format without %d left the day of the month at 0, which mktime and
todate read as the last day of the previous month, and the weekday was
wrong for dates in January and February of 1900 and 2100.
Start parsing from the 1st, so a format without a day now gives the
first of the month, and a time-only format no longer outputs the 8 and
367 placeholders for the weekday and day of the year. This also fixes
the weekday of dates in January and February of 1900 and 2100.
Both were defined as explode | map(...) | implode, which took 4 seconds
on a 10 MB string, about 90 times as long as parsing it.
Change the case of A-Z or a-z a byte at a time in C, copying multibyte
UTF-8 sequences unchanged, so the output stays the same and the 10 MB
case now takes about 60 ms including parsing. On a non-string input the
error now names the function, "ascii_downcase input must be a string",
instead of "explode input must be a string".
Since jq 1.8.0, negating a number literal computed 0 - x, so -0 and
0 | -. printed 0 while (0+0) | -. printed -0.
Flip the sign bit with decNumberCopyNegate() instead, so -0 prints -0
again as it did in jq 1.7, and -(-0) is still 0.
jvp_utf8_next() had a fast path for ASCII, but it is not inlined into
its callers, so every byte of a string cost a function call.
Move the end check and the ASCII case into a static inline function in
jv_unicode.h and keep the rest out of line. This speeds up parsing of
string-heavy input by up to 1.17x and string functions such as length,
explode and @html, with the same output for every input.
These functions failed with "unknown system failure" whenever strftime()
returned 0 for a non-empty format, but 0 also means the format expands to
an empty string, as %p does in a locale without AM/PM.
Prepend a character to the format and trim it from the result, so that 0
can only mean the result is too long. Now 0 | strflocaltime("%p") gives
"" in fr_FR.UTF-8, and a result that is still too long fails with
"result too long" error instead. The size limit is unchanged.
Printing a string wrote each character with a separate call, one
fwrite() per character, or one jv_string_append_buf() for tojson and
tostring, which made output of string-heavy data slow.
Scan ahead for the next byte that needs escaping and write the run of
bytes before it with a single call, without changing the output.
Printing strings is now significantly faster, most of all for long
strings with few characters to escape, and the same applies to tojson
and tostring. With --ascii-output, text that is almost all non-ASCII is
about the same speed, since nearly every byte is escaped anyway.
Converting a double outside the int range to int is undefined behavior,
and has() and del() did it on array indices like infinite or 1e10, which
UBSan reports as "inf is outside the range of representable values of
type 'int'". Only NaN was guarded against.
Clamp the index to the int range in has(), as getting and setting array
elements already do, and drop out-of-range indices in del() before they
are converted. Such indices can never refer to an element, so has()
returns false and del() leaves the array unchanged, the same as for
indices past the end of the array. Valid indices deleted together with
out-of-range ones, as in del(.[0, 1e10]), are still deleted.
The UTF-8 BOM was stripped only at the start of the parser stream, but jq
feeds all input files into a single parser, so a BOM at the start of the
second and later files reached the parser as content and failed with a
confusing "Invalid numeric literal" error. Strip it in the file readers
instead, at the start of each input file including stdin, and in
jv_load_file.
This also strips the BOM for raw input, so -R and --rawfile no longer
yield a leading U+FEFF, and a BOM at the start of a -f program file or an
imported module is now accepted rather than reported as a syntax error.
In exchange, "\uFEFF1" | fromjson is now a parse error, since a U+FEFF
inside a string is content rather than an encoding marker, and the
dedicated "Malformed BOM" error is gone.
`fromdateiso8601` was `strptime("%Y-%m-%dT%H:%M:%SZ") | mktime`, which
accepts only one of the many representations ISO 8601 permits. strptime
cannot be extended to cover the rest portably, since it has no specifier
for fractional seconds and glibc discards `%G` and `%V`.
Replace it with a parser in src/iso8601.c that handles basic and extended
representations, ordinal and week dates, fractional seconds, offsets and
date-only input. It uses integer arithmetic instead of `time_t`, so the
results no longer depend on the platform.
This is stricter than before. Impossible dates like `2015-02-30`, which
`mktime` used to normalize into March, and unpadded components like
`2015-3-5T6:00:00Z` are now errors. So is any trailing input, including
whitespace. Leading whitespace is rejected too; before, glibc skipped it
and macOS did not. A leap second now maps to the second before it rather
than the next minute. The error message has changed, and the result can
now be non-integer.
The lexer returned a single ALTERNATION token for `?//`, so any `?`
followed by `//` was consumed by it, and `.name?//"unknown"` failed
to parse even though they read as the optional operator followed by
the alternative operator.
Drop the lexer rule and spell the destructuring alternative operator
as `'?' "//"` instead. So `x ?// y` now parses as `x? // y`, while
`. as [$a] ?// [$b] | ...` keeps its meaning. Note that `?` and `//`
may now be separated by whitespaces inside a pattern.
join/1 folded each element as `(acc + $x) + elem`, with the element
branch ending in `.//""`. Both leave a second live reference to the
accumulator: the nested + feeds the same `.` to both operands, and the
`//` leaves a forkpoint below it on the stack. `jvp_string_append`
therefore cannot grow the accumulator in place, and copies the whole
string once per element.
Pipe `acc + $x` into `. + elem` and drop the `//`, since adding null to
a string is already a no-op. join/1 is now linear: joining 8000
strings of 1000 chars goes from 6.5s to 0.06s, with output and error
messages unchanged.
`reduce f as $x (init; update)` evaluated `f` against a stack slot
that forkpoints created while evaluating `init` restore. Both the
`DUPN` opening the reduction loop and the `LOADVN` retrieving the
result pop that slot destructively, nulling it out, so an `init`
that backtracks (e.g. `0, 0`) made `f` see `null` and fail with
"Cannot iterate over null".
Duplicate the input before the `FORK` instead of inside the loop,
and drop the leftover copy with `POP` once the loop is exhausted.
The slot left live at the `FORK` is then a block pushed after
`init` completed, which no earlier forkpoint can reach, so nulling
it is harmless. This is the same approach `gen_foreach` has used
since c70ae1a.
`DUPN` had no other user, so remove the opcode.
block_mark_referenced walks the joined block backwards and skips the
body of any definition not yet marked referenced, so a library must
precede its importers. f58787c41 registered a library before walking
its dependencies, to have somewhere to keep the `loading` flag, which
reversed that order and made block_drop_unreferenced free definitions
that are still referenced through bound_by.
Keep the libraries being loaded on a stack threaded through the
load_library frames instead, so registration can move back after the
dependency walk.
jv_equal and jv_cmp overflows the C stack on deeply nested
input. Cap recursion at 10000 with -1 / INT_MIN sentinels;
operators that compose user expressions surface this as
"Equality check too deep" / "Comparison too deep".
Fixes CVE-2026-47770.
Deeply nested path arrays can cause unbounded recursion in
`jv_setpath`, `jv_getpath`, and `jv_delpaths`, leading to
stack overflow. Add a depth limit of 10000 to match the
existing `tojson` depth limit. This fixes CVE-2026-33947.
The `jv_string_empty` did not properly null-terminate the string data,
which could lead to a heap buffer overflow. The test case of
GHSA-p7rr-28xf-3m5w (`0[""*0]`) was fixed by the commit dc849e9bb7,
but another case (`0[[]|implode]`) was still vulnerable. This commit
ensures string data is properly null-terminated, and fixes CVE-2025-48060.
* Handle input errors for --indent
Handle malformed and overflowing arguments. Also, forbid leading and
trailing spaces to match the behavior of tonumber from commit ce0e788
(improve tonumber/0 performance by parsing input as number literal,
2024-03-02).
* Do not skip leading whitespace in jvp_strtod
`jvp_strtod` skips leading whitespace, but its decnum counterpart
`decNumberFromString` (called within `jv_number_with_literal`) does not.
Those two are called interchangeably, so it leads to inconsistent
behavior depending on whether the decnum feature is enabled.
Additionally, `classify`, used in the token scanner, only considers
[ \t\n\r] to be whitespace, but `jvp_strtod` consumes the larger set
[ \t\n\v\f\r], so those extra characters are considered literals.
Changing this deviates from the behavior of `strdod` from <stdlib.h> and
is technically a breaking API change, since it is a public symbol.
Previously, the precedence of `as` binding was higher than the binary
operators. This results in unexpected behaviors, `1 + 2 as $x | -$x`
was `-1` instead of `-3`. The commit 92daac90f6 (PR #3053) changed
the behavior to emit syntax error on this case. Now, this commit allows
binary operators with `as` binding. This fixes#2446, and fixes#3284.
This commit fixes signed integer overflow and SEGV issues on growing
arrays and objects. The size of arrays and objects is now limited to
`536870912` (`0x20000000`). This fixes CVE-2024-23337 and fixes#3262.
This commit improves the parser and compiler error messages, making
it more easier to understand where in the filter the problem is.
Like modern compilers, jq will show the line in the filter where the
error occurred, will display squiggly line under the invalid token.
This commit improves the parser to fix some unintuitive results of the unary
operator `-` (e.g. `[-1 as $x | 1,$x]` is now `[1,-1]`, previously `[-1,-1]`).
Also, this commit now allows to use binary operator expressions, `reduce`,
`foreach`, `if`, and `try` expressions for object values.
Simplifies the compilation of `foreach` to avoid the problem in #3227 by
removing unnecessary `FORK ... BACKTRACK`.
Does not address `reduce`, which is less trivial.
This commit drops support for parsing NaN with payload in JSON like
`NaN123` and fixes CVE-2024-53427. Other JSON extensions like `NaN` and
`Infinity` are still supported. Fixes#3023, fixes#3196, fixes#3246.
This commit improves the performance of repeating strings, by copying
the result string instead of the string being repeated. Also it adds
an error message when the result string is too long.
When the UPDATE query in a reduce or foreach syntax does not emit
a value, it should simply skip updating the state variable. But
currently it resets the state variable to null.
Jay Hemnani