Commit Graph
294 Commits
Author SHA1 Message Date
DavidKorczynski 693a7261e5 jq_fuzz_execute: fix memory leak
Signed-off-by: David Korczynski <david@adalogics.com>
2023-12-03 03:53:49 +01:00
David Korczynski bfb7fd570f jq_fuzz_execute: cleanup un-needed extern
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
David Korczynski 252ab244ce Add fuzzer targeting jq_next
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
David Korczynski 13353515bd jq_fuzz_compile: dump disassembly
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
Mattias Wadman 98a206964d Convert decnum to binary64 (double) instead of decimal64
This is what the JSON spec suggests and will also be less confusing compared to other jq implementations and langauges.

Related to #2939
2023-11-29 09:36:33 +01:00
Emanuele Torre 460a5c12b4 jq_fuzz_parse_extended.c: don't jv_free() twice
jv_dump() frees its argument.

I missed this problem before merging #2952, whoops! =)

fixup from eb3b5654bb
2023-11-28 20:51:07 +01:00
David Korczynski eb3b5654bb test: add jv_dump to extended fuzzer
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-28 20:36:59 +01:00
David Korczynski 44300e4310 Extend fuzzing set up
Adds a parse function ins `jv_parse.c` that enables parsing using custom
flags for the parser. This is then used by two fuzzers added as well.

This is to make sure fuzzing hits various code parts currently not
fuzzed, e.g. `stream_token`:
https://storage.googleapis.com/oss-fuzz-coverage/jq/reports/20231125/linux/src/jq/src/jv_parse.c.html#L241

Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-28 20:36:59 +01:00
Emanuele Torre cca1f7d18f Comment bug fixes, and fully support Tcl-style multiline comments
* bugfix: comments were incorrectly being terminated by CR; for example
    jq -n $'1 #foo\r'
  fails to compile because the CR character terminates the comment, and
  CR is not a valid character in jq syntax.

* improvement: comments fully support Tcl-style line continuation.
  Previously this was only "supported" in `-f' scripts, whose first line
  starts with "#!", and second line starts with # and ends with \, only
  for the comment on the second line, only for one extra line.

* man: document comment syntax, which was previously undocumented.

* tests: add regression tests for the bugfix, and some tests for line
  continuation in comments.
2023-11-03 15:52:02 -05:00
Emanuele Torre 6b6feaffbb Revert "Revert "od -c => od -tc: od -c is an XSI extension..."
This reverts commit 0e70f7a57e.

There is no reason to revert this change.

In #2922, I only disagreed with the commit message suggesting that
  LC_CTYPE=C od -t c    is   equivalent to   od -c

The only documented differences are that -tc is required to be
influenced by -N and -j, while -c is not, and that -c is required to
only support a subset of the backslash sequences that -tc should
support.
2023-10-05 15:37:44 +02:00
Nicolas Williams 0e70f7a57e Revert "od -c => od -tc: od -c is an XSI extension equivalent to LC_CTYPE=C od -tc and not universally available"
This reverts commit 0bce9fb8ed.
2023-10-04 22:28:24 -05:00
наб 0bce9fb8ed od -c => od -tc: od -c is an XSI extension equivalent to LC_CTYPE=C od -tc and not universally available 2023-10-04 17:46:13 -05:00
Emanuele Torre 4ebd21e1eb Allow passing the inline jq script before --
jq previously only allowed passing the inline script before -- (as if
they were options) even though one would expect the inline script to be
a positional argument.

Since jq previously also refused to run with a usage error if the script
was passed after -- (It was not assuming  .  as script as it does when
no arguments are passed), and positional arguments are allowed before --
and even before other options, it should not be a breaking change to
change that weird behaviour, and allow the script to appear after --.

It also simplifies the option parsing code a bunch.

Fixes #2918
2023-10-04 00:00:46 -05:00
Thomas Bozeman th026106 7f547827e4 Simplify pick example
Old pick example included input array in command line, making `input` confusing
and redundant.
2023-10-03 22:48:12 +02:00
taoky 8f81668014 Fix the default colors to use 39, the default foreground color (#2904) 2023-09-22 09:18:41 +09:00
Nicolas Williams 11c528d04d Add setlocale() call (fix #1740) 2023-09-06 14:05:10 -05:00
Nicolas Williams 0e067ef936 Improve handling of non-integer numeric indices (fix #2815) 2023-08-27 12:32:12 +02:00
itchyny 7d64381016 Fix overflow on numeric comparison (#2849)
Although #2839 fixed the overflow of exponent subtraction,
there still is possibility of overflow in the `D2U` macro.
This patch fixes the overflow in the `D2U` macro, and also
truncates the maximum digits to `DEC_MAX_DIGITS`.
2023-08-17 15:56:57 +09:00
itchyny 0733fd3d58 Add a regression test for negative indices and fix a pick/1 test 2023-08-16 17:49:44 -05:00
itchyny 3fa10e8cc1 Fix crash on numeric comparison again (ref #2825)
The decNumber library subtracts the exponents of two numbers,
we make sure to limit the number of digits not to make it overflows.
Since the maximum adjusted exponent is `emax` and the minimum is
`emin - digits + 1`, we follow `emax - (emin - digits + 1) <= INT32_MAX`.
2023-08-13 14:17:25 +02:00
Alex Wilson 65ed95c93a Expose the names of a module's defined function via modulemeta
This reuses the existing `block_list_funcs` capability and adds an extra field on the `modulemeta` output, called `defs`, containing that list of functions.
2023-08-12 20:46:36 -05:00
Mattias WadmanandNicolas Williams fdab39bc7b Build windows 64bit binary using UCRT64 (#2832)
* Build windows 64bit binary using UCRT64

Is the default and recommended msystem setting. Will produce
binaries that are compatible with windows 10 and later.

Also run tests for 32bit build.

Related to #2831

* Use jq -b in tests/shtest

* Add Windows strptime

* Make Windows-optional tests not run on Windows again

---------

Co-authored-by: Nicolas Williams <nico@cryptonector.com>
2023-08-12 20:18:45 +02:00
itchyny a692060129 Change the default color of null to Bright Black 2023-08-05 23:21:12 -05:00
Emanuele Torre f94a9d463f Let error(null) throw null
This patch removes the weird behaviour of jv_invalid_with_msg(jv_null())
that returns jv_invalid() (i.e. empty), instead of a boxed jv_null().

The previous behaviour of  null|error  was obviously unintentional, and
allowing is jv_invalid_with_msg() to return values on which you can't
call jv_invalid_get_msg() is only error prone.
2023-08-04 17:38:52 -05:00
Nicolas Williams ddef804945 Clarify the // operator (close #2189) 2023-08-03 14:41:53 -05:00
itchyny 949d38e6dc Fix crash on numeric comparison (ref #2804) (#2818) 2023-08-03 23:19:52 +09:00
itchyny fd0d475694 Fix a test case added in #2790 2023-08-02 22:33:09 +09:00
itchynyandLeonid S. Usov 680baeffeb Fix rounding small (but not too small) numbers to zero on calculation
Co-authored-by: Leonid S. Usov <leonid.s.usov@gmail.com>
2023-08-02 00:32:32 -05:00
Nicolas Williams 27a4d5757e Require a main program (fix #2785) 2023-08-01 23:06:01 -05:00
pkoppstein 4d4c17ca6a Simple and efficient implementation of walk/1 (#2795) 2023-08-01 07:54:21 +09:00
itchyny 0f80921268 Fix constant folding of division and reminder with zero divisor
Previously constant folding of zero division (e.x. 1/0) produces a
compile error. This was incorrectly implemented by checking if the
division result is infinite, so produces wrong results compared to the
query where no constant folding is processed (e.x. 1e308/0.1). This
patch delays the operation when the divisor is zero. This makes the
results more consistent, but changes the exit code on zero division from
3 to 5. Also 0/0 now produces the zero division error, not NaN.

This patch also fixes the modulo operation. Previously constant folding
logic does not take care of the % operator, but now it folds if the both
dividend and divisor are safe numbers to cast to the integer type, and
the divisor is not zero. This patch also fixes some code that relies on
undefined cast behaviors in C. The modulo operation produces NaN if
either the dividend or divisor is NaN.
2023-07-31 15:39:13 -05:00
pkoppstein 336c3deb8a Test negative indices in path expressions 2023-07-28 12:34:26 -05:00
Emanuele Torre 17889a1a85 Fix memory leak for { $foo: bar }
{ BINDING: ExpD } wasn't freeing BINDING.

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60865
2023-07-27 20:59:35 -05:00
itchyny a1e791acf8 Rename --nul-output to --raw-output0, abort on string containing NUL
The option naming --nul-output was confusing, especially when we have a
similar option for input stream in the future (--nul-input vs --null-input).
Based on the observation of other command line tools, we rename the option
to --raw-output0. We also drop the short option -0 to avoid confusion on
introducing the NUL-delimited input option.

Unlike the other command line tools outputting file names with NUL delimiter,
jq deals with JSON, and its strings may contain NUL character. To protect
users from the risk of injection attacks, we abort the program and print an
error message before outputting strings including NUL character. Closes #2683.
2023-07-26 23:55:00 -05:00
pkoppstein 13fbe98dff manual.yml: remove contingent tests
Also clarify non-prescriptive nature of some tests in jq.test
2023-07-25 23:43:57 -05:00
pkoppstein 7b725378b9 builtin.jq: naive abs/0
manual.yml explains that the def is naive, and mentions fabs, etc.
2023-07-25 23:43:57 -05:00
Mattias Wadman a949745059 implode: Better invalid input validation and handling
Error on non-number and nan codepoint, would asserd before
Replace negative codepoint and surrogate range with unicode replacement character, would assert before

Fixes #1160
2023-07-25 13:38:41 -05:00
pkoppstein 8f49600a6b builtin.jq: simpler and faster transpose (#2758) 2023-07-25 09:12:23 +09:00
Mattias Wadman 16ae275d4c Fix leaks on invalid input for strftime, strflocaltime and mktime 2023-07-24 16:45:43 +02:00
David Korczynski 97c6d2848c Add fuzzer targeting load_file
Signed-off-by: David Korczynski <david@adalogics.com>
2023-07-24 15:45:12 +02:00
David Korczynski cf3c11bde8 Add compile fuzzer for OSS-Fuzz
Signed-off-by: David Korczynski <david@adalogics.com>
2023-07-24 13:44:27 +02:00
itchyny ed334b536f Improve manual in various ways (inputs, sort_by, foreach sections, etc.) (#2744)
- Add error/0 and mentions null input behavior (close #2231)
- Explain value iterator suffix syntax .foo[] (close #1047)
- Mention array slicing is also zero-based (close #2094)
- Add examples of input and inputs filters (close #2216, close #2470)
- Improve sort_by about multiple values (close #2103, close #2467, close #2474)
- Improve foreach section and simplify examples (close #1148, close #2169)
- Fix recurse/1 document on how it is identical using recurse/2 (close #2036, close #2412)
- Add non-string examples of index/1, rindex/1 (close #1422)
- Simplify the example of truncate_stream/1 (close #1736)
2023-07-24 20:24:44 +09:00
Nico Williams 1cf6515c06 Fix try/catch catches more than it should #1859
Close #1885, #2140, #2011, #2220, #2485, #2073

Rename the FORK_OPT opcode to TRY_BEGIN, add a TRY_END opcode, and wrap
errors when raising through a TRY_END so that they will not be caught by
the matching TRY_BEGIN.

Now a `try exp catch handler` expression generates code like:

    TRY_BEGIN handler
    <exp>
    TRY_END
    JUMP past_handler
    handler: <handler>
    past_handler:
    ...

On backtrack through TRY_BEGIN it just backtracks.

If anything past the whole thing raises when <exp> produced a value,
then the TRY_END will catch the error, wrap it in another, and
backtrack.  The TRY_BEGIN will see a wrapped error and then it will
unwrap and re-raise the error.

If <exp> raises, then TRY_BEGIN will catch the error and jump to the
handler, but the TRY_BEGIN will not stack_save() in that case, so on
raise/backtrack the TRY_BEGIN will not execute again (nor will the
TRY_END).
2023-07-24 13:20:21 +02:00
itchyny 98f709d0c1 Fix stderr to output string with no decoration (fix #2063) 2023-07-23 17:06:00 -05:00
Nicolas Williams f64ef1a7d7 Fix --stream-errors w/o --stream, and doc+test (close #2748) 2023-07-23 03:33:28 -05:00
itchyny c9c45d7f03 Add tests for --arg, --argjson, and similar options 2023-07-22 12:29:24 -05:00
itchyny 99a77f724b Add a regression test for #2000 2023-07-21 09:01:28 +09:00
pkoppstein 3ec66c858c manual.yml: revise section on identity, clarifying what is subject to change
Mostly clarifications w.r.t. numbers.

The anomalous behavior of leading minus signs is documented, with an explicit notice that this will probably change.
2023-07-19 22:32:24 -05:00
itchyny 0615d4982c Respect NO_COLOR environment variable to disable color output 2023-07-19 00:05:41 -05:00
0b558f6ed4 Make object key color configurable (close #1739, #1791, #2638)
Co-authored-by: David Haguenauer <ml@kurokatta.org>
Co-authored-by: Eric Pruitt <eric.pruitt@gmail.com>
2023-07-18 17:53:29 -05:00