Previously, with clang++:
jq.h:46:37: error: typedef redefinition with different
types ('struct jq_util_input_state *' vs 'jq_util_input_state')
With g++:
jq.h:46:37: error: conflicting declaration
‘typedef struct jq_util_input_state* jq_util_input_state’
This typedef was added to libjq by commit 0d41447 which was
after the 1.4 release, so although it is a public API, this
is not a backcompat break because it has never been in a
release.
Specifying the "*" at all uses of jq_util_input_state is
slightly tedious, but jq_state already works that way, so at
least it will be consistent.
With this patch, jq run-time errors printed to stderr will contain the
filename and line of the offending input.
But note that the JSON text parser does (yet) print the input filename
on parse error. A jq run-time error is a jq program error, not
including syntax errors nor JSON text input format errors.
Examples:
With stdin and multiple lines:
$ printf '{"a":43}\n{"a":{"b":66}}\n' | ./jq '.a+1'
44
jq: error (at stdin:2): object and number cannot be added
With multiple files:
$ printf '{"a":43}' > 1.json
$ printf '{"a":"hello"}\n' > 2.json
$ printf '{"a":{"b":66}}\n' > 3.json
$ ./jq '[.a]|@tsv' 1.json 2.json 3.json
"43"
"hello"
jq: error (at 3.json:1): object is not valid in a csv row
With very long lines (spanning multiple `fgets` calls):
$ ( printf '{"a":43}\n' ;
printf '{"a":{"b":[' ; seq 10000 | paste -d, -s | tr -d '\n' ;
printf ']}}\n' ;
printf '{"a":"hello"}\n' ) | ./jq '[.a] | @tsv'
"43"
jq: error (at stdin:2): object is not valid in a csv row
"hello"
With raw input:
$ seq 1000 | ./jq --raw-input 'select(.=="700") | . + 10'
jq: error (at stdin:700): string and number cannot be added
Caveat:
The reported line will be the last line of the (valid) parsed JSON data.
Example:
$ printf '{\n"a":\n"hello"\n\n\n}\n' | ./jq '.a+4'
jq: error (at stdin:6): string and number cannot be added
Detect output errors when the program exits.
Currently:
$ echo '{}' | jq . > /dev/full && echo ok
ok
with the patch:
$ echo '{}' | jq . > /dev/full && echo ok
Error: writing output failed: No space left on device
also apply to hardware/network/other I/O errors.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Improve robustness in automated system when using exit code in shell scripts,
by exiting with code 2 if there was any input error (even overriding other
possible error exit codes).
Exit code 2 is already used to indicate system errors.
Without the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
0
With the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
2
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
With this change, runtime exceptions are propagated to non-zero exit
code of 'jq', allow better scripting and automation. The new exit code
value is 5.
This allows using the shell's and/or operations ('&&' and '||') to
detect input runtime exceptions.
Before:
runtime exceptions are printed to STDERR, but not reported as non-zero exit-code:
$ echo '"hello"' | jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
0
After:
$ echo '"hello"' | ./jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Note that there's a subtle interplay when using "-e" option.
The value of the non-zero exit code changes from 4 to 5, but it still
indicates a 'failure' or non-true value.
Before:
$ echo '"hello"' | jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
4
After:
$ echo '"hello"' | ./jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Much of this could be in libjq. Eventually all of the work of reading
from files and looping over `jq_next()` should move into libjq, with
`main()` mostly doing all the command-line option processing.
To import a module now use:
# Import module.jq file:
import "relative/path/to/module" as foo;
# Use the module's defs as foo::<def-name>
To import a JSON file:
# Read file.json:
import "relative/path/to/file" as $foo;
#
# Use as $foo::foo
Using `-L` now drops the builtin library path and appends the requested
path to the empty array (or the result of an earlier `-L`).
Support for the `$JQ_LIBRARY_PATH` environment variable has been
removed.
This is to fix the problem where `break` is dynamic, not lexical.
With this it should be possible to do this sort of thing:
label $break | inputs | if ... then $break|error else . end
This is a backwards-incompatible change for master, but the previous
`break` hadn't shipped yet.
Still needed:
- testing
Streaming means that outputs are produced as soon as possible. With the
`foreach` syntax one can write programs which reduce portions of the
streaming parse of a large input (reduce into proper JSON values, for
example), and discard the rest, processing incrementally.
This:
$ jq -c --stream .
should produce the same output as this:
$ jq -c '. as $dot | path(..) as $p | $dot | getpath($p) | [$p,.]'
The output of `jq --stream .` should be a sequence of`[[<path>],<leaf>]`
and `[[<path>]]` values. The latter indicate that the array/object at
that path ended.
Scalars and empty arrays and objects are leaf values for this purpose.
For example, a truncated input produces a path as soon as possible, then
later the error:
$ printf '[0,\n'|./jq -c --stream .
[[0],0]
parse error: Unfinished JSON term at EOF at line 3, column 0
$
The search path listed in an import directive can now be an array. The
top-level search path is appended. Null and empty strings in the path
terminate any search. The "." in "." and "./*" is replaced with the
directory containing the file doing the import (for command-line
programs this is the current directory, though that may be a bad idea).
No version numbers or anything of the sort are gratuitously added to the
search paths.
All this makes external package managers possible by allowing
dependencies to be installed local to dependents.
David Tolnay