From 3ec66c858ca38306192d64b274e4819accbd1a9d Mon Sep 17 00:00:00 2001 From: pkoppstein Date: Wed, 19 Jul 2023 22:25:10 -0400 Subject: [PATCH] manual.yml: revise section on identity, clarifying what is subject to change Mostly clarifications w.r.t. numbers. The anomalous behavior of leading minus signs is documented, with an explicit notice that this will probably change. --- docs/content/manual/manual.yml | 66 ++++++++++++++++++++++++---------- jq.1.prebuilt | 44 +++++++++++++++++++---- tests/man.test | 12 +++++-- 3 files changed, 95 insertions(+), 27 deletions(-) diff --git a/docs/content/manual/manual.yml b/docs/content/manual/manual.yml index f031c927..2f4bd3c0 100644 --- a/docs/content/manual/manual.yml +++ b/docs/content/manual/manual.yml @@ -312,36 +312,66 @@ sections: - title: "Identity: `.`" body: | - The absolute simplest filter is `.` . This is a filter that - takes its input and produces it unchanged as output. That is, - this is the identity operator. + The absolute simplest filter is `.` . This filter takes its + input and produces the same value as output. That is, this + is the identity operator. - Since jq by default pretty-prints all output, this trivial - program can be a useful way of formatting JSON output from, - say, `curl`. + Since jq by default pretty-prints all output, a trivial + program consisting of nothing but `.` can be used to format + JSON output from, say, `curl`. - An important point about the identity filter is that it - guarantees to preserve the literal decimal representation - of values. This is particularly important when dealing with numbers - which can't be losslessly converted to an IEEE754 double precision + Although the identity filter never modifies the value of its + input, jq processing can sometimes make it appear as though + it does. For example, using the current implementation of + jq, we would see that the expression: + + `1E1234567890 | .` + + produces `1.7976931348623157e+308` on at least one platform. + This is because, in the process of parsing the number, this + particular version of jq has converted it to an IEEE754 + double-precision representation, losing precision. + + The way in which jq handles numbers has changed over time + and further changes are likely within the parameters set by + the relevant JSON standards. The following remarks are + therefore offered with the understanding that they are + intended to be descriptive of the current version of jq and + should not be interpreted as being prescriptive: + + (1) Any arithmetic operation on a number that has not + already been converted to an IEEE754 double precision + representation will trigger a conversion to the IEEE754 representation. - jq doesn't truncate the literal numbers to double unless there - is a need to make arithmetic operations with the number. - Comparisons are carried out over the untruncated big decimal - representation of the number. + (2) jq will attempt to maintain the original decimal + precision of number literals, but in expressions such + `1E1234567890`, precision will be lost if the exponent is + too large. - jq will also try to maintain the original decimal precision of the provided - number literal. See below for examples. + (3) In jq programs, a leading minus sign will trigger the + conversion of the number to an IEEE754 representation. + + (4) Comparisons are carried out using the untruncated + big decimal representation of numbers if available, as + illustrated in one of the following examples. examples: - program: '.' input: '"Hello, world!"' output: ['"Hello, world!"'] - - program: '. | tojson' + - program: '.' + input: '0.12345678901234567890123456789' + output: ['0.12345678901234567890123456789'] + + - program: '[., tojson]' input: '12345678909876543212345' - output: ['"12345678909876543212345"'] + output: ['[12345678909876543212345,"12345678909876543212345"]'] + + - program: '. < 0.12345678901234567890123456788' + input: '0.12345678901234567890123456789' + output: ['false'] - program: 'map([., . == 1]) | tojson' input: '[1, 1.000, 1.0, 100e-2]' diff --git a/jq.1.prebuilt b/jq.1.prebuilt index bfa4a2b7..c4f99dd6 100644 --- a/jq.1.prebuilt +++ b/jq.1.prebuilt @@ -246,19 +246,41 @@ Be warned that this option can change backwards\-incompatibly\. .SH "BASIC FILTERS" . .SS "Identity: \." -The absolute simplest filter is \fB\.\fR \. This is a filter that takes its input and produces it unchanged as output\. That is, this is the identity operator\. +The absolute simplest filter is \fB\.\fR \. This filter takes its input and produces the same value as output\. That is, this is the identity operator\. . .P -Since jq by default pretty\-prints all output, this trivial program can be a useful way of formatting JSON output from, say, \fBcurl\fR\. +Since jq by default pretty\-prints all output, a trivial program consisting of nothing but \fB\.\fR can be used to format JSON output from, say, \fBcurl\fR\. . .P -An important point about the identity filter is that it guarantees to preserve the literal decimal representation of values\. This is particularly important when dealing with numbers which can\'t be losslessly converted to an IEEE754 double precision representation\. +Although the identity filter never modifies the value of its input, jq processing can sometimes make it appear as though it does\. For example, using the current implementation of jq, we would see that the expression: +. +.IP "" 4 +. +.nf + +`1E1234567890 | \.` +. +.fi +. +.IP "" 0 . .P -jq doesn\'t truncate the literal numbers to double unless there is a need to make arithmetic operations with the number\. Comparisons are carried out over the untruncated big decimal representation of the number\. +produces \fB1\.7976931348623157e+308\fR on at least one platform\. This is because, in the process of parsing the number, this particular version of jq has converted it to an IEEE754 double\-precision representation, losing precision\. . .P -jq will also try to maintain the original decimal precision of the provided number literal\. See below for examples\. +The way in which jq handles numbers has changed over time and further changes are likely within the parameters set by the relevant JSON standards\. The following remarks are therefore offered with the understanding that they are intended to be descriptive of the current version of jq and should not be interpreted as being prescriptive: +. +.P +(1) Any arithmetic operation on a number that has not already been converted to an IEEE754 double precision representation will trigger a conversion to the IEEE754 representation\. +. +.P +(2) jq will attempt to maintain the original decimal precision of number literals, but in expressions such \fB1E1234567890\fR, precision will be lost if the exponent is too large\. +. +.P +(3) In jq programs, a leading minus sign will trigger the conversion of the number to an IEEE754 representation\. +. +.P +(4) Comparisons are carried out using the untruncated big decimal representation of numbers if available, as illustrated in one of the following examples\. . .IP "" 4 . @@ -268,9 +290,17 @@ jq \'\.\' "Hello, world!" => "Hello, world!" -jq \'\. | tojson\' +jq \'\.\' + 0\.12345678901234567890123456789 +=> 0\.12345678901234567890123456789 + +jq \'[\., tojson]\' 12345678909876543212345 -=> "12345678909876543212345" +=> [12345678909876543212345,"12345678909876543212345"] + +jq \'\. < 0\.12345678901234567890123456788\' + 0\.12345678901234567890123456789 +=> false jq \'map([\., \. == 1]) | tojson\' [1, 1\.000, 1\.0, 100e\-2] diff --git a/tests/man.test b/tests/man.test index 02f95329..00fae382 100644 --- a/tests/man.test +++ b/tests/man.test @@ -2,9 +2,17 @@ "Hello, world!" "Hello, world!" -. | tojson +. +0.12345678901234567890123456789 +0.12345678901234567890123456789 + +[., tojson] 12345678909876543212345 -"12345678909876543212345" +[12345678909876543212345,"12345678909876543212345"] + +. < 0.12345678901234567890123456788 +0.12345678901234567890123456789 +false map([., . == 1]) | tojson [1, 1.000, 1.0, 100e-2]