Improve performance of join/1 on large arrays (#3639)
join/1 folded each element as `(acc + $x) + elem`, with the element branch ending in `.//""`. Both leave a second live reference to the accumulator: the nested + feeds the same `.` to both operands, and the `//` leaves a forkpoint below it on the stack. `jvp_string_append` therefore cannot grow the accumulator in place, and copies the whole string once per element. Pipe `acc + $x` into `. + elem` and drop the `//`, since adding null to a string is already a no-op. join/1 is now linear: joining 8000 strings of 1000 chars goes from 6.5s to 0.06s, with output and error messages unchanged.
This commit is contained in:
1 parent
32580f5d83
commit
9a75bb0d43
2 files changed
+12
-2
No files matched your search
+4
-2
@@ -63,9 +63,11 @@ def strings: select(type == "string");
|
||||
def nulls: select(. == null);
|
||||
def values: select(. != null);
|
||||
def scalars: select(type|. != "array" and . != "object");
|
||||
# Keep the accumulator the only reference on the left of + so the string is
|
||||
# appended in place; a nested + or a // inside the reduce copies it every step.
|
||||
def join($x): reduce .[] as $i (null;
|
||||
(if .==null then "" else .+$x end) +
|
||||
($i | if type=="boolean" or type=="number" then tostring else .//"" end)
|
||||
(if .==null then "" else .+$x end) |
|
||||
. + ($i | if type=="boolean" or type=="number" then tostring end)
|
||||
) // "";
|
||||
def _flatten($x): reduce .[] as $i ([]; if $i | type == "array" and $x != 0 then . + ($i | _flatten($x-1)) else . + [$i] end);
|
||||
def flatten($x): if $x < 0 then error("flatten depth must not be negative") else _flatten($x) end;
|
||||
|
||||
Reference in new issue
Block a user