Special characters in the string are incorrectly encoded

[peterq]

test code

func TestStdJsonParseStd(t *testing.T) {
	var s = `hello � world`

	bin, err := json.Marshal(s)
	log.Println(string(bin), err)

	var d interface{}
	err = json.Unmarshal(bin, &d)
	log.Println(d, err)

	a := fastjson.Arena{}
	bin = a.NewString(s).MarshalTo(nil)
	err = fastjson.ValidateBytes(bin)
	log.Println(string(bin), err)
}

Output

=== RUN   TestStdJsonParseStd
2023/02/20 15:36:09 "hello \u0014 world" <nil>
2023/02/20 15:36:09 hello � world <nil>
2023/02/20 15:36:09 "hello \x14 world" cannot parse JSON: cannot parse string: unknown escape sequence \x; unparsed tail: ""
--- PASS: TestStdJsonParseStd (0.00s)
PASS

Expected:
like the standard library, encode � to \u0014 insteadof \x14 (which will cuase a problem when decodeing the json string later)

[peterq]

@Joswu-945

[kostaspap]

Encountered the same issue today. It seems that MarshalTo generates output that fails to be parsed by Go standard json library.

[kostaspap]

Seems to be the same issue as #86

[rleighton]

Is this fixed now? I ran the above test code and it worked.

=== RUN   TestStdJsonParseStd
2024/08/14 16:06:06 "hello � world" <nil>
2024/08/14 16:06:06 hello � world <nil>
2024/08/14 16:06:06 "hello � world" <nil>
--- PASS: TestStdJsonParseStd (0.00s)

[rleighton]

Not fixed:

func TestStdJsonParseStd(t *testing.T) {
	var s = `hello � world`
	s = "hello \u0014 world"

	bin, err := json.Marshal(s)
	log.Println(string(bin), err)

	var d interface{}
	err = json.Unmarshal(bin, &d)
	log.Println(d, err)

	a := fastjson.Arena{}
	bin = a.NewString(s).MarshalTo(nil)
	err = fastjson.ValidateBytes(bin)
	log.Println(string(bin), err)
}

=== RUN TestStdJsonParseStd
2024/08/14 16:48:32 "hello \u0014 world"
2024/08/14 16:48:32 hello world
2024/08/14 16:48:32 "hello \x14 world" cannot parse JSON: cannot parse string: unknown escape sequence \x; unparsed tail: ""

[rleighton]

The way we worked around this is we wrote code to scan the output and "fix" any unicode chars not properly escaped.

[klondikedragon]

There was a PR in progress in #87 that fixed this. In aperturerobotics#2 I added unit tests (including the suggested test case above) that verified its fixed with those changes. Hopefully these changes can get merged into one/both of these repos...