crash on one of our instances -- source sequence is illegal/malformed utf-8

[SteveDevOps]

any ideas? It loops through rescue, never resolves, and I have to manually stop it.

[16-03-03 13:28:05] listening on http#0.0.0.0:4567
[16-03-03 13:28:15] !!! Redmon has shit the bed, restarting... source sequence is illegal/malformed utf-8
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `to_json'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:13:in `record_stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:7:in `block in run!'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/em/timers.rb:56:in `fire'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:20:in `start_em'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:9:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/bin/redmon:78:in `<top (required)>'^C
[root@evest-redis-1 redmon]# service redmon_4567 stop

[SteveDevOps]

we traced through your code and found where it's trying to load stats json from the associated redis key which we identified as:

redmon:redis:<ip address>:6379:stats

We then attempted to clear it out from the redis-cli

del redmon:redis:<ip address>:6379:stats

and restart redmon, but ui fails to load and we re-enter this crash cycle. We are running redmon standalone, no integrations. Other things to try?

[SteveDevOps]

I have been able to determine that the cause of the break in run is slowlog contents. There are some HMGETs included in the slowlog get and these end with (the following for example) :

835510:516818
839761:520340
843986:524044
654885:20391
... (995 more arguments) 

which I believe is what is breaking the json convert. For the moment we are running without slowlog data but would like to know a workaround?

[steelThread]

@SteveDevOps - I haven't really touched this code for a couple of years. I can take a look this weekend and get back with you. Thanks for reporting this.

[SteveDevOps]

Greatly appreciated.. let me know. Thx!