Refactor all systems to use Values; connmon becomes separate

The "connmon" tool can now display stats for SQLAlchemy stats that are sent
to a collectd server via the "network" plugin or through any other means.
Previously, the connmon tool could only display stats for messages that
were sent to a the collectd server by the SQLAlchemy-collectd plugin
itself.

The server side configuration for "connmon" is now separate from that of
the SQLAlchemy-collectd plugin, and the two plugins can run independently
of each other.  This allows for a configuration where many hosts send
SQLAlchemy-collectd messages to local collectd servers for aggregation, and
those servers then pass their messages onto another collectd server, where
the "connmon" tool can provide a view inside the current stats.

In order to achieve this, major refactoring such that the internals now
deal with data in terms of a structure which mirrors the collectd-python
"Values" object is in place, along with a rearchitecture of the connmon
tool such that it now consumes collectd "Values" objects from a particular
collectd server regardless of how those "Values" arrived in that server.

Fixes: #8
Change-Id: Idbbf607ad571f9eb723f2230ec1a3adad33aa536

Author: zzzeek

README.rst

@@ -185,16 +185,6 @@ if you are using gevent, eventlet, asyncio, gunicorn, etc.  threads are your
 friend).
 
 
-TODO
-^^^^
-
-We can add options so that stats are still grouped under parent pids, that
-is instead of using ``<progname>`` as the classifier we use
-``<progname>-<parentpid>``, like ``nova_api-15840`` vs. ``nova_api-4573``, etc.
-Of course we can report on the raw subprocess identifiers as well but this
-doesn't appear to be that useful.
-
-
 
 Server
 ------
@@ -434,8 +424,8 @@ As an added feature, the **connmon** UX has now been integrated into SQLAlchemy-
 This is a console application that displays a "top"-like display of the current
 status of connections.
 
-Using the configuration above, we can add a "monitor" line to our collectd
-server configuration::
+Using the configuration above, we can add a a plugin configuration for the
+connmon server plugin::
 
 
     LoadPlugin python
@@ -451,15 +441,29 @@ server configuration::
             # set to "debug" to show messages received
             loglevel "info"
 
-            # connmon monitor port
+        </Module>
+
+        Import "sqlalchemy_collectd.connmon.plugin"
+        <Module "sqlalchemy_collectd.connmon.plugin">
             monitor "localhost" 25828
+
+            # set to "debug" to show messaging
+            #loglevel "debug"
         </Module>
+
     </Plugin>
 
 We can now run "connmon" on localhost port 25828::
 
     connmon --port 25828
 
+The connmon plugin and command line tool as of version 0.6 works independently
+of the "server" plugin, and may be configured by itself without the server
+plugin being present.  It now consumes sqlalchemy-collectd events not only from
+the local server plugin if present, but from also from any other
+sqlalchemy-collectd messages that are forwarded to the server from elsewhere,
+typically via the "network" plugin.
+
 Screenshot of connmon:
 
 |connmon_screenshot|

examples/helloworld/README.rst

@@ -11,9 +11,9 @@ In one terminal window, run collectd as an interactive application with
 the -f flag::
 
     # cd examples/helloworld
-    # collectd -f -C collectdconsole.conf
+    # collectd -f -C client_plus_connmon.conf
 
-The ``collectdconsole.conf`` includes a relative path to the
+The ``client_plus_connmon.conf`` file includes a relative path to the
 sqlalchemy-collectd checkout as the module path.
 
 Step Two - Run the demo program
@@ -29,7 +29,7 @@ Step Three - watch collectd console
 
 Output looks something like::
 
-    $ collectd -f -C collectdconsole.conf
+    $ collectd -f -C client_plus_connmon.conf
     [2018-02-11 18:29:35] plugin_load: plugin "logfile" successfully loaded.
     [2018-02-11 18:29:35] [info] plugin_load: plugin "write_log" successfully loaded.
     [2018-02-11 18:29:35] [info] plugin_load: plugin "python" successfully loaded.

examples/helloworld/collectdconsole.conf examples/helloworld/client.confexamples/helloworld/collectdconsole.conf renamed to examples/helloworld/client.conf

@@ -17,6 +17,14 @@ LoadPlugin write_log
    Format JSON
 </Plugin>
 
+LoadPlugin network
+<Plugin network>
+  <Server "localhost" "25826">
+    SecurityLevel "None"
+
+  </Server>
+</Plugin>
+
 LoadPlugin python
 <Plugin python>
     ModulePath "../../"
@@ -27,13 +35,19 @@ LoadPlugin python
     <Module "sqlalchemy_collectd.server.plugin">
         listen "localhost" 25827
 
-        # set to "debug" to show messages received
-        loglevel "info"
+        # set to "debug" to show messaging
+        loglevel "debug"
 
-        # connmon monitor port
-        monitor "localhost" 25828
+    </Module>
+
+    Import "sqlalchemy_collectd.connmon.plugin"
+    <Module "sqlalchemy_collectd.connmon.plugin">
+      monitor "localhost" 25828
 
+        # set to "debug" to show messaging
+        #loglevel "debug"
     </Module>
+
 </Plugin>
 
 

examples/helloworld/client_plus_connmon.conf

@@ -0,0 +1,54 @@
+# this config file is the same as client.conf, but also
+# includes the optional "connmon" plugin.
+#
+#
+# Run the server as:
+# collectd -f -C collectdconsole.conf
+#
+# received messages will be dumped as JSON to stdout.
+#
+# then in another window run connmon:
+# connmon
+
+# send logging to STDOUT
+LoadPlugin logfile
+<Plugin logfile>
+      # note "debug" will usually not work
+     LogLevel "info"
+     File STDOUT
+     Timestamp true
+     PrintSeverity true
+</Plugin>
+
+# write messages in JSON format to the log
+LoadPlugin write_log
+<Plugin write_log>
+   Format JSON
+</Plugin>
+
+LoadPlugin python
+<Plugin python>
+    ModulePath "../../"
+    LogTraces true
+
+    Import "sqlalchemy_collectd.server.plugin"
+
+    <Module "sqlalchemy_collectd.server.plugin">
+        listen "localhost" 25827
+
+        # set to "debug" to show messaging
+        loglevel "debug"
+
+    </Module>
+
+    Import "sqlalchemy_collectd.connmon.plugin"
+    <Module "sqlalchemy_collectd.connmon.plugin">
+      monitor "localhost" 25828
+
+        # set to "debug" to show messaging
+        #loglevel "debug"
+    </Module>
+
+</Plugin>
+
+

examples/multiple_collectd/README.rst

@@ -0,0 +1,17 @@
+=========================
+Multiple Collectd Example
+=========================
+
+In this example, multiple collectd servers can relay their messages
+to a central "aggregation" collectd server via the network plugin.  That
+aggregation server then has stats for all the endpoints together which
+can be viewed with collectd clients, and can also be inspected "live" using
+connmon.
+
+* local_client.conf - defines the setup for the initial collectd servers that
+  are close to the SQLAlchemy applications generating stats.   Includes the
+  network plugin instructing them to call out to the central aggregation server.
+
+* receiver.conf - defines the central aggregation server.  Note this includes
+  only the network plugin and the connmon plugin.  SQLAlchemy clients don't
+  connect directly to this server.

examples/multiple_collectd/local_client.conf

@@ -0,0 +1,26 @@
+LoadPlugin network
+<Plugin "network">
+    ## send udp on 25826
+    Server "localhost" "25826"
+</Plugin>
+
+LoadPlugin python
+<Plugin python>
+    ModulePath "../../"
+    LogTraces true
+
+    Import "sqlalchemy_collectd.server.plugin"
+
+    # SQLAlchemy clients connect to port 25827
+    <Module "sqlalchemy_collectd.server.plugin">
+        listen "localhost" 25827
+
+        # set to "debug" to show messaging
+        loglevel "info"
+
+    </Module>
+
+
+</Plugin>
+
+

examples/multiple_collectd/receiver.conf

@@ -0,0 +1,25 @@
+
+LoadPlugin network
+<Plugin "network">
+    ## receive udp on 25826
+    Listen "localhost" "25826"
+</Plugin>
+
+LoadPlugin python
+<Plugin python>
+    ModulePath "../../"
+    LogTraces true
+
+    Import "sqlalchemy_collectd.connmon.plugin"
+
+    # connmon clients connect to 25828
+    <Module "sqlalchemy_collectd.connmon.plugin">
+        monitor "localhost" 25828
+
+        # set to "debug" to show messaging
+        loglevel "info"
+    </Module>
+
+</Plugin>
+
+

sqlalchemy_collectd/client/internal_types.py

@@ -1,3 +1,6 @@
+from __future__ import absolute_import
+
+import logging
 import os
 import socket
 import sys
@@ -7,7 +10,8 @@
 from . import collector
 from . import sender
 from . import worker
-from .. import protocol
+
+log = logging.getLogger("sqlalchemy_collectd")
 
 
 class Plugin(CreateEnginePlugin):
@@ -69,14 +73,18 @@ def start_plugin(
     if progname is None:
         progname = os.path.basename(sys.argv[0])
 
-    sender_ = sender.Sender(hostname, progname)
+    # registry on progname
     collection_target = collector.CollectionTarget.collection_for_name(
         progname
     )
+
+    # unique per Engine
     collector.EngineCollector(collection_target, engine)
 
-    connection = protocol.ClientConnection.for_host_port(
-        collectd_host, collectd_port
+    # registry on host/prog/host/port
+    sender_ = sender.Sender.get_sender(
+        hostname, progname, collectd_host, collectd_port, log
     )
 
-    worker.add_target(connection, collection_target, sender_)
+    # registry on collection_target / sender
+    worker.add_target(collection_target, sender_)