Merge branch 'kpaulisse-release-0-7' into kpaulisse-spec-api-v1

Author: kpaulisse

doc/dev/api/v1/calls/catalog.md

@@ -5,18 +5,111 @@
 `catalog` returns an `OctocatalogDiff::Catalog` object built with the octocatalog-diff compiler, obtained from a Puppet server, or read in from a file. This is analogous to using the `--catalog-only` option with the octocatalog-diff command-line script.
 
 ```
-catalog_obj = OctocatalogDiff::API::V1.catalog(
-
-)
+catalog_obj = OctocatalogDiff::API::V1.catalog(<Hash>)
+#=> OctocatalogDiff::API::V1::Catalog
 ```
 
-## Options
+The return value is an [`OctocatalogDiff::API::V1::Catalog`](/doc/dev/api/v1/objects/catalog.md) object.
+
+For an example, see [catalog-builder-local-files.rb](/examples/api/v1/catalog-builder-local-files.rb).
 
+## Parameters
 
-**NOTE**: Additional options as described in the [options reference](/doc/optionsref.md) may also have an effect on catalog generation.
+The `catalog` method takes one argument, which is a Hash containing parameters.
 
-## Return value
+The list of parameters here is not exhaustive. The `.catalog` method accepts most parameters described in [Configuration](/doc/configuration.md), [Building catalogs instead of diffing catalogs](/doc/advanced-catalog-only.md), and [Command line options reference](/doc/optionsref.md).
 
-The return value is an [`OctocatalogDiff::API::V1::Catalog`](/doc/dev/api/v1/objects/catalog.md) object.
+It is also possible to use the parameters from [OctocatalogDiff::API::V1.config](/doc/dev/api/v1/calls/config.md) for the catalog compilation. Simply combine the hash returned by `.config` with any additional keys, and pass the merged hash to the `.catalog` method.
+
+### Global parameters
+
+#### `:logger` (Logger, Optional)
+
+Debugging and informational messages will be logged to this object as the catalog is built.
+
+If no Logger object is passed, these messages will be silently discarded.
+
+#### `:node` (String, Required)
+
+The node name whose catalog is to be compiled or obtained. This should be the fully qualified domain name that matches the node's name as seen in Puppet.
+
+### Computed catalog parameters
+
+#### `:basedir` (String, Optional)
+
+Directory that contains a git repository with the Puppet code. Use in conjunction with `:to_branch` to specify the branch name that should be checked out.
+
+If your Puppet code is not in a git repository, or you already have the branch checked out via some other process, use `:bootstrapped_to_dir` instead.
+
+#### `:bootstrap_script` (String, Optional)
+
+Path to a script to run after checking out the selected branch of Puppet code from the git repository. See [Bootstrapping your Puppet checkout](/doc/advanced-bootstrap.md) for details.
+
+#### `:bootstrapped_to_dir` (String, Optional)
+
+Directory that is already prepared ("bootstrapped") and can have Puppet run against its contents to compile the catalog.
+
+Often this means that you have done the following to this directory:
+
+- Checked out the necessary branch of Puppet code from your version control system
+- Installed any third party modules (e.g. with librarian-puppet or r10k)
+
+#### `:enc` (String, Optional)
+
+File path to your External Node Classifier.
+
+See [Configuring octocatalog-diff to use ENC](/doc/configuration-enc.md) for details on using octocatalog-diff with an External Node Classifier.
+
+#### `:fact_file` (String, Optional)
+
+Path to the file that contains the facts for the node whose catalog you are going to compile.
+
+Generally, must either specify the fact file, or [configure octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) to retrieve node facts.
+
+#### `:hiera_config` (String, Optional)
+
+Path to the Hiera configuration file (generally named `hiera.yaml`) for your Puppet installation. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration.
+
+#### `:hiera_path` (String, Optional)
+
+Directory within your Puppet installation where Hiera data is stored. Please see [Configuring octocatalog-diff to use Hiera](/doc/configuration-hiera.md) for details on Hiera configuration.
+
+If your Puppet setup is modeled after the [Puppet control repository template](https://github.com/puppetlabs/control-repo), the correct setting for `:hiera_path` is `'hieradata'`.
+
+#### `:puppet_binary` (String, Required)
+
+Path to the Puppet binary on your system.
+
+Please refer to [Configuring octocatalog-diff to use Puppet](/doc/configuration-puppet.md) for details of connecting octocatalog-diff to your Puppet installation.
+
+#### `:puppetdb_url` (String, Optional)
+
+URL to PuppetDB. See [Configuring octocatalog-diff to use PuppetDB](/doc/configuration-puppetdb.md) for instructions on this setting, and other settings that may be needed in your environment.
+
+#### `:to_branch` (String, Optional)
+
+Branch name in git repository to use for Puppet code. This option must be used in conjunction with `:basedir` so that code can be located.
 
 ## Exceptions
+
+The following exceptions may occur during the compilation of a catalog:
+
+- `OctocatalogDiff::Errors::BootstrapError`
+
+  Bootstrapping failed.
+
+- `OctocatalogDiff::Errors::CatalogError`
+
+  Catalog failed to compile. Please note that whenever possible, a `OctocatalogDiff::API::V1::Catalog` object is still constructed for a failed catalog, with `#valid?` returning false.
+
+- `OctocatalogDiff::Errors::GitCheckoutError`
+
+  Git checkout failed.
+
+- `OctocatalogDiff::Errors::PuppetVersionError`
+
+  The version of Puppet could not be determined, generally because the Puppet binary was not found, or does not respond as expected to `puppet version`.
+
+- `OctocatalogDiff::Errors::ReferenceValidationError`
+
+  See [Catalog validation](/doc/advanced-catalog-validation.md).

doc/optionsref.md

@@ -1092,4 +1092,12 @@ parameters are to be checked. (<a href="../lib/octocatalog-diff/cli/options/vali
     </td>
   </tr>
 
-</table>
+</table>
+
+## Using these options in API calls
+
+Most of these options can also be used when making calls to the [API](/doc/dev/api.md).
+
+Generally, parameters for the API are named corresponding to the names of the command line parameters, with dashes (`-`) converted to underscores (`_`). For example, the command line option `--hiera-config` is passed to the API as the symbol `:hiera_config`.
+
+Each of the options above has a link to the source file where it is declared, should you wish to review the specific parameter names and data structures that are being set.

examples/api/v1/catalog-builder-local-files.rb

@@ -0,0 +1,93 @@
+#!/usr/bin/env ruby
+
+# ------------------------------------------------------------------------------------
+# This is a script that demonstrates the use of the OctocatalogDiff::API::V1.catalog
+# method to build a catalog.
+#
+# Run this with:
+#   bundle exec examples/api/v1/catalog-builder-local-files.rb
+# ------------------------------------------------------------------------------------
+
+# Once you have installed the gem, you want this:
+#
+# require 'octocatalog-diff'
+#
+# To make the script run correctly from within the `examples` directory, this locates
+# the `octocatalog-diff` gem directly from this checkout.
+require_relative '../../../lib/octocatalog-diff'
+
+# Here are a few variables we'll use to compile this catalog. To ensure that this is a
+# working script, it will use resources from the test fixtures. You will need adjust these
+# to the actual values in your application.
+
+FACT_FILE = File.expand_path('../../../spec/octocatalog-diff/fixtures/facts/facts.yaml', File.dirname(__FILE__))
+HIERA_CONFIG = File.expand_path('../../../spec/octocatalog-diff/fixtures/repos/default/config/hiera.yaml', File.dirname(__FILE__))
+NODE = 'rspec-node.github.net'.freeze
+PUPPET_REPO = File.expand_path('../../../spec/octocatalog-diff/fixtures/repos/default', File.dirname(__FILE__))
+PUPPET_BINARY = File.expand_path('../../../script/puppet', File.dirname(__FILE__))
+
+# To compile this catalog, call the octocatalog-diff API.
+catalog = OctocatalogDiff::API::V1.catalog(
+  bootstrapped_to_dir: PUPPET_REPO,
+  fact_file: FACT_FILE,
+  hiera_config: HIERA_CONFIG,
+  hiera_path: 'hieradata',
+  node: NODE,
+  puppet_binary: PUPPET_BINARY
+)
+
+# Let's see what kind of an object we got...
+# #=> OctocatalogDiff::API::V1::Catalog
+puts "Object returned from OctocatalogDiff::API::V1.catalog is: #{catalog.class}"
+
+# If it's not valid, the error message will be available.
+unless catalog.valid?
+  puts 'The catalog is not valid.'
+  puts catalog.error_message
+  exit 1
+end
+
+# If it is valid, many other methods may be of interest.
+puts 'The catalog is valid.'
+
+# We can determine which backend was used to build it. With the arguments in this example,
+# the catalog was computed.
+# #=> OctocatalogDiff::Catalog::Computed
+puts "The catalog was built using #{catalog.builder}"
+
+# We can determine which directory was used as the temporary compilation directory.
+# This is only defined for the Computed backend.
+puts "Puppet was run in #{catalog.compilation_dir}"
+
+# We can report on which version of Puppet was run. This is only defined for the
+# Computed backend.
+puts "The version of Puppet used was #{catalog.puppet_version}"
+
+# We can get all resources in the catalog via the .resources method, which returns
+# an Array<Hash>. The Hash comes directly from the catalog structure.
+puts "There is/are #{catalog.resources.count} resource(s) in this catalog"
+catalog.resources.each do |resource|
+  puts "- #{resource['type']} - #{resource['title']}"
+end
+
+# You can also locate a resource by its type and title. We'll choose an element at
+# random from the array. We will then locate it via the `.resource` method.
+selected_resource = catalog.resources.sample
+puts "Randomly selected type=#{selected_resource['type']} title=#{selected_resource['title']}"
+
+param = { type: selected_resource['type'], title: selected_resource['title'] }
+looked_up_resource = catalog.resource(param)
+puts "Looked up using catalog.resource: type=#{looked_up_resource['type']}, title=#{looked_up_resource['title']}"
+
+if selected_resource == looked_up_resource
+  puts 'The resources are equal!'
+else
+  # If this happens, it's a bug - please report it to us!
+  puts 'The resources do not match!'
+end
+
+# If we want the JSON representation of the catalog, we can get that too. You'd
+# normally want to write this out to a file. We'll just print the first 80 characters.
+json_text = catalog.to_json
+puts "The JSON representation of the catalog is #{json_text.length} characters long"
+puts json_text[0..80]

lib/octocatalog-diff/catalog/computed.rb

@@ -89,7 +89,15 @@ def environment
       def cleanup_checkout_dir(checkout_dir, logger)
         return unless File.directory?(checkout_dir)
         logger.debug("Cleaning up temporary directory #{checkout_dir}")
-        FileUtils.remove_entry_secure checkout_dir
+        # Sometimes this seems to break when handling the recursive removal when running under
+        # a parallel environment. Trap and ignore the errors here if we don't care about them.
+        begin
+          FileUtils.remove_entry_secure checkout_dir
+        rescue Errno::ENOTEMPTY, Errno::ENOENT => exc
+          # :nocov:
+          logger.debug "cleanup_checkout_dir(#{checkout_dir}) logged #{exc.class} - this can be ignored"
+          # :nocov:
+        end
       end
 
       # Private method: Bootstrap a directory

lib/octocatalog-diff/cli.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
-require_relative 'api/v1/catalog-compile'
-require_relative 'api/v1/catalog-diff'
+require_relative 'api/v1'
 require_relative 'catalog-util/cached_master_directory'
 require_relative 'errors'
 require_relative 'util/catalogs'
@@ -111,8 +110,8 @@ def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {})
       end
 
       # Compile catalogs and do catalog-diff
-      catalog_diff = OctocatalogDiff::API::V1::CatalogDiff.catalog_diff(options.merge(logger: logger))
-      diffs = catalog_diff.diffs.map(&:raw)
+      catalog_diff = OctocatalogDiff::API::V1.catalog_diff(options.merge(logger: logger))
+      diffs = catalog_diff.diffs
 
       # Display diffs
       printer_obj = OctocatalogDiff::Cli::Printer.new(options, logger)
@@ -159,19 +158,23 @@ def self.setup_logger(logger, options, argv_save)
     # Compile the catalog only
     def self.catalog_only(logger, options)
       opts = options.merge(logger: logger)
-      to_catalog = OctocatalogDiff::API::V1::CatalogCompile.catalog(opts).raw
+      to_catalog = OctocatalogDiff::API::V1.catalog(opts)
 
       # If the catalog compilation failed, an exception would have been thrown. So if
       # we get here, the catalog succeeded. Dump the catalog to the appropriate place
       # and exit successfully.
       if options[:output_file]
-        File.open(options[:output_file], 'w') { |f| f.write(to_catalog.catalog_json) }
+        File.open(options[:output_file], 'w') { |f| f.write(to_catalog.to_json) }
         logger.info "Wrote catalog to #{options[:output_file]}"
       else
-        puts to_catalog.catalog_json
+        puts to_catalog.to_json
       end
-      return [OctocatalogDiff::Catalog.new(backend: :noop), to_catalog] if options[:RETURN_DIFFS] # For integration testing
-      EXITCODE_SUCCESS_NO_DIFFS
+
+      return EXITCODE_SUCCESS_NO_DIFFS unless options[:RETURN_DIFFS] # For integration testing
+      # :nocov:
+      dummy_catalog = OctocatalogDiff::API::V1::Catalog.new(OctocatalogDiff::Catalog.new(backend: :noop))
+      [dummy_catalog, to_catalog]
+      # :nocov:
     end
 
     # --bootstrap-then-exit command

rake/templates/optionsref.erb

@@ -34,3 +34,11 @@ Usage: octocatalog-diff [command line options]
   </tr>
 <% end %>
 </table>
+
+## Using these options in API calls
+
+Most of these options can also be used when making calls to the [API](/doc/dev/api.md).
+
+Generally, parameters for the API are named corresponding to the names of the command line parameters, with dashes (`-`) converted to underscores (`_`). For example, the command line option `--hiera-config` is passed to the API as the symbol `:hiera_config`.
+
+Each of the options above has a link to the source file where it is declared, should you wish to review the specific parameter names and data structures that are being set.