Merge branch 'kpaulisse-arbitrary-command-line-pass' into kpaulisse-validate-references-alias

Author: Kevin Paulisse

doc/advanced-environments.md

@@ -0,0 +1,93 @@
+# Environment setup
+
+When building a catalog, the default behavior of `octocatalog-diff` is to:
+
+1. Create a temporary directory
+2. Create a symlink from `<temporary directory>/environments/production` to the checkout of your code
+3. Run Puppet using `environment=production`
+
+If you are using environment names to control the behavior of Puppet, this default behavior may not be suitable. In that case you can invoke the alternate behavior: preserving environments.
+
+## Command line options
+
+### Preserving the environments
+
+When you supply the command line argument `--preserve-environments` (or set `settings[:preserve_environments] = true` in your [configuration file](/doc/configuration.md)), `octocatalog-diff` will instead do the following:
+
+1. Create a temporary directory
+2. Create the following symlinks from `<temporary directory>` to the corresponding directories in the checkout of your code:
+
+  - `environments`
+  - `manifests`
+  - `modules`
+
+3. Run Puppet using an environment you specify via the command line
+
+Note that you must have set `--preserve-environments` in order for the `--environment` and/or `--create-symlinks` options (described below) to have any effect.
+
+### Changing the environment
+
+If you wish to use an environment name other than `production` you can use the `--environment <environment_name>` command line option. This will set the environment for both the `to` and `from` compiles.
+
+```
+octocatalog-diff ... --preserve-environments --environment some-env-name
+```
+
+If you need to specify different environments for the `to` and `from` compiles, you can use `--to-environment <environment_name>` and `--from-environment <environment_name>`.
+
+```
+octocatalog-diff ... --preserve-environments --to-environment first-environment --from-environment second-environment
+```
+
+### Controlling symlinks that are created
+
+Within the temporary directory, the `environments` symlink will always be created.
+
+By default, `manifests` and `modules` will also be created from the temporary directory to the corresponding directories in your Puppet code base. If you need to customize the symlinks that are created, you can use the `--create-symlinks <dir1>,<dir2>,...` to list the symlinks that you need.
+
+For example, if you have some code stored in a directory called `modules` and more code stored in a directory called `site`, you could do the following to create the symlinks as desired:
+
+```
+octocatalog-diff ... --preserve-environments --create-symlinks manifests,modules,site
+```
+
+## Examples
+
+Consider that your Puppet code base is organized as follows:
+
+```
+- /opt/puppet
+  - environments
+    - old
+      - environment.conf
+      - manifests
+        - site.pp
+      - modules
+        - module_zero
+    - new
+      - environment.conf
+      - manifests
+        - site.pp
+      - modules
+        - module_zero
+  - modules
+    - module_one
+    - module_two
+  - site
+    - module_three
+    - module_four
+```
+
+To calculate the difference between the "old" and "new" environment, you could use:
+
+```
+octocatalog-diff \
+  --bootstrapped-from-dir /opt/puppet \
+  --bootstrapped-to-dir /opt/puppet \
+  --preserve-environments \
+  --from-environment old \
+  --to-environment new \
+  --create-symlinks modules,site
+```
+
+(Note that `--bootstrapped-from-dir` and `--bootstrapped-to-dir` are used to specify the directory path to your code, and `-t` and `-f` are not used. That's because the difference in the catalog is derived from the environment used, and not the branch from a git repository.)

doc/advanced.md

@@ -22,6 +22,7 @@ See also:
 - [Puppet Enterprise node classification service](/doc/advanced-pe-enc.md)
 - [Using `octocatalog-diff` without git](/doc/advanced-using-without-git.md)
 - [Catalog validation](/doc/advanced-catalog-validation.md)
+- [Environment setup](/doc/advanced-environment.md)
 
 ### Controlling output
 

doc/optionsref.md

@@ -127,6 +127,22 @@ Usage: octocatalog-diff [command line options]
         --no-ignore-tags             Disable ignoring based on tags
         --ignore-tags STRING1[,STRING2[,...]]
                                      Specify tags to ignore
+        --[no-]preserve-environments Enable or disable environment preservation
+        --environment STRING         Environment for catalog compilation globally
+        --to-environment STRING      Environment for catalog compilation for the to branch
+        --from-environment STRING    Environment for catalog compilation for the from branch
+        --create-symlinks STRING1[,STRING2[,...]]
+                                     Symlinks to create globally
+        --to-create-symlinks STRING1[,STRING2[,...]]
+                                     Symlinks to create for the to branch
+        --from-create-symlinks STRING1[,STRING2[,...]]
+                                     Symlinks to create for the from branch
+        --command-line STRING1[,STRING2[,...]]
+                                     Command line arguments globally
+        --to-command-line STRING1[,STRING2[,...]]
+                                     Command line arguments for the to branch
+        --from-command-line STRING1[,STRING2[,...]]
+                                     Command line arguments for the from branch
         --pass-env-vars VAR1[,VAR2[,...]]
                                      Environment variables to pass
         --[no-]suppress-absent-file-details
@@ -831,6 +847,21 @@ https://your-pe-console-server:4433/classifier-api (<a href="../lib/octocatalog-
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--preserve-environments
+--no-preserve-environments </code></pre>
+    </td>
+    <td valign=top>
+      Enable or disable environment preservation
+    </td>
+    <td valign=top>
+      Preserve the `environments` directory from the repository when compiling the catalog. Likely
+requires some combination of `--to-environment`, `--from-environment`, and/or `--create-symlinks`
+to work correctly. (<a href="../lib/octocatalog-diff/catalog-diff/cli/options/preserve_environments.rb">preserve_environments.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--puppetdb-api-version N</code></pre>

lib/octocatalog-diff/catalog-diff/cli/catalogs.rb

@@ -198,7 +198,7 @@ def add_parallel_result(result, parallel_catalog_obj, key_task_tuple)
               # and remove the compilation directory (which is a tmpdir) to reveal only the relative
               # path to the files involved.
               dir = catalog.compilation_dir || ''
-              dir_regex = Regexp.new(Regexp.escape(dir) + '/environments/production/')
+              dir_regex = Regexp.new(Regexp.escape(dir) + '/environments/[^/]+/')
               error_display = catalog.error_message.split("\n").map do |line|
                 line.sub(/^Error:/, '[Puppet Error]').gsub(dir_regex, '')
               end.join("\n")

lib/octocatalog-diff/catalog-diff/cli/options/command_line.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+# Provide additional command line flags to set when running Puppet to compile catalogs.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::CatalogDiff::Cli::Options::Option.newoption(:command_line) do
+  has_weight 510
+
+  def parse(parser, options)
+    OctocatalogDiff::CatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      cli_name: 'command-line',
+      option_name: 'command_line',
+      desc: 'Command line arguments',
+      datatype: []
+    )
+  end
+end

lib/octocatalog-diff/catalog-diff/cli/options/create_symlinks.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+# Specify which directories from the base should be symlinked into the temporary compilation
+# environment. This is useful only in conjunction with `--preserve-environments`.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::CatalogDiff::Cli::Options::Option.newoption(:create_symlinks) do
+  has_weight 503
+
+  def parse(parser, options)
+    OctocatalogDiff::CatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      cli_name: 'create-symlinks',
+      option_name: 'create_symlinks',
+      desc: 'Symlinks to create',
+      datatype: []
+    )
+  end
+end

lib/octocatalog-diff/catalog-diff/cli/options/environment.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+# Specify the environment to use when compiling the catalog. This is useful only in conjunction
+# with `--preserve-environments`.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::CatalogDiff::Cli::Options::Option.newoption(:environment) do
+  has_weight 502
+
+  def parse(parser, options)
+    OctocatalogDiff::CatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      cli_name: 'environment',
+      option_name: 'environment',
+      desc: 'Environment for catalog compilation'
+    )
+  end
+end

lib/octocatalog-diff/catalog-diff/cli/options/preserve_environments.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+# Preserve the `environments` directory from the repository when compiling the catalog. Likely
+# requires some combination of `--to-environment`, `--from-environment`, and/or `--create-symlinks`
+# to work correctly.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::CatalogDiff::Cli::Options::Option.newoption(:preserve_environments) do
+  has_weight 501
+
+  def parse(parser, options)
+    parser.on('--[no-]preserve-environments', 'Enable or disable environment preservation') do |x|
+      options[:preserve_environments] = x
+    end
+  end
+end

lib/octocatalog-diff/catalog-util/builddir.rb

@@ -48,7 +48,7 @@ def initialize(options = {}, logger = nil)
         @facts_terminus = options.fetch(:facts_terminus, 'yaml')
 
         create_structure
-        install_directory_symlink(logger, options[:basedir])
+        create_symlinks(logger)
 
         # These configurations are optional. Don't call the methods if parameters are nil.
         unless options[:puppetdb_url].nil?
@@ -64,12 +64,36 @@ def initialize(options = {}, logger = nil)
 
       # Create common structure
       def create_structure
-        %w(environments facts var var/ssl var/yaml var/yaml/facts).each do |dir|
+        %w(facts var var/ssl var/yaml var/yaml/facts).each do |dir|
           Dir.mkdir(File.join(@tempdir, dir))
           FileUtils.chmod 0o755, File.join(@tempdir, dir)
         end
       end
 
+      # Create symlinks.
+      #
+      # If the `--preserve-environments` option is used, the `environments` directory, plus `modules` and
+      # `manifests` symlinks are created. Otherwise, `environments/production` is pointed at the base
+      # directory.
+      #
+      # @param logger [Logger] Logger object
+      def create_symlinks(logger = nil)
+        if @options[:preserve_environments]
+          install_directory_symlink(logger, File.join(@options[:basedir], 'environments'), 'environments')
+          @options.fetch(:create_symlinks, %w(modules manifests)).each do |x|
+            install_directory_symlink(logger, File.join(@options[:basedir], x), x)
+          end
+        else
+          if @options[:environment]
+            logger.warn '--environment is ignored unless --preserve-environments is used' unless logger.nil?
+          end
+          if @options[:create_symlinks]
+            logger.warn '--create-symlinks is ignored unless --preserve-environments is used' unless logger.nil?
+          end
+          install_directory_symlink(logger, @options[:basedir])
+        end
+      end
+
       # Install puppetdb.conf file in temporary directory
       # @param server_urls [String] String for server_urls in puppetdb.conf
       # @param server_url_timeout [Fixnum] Value for server_url_timeout in puppetdb.conf
@@ -150,14 +174,20 @@ def install_fact_file(logger, options)
 
       # Install symbolic link to puppet environment
       # @param dir [String] Directory to link to
-      def install_directory_symlink(logger, dir)
+      # @param target [String] Where the symlink is created, relative to tempdir
+      def install_directory_symlink(logger, dir, target = 'environments/production')
         raise ArgumentError, "Called install_directory_symlink with #{dir.class} argument" unless dir.is_a?(String)
         raise Errno::ENOENT, "Specified directory #{dir} doesn't exist" unless File.directory?(dir)
+        symlink_target = File.join(@tempdir, target)
+
+        if target =~ %r{/}
+          parent_dir = File.dirname(symlink_target)
+          FileUtils.mkdir_p parent_dir
+        end
 
-        environment_symlink = File.join(@tempdir, 'environments', 'production')
-        FileUtils.rm_f environment_symlink if File.exist?(environment_symlink)
-        FileUtils.symlink dir, environment_symlink
-        logger.debug("Symlinked #{environment_symlink} -> #{dir}")
+        FileUtils.rm_f symlink_target if File.exist?(symlink_target)
+        FileUtils.symlink dir, symlink_target
+        logger.debug("Symlinked #{symlink_target} -> #{dir}")
       end
 
       # Install ENC

lib/octocatalog-diff/catalog-util/command.rb

@@ -73,9 +73,14 @@ def puppet_command
           --no-ca
           --color=false
           --config_version="/bin/echo catalogscript"
-          --environment=production
         )
 
+        # Add environment - only make this variable if preserve_environments is used.
+        # If preserve_environments is not used, the hard-coded 'production' here matches
+        # up with the symlink created under the temporary directory structure.
+        environ = @options[:preserve_environments] ? @options.fetch(:environment, 'production') : 'production'
+        cmdline << "--environment=#{Shellwords.escape(environ)}"
+
         # For people who aren't running hiera, a hiera-config will not be generated when @options[:hiera_config]
         # is nil. For everyone else, the hiera config was generated/copied/munged in the 'builddir' class
         # and was installed into the compile directory and named hiera.yaml.
@@ -90,9 +95,51 @@ def puppet_command
         cmdline << "--ssldir=#{Shellwords.escape(File.join(@compilation_dir, 'var', 'ssl'))}"
         cmdline << "--confdir=#{Shellwords.escape(@compilation_dir)}"
 
+        # Other parameters provided by the user
+        override_and_append_commandline_with_user_supplied_arguments(cmdline)
+
         # Return full command
         cmdline.join(' ')
       end
+
+      private
+
+      # Private: Mutate the command line with arguments that were passed directly from the
+      # user. This appends new arguments and overwrites existing arguments.
+      # @param cmdline [Array] Existing command line - mutated by this method
+      def override_and_append_commandline_with_user_supplied_arguments(cmdline)
+        return unless @options[:command_line].is_a?(Array)
+
+        @options[:command_line].each do |opt|
+          # Validate format: Accept '--key=value' or '--key' only.
+          unless opt =~ /\A--([^=\s]+)(=.+)?\z/
+            raise ArgumentError, "Command line option '#{opt}' does not match format '--SOME_OPTION=SOME_VALUE'"
+          end
+          key = Shellwords.escape(Regexp.last_match(1))
+          val = Regexp.last_match(2)
+          val.sub!(/\A=/, '') if val.is_a?(String)
+
+          # Determine if command line already contains this setting. If yes, the setting provided
+          # here should override. If no, then append to the commandline.
+          new_setting = val.nil? ? "--#{key}" : "--#{key}=#{Shellwords.escape(val)}"
+          ind = key_position(cmdline, key)
+          if ind.nil?
+            cmdline << new_setting
+          else
+            cmdline[ind] = new_setting
+          end
+        end
+      end
+
+      # Private: Determine if the key (given by --key) is already defined in the
+      # command line. Returns nil if it is not already defined, otherwise returns
+      # the index.
+      # @param cmdline [Array] Existing command line
+      # @param key [String] Key to look up
+      # @return [Fixnum] Index of where key is defined (nil if undefined)
+      def key_position(cmdline, key)
+        cmdline.index { |x| x == "--#{key}" || x =~ /\A--#{key}=/ }
+      end
     end
   end
 end