diff --git a/AGENTS.md b/AGENTS.md index ddc17f3..f1d729c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,7 +18,12 @@ bundle exec rake console # Interactive console ## Architecture -Single-file library (`lib/simple-rss.rb`) using regex-based XML parsing for flexibility with malformed feeds. +The core library (`lib/simple-rss.rb`) uses regex-based XML parsing for flexibility +with malformed feeds. `lib/simple-rss/xml_element.rb` holds shared XML tokenization +and scoped element metadata. The optional normalized view lives in +`entry_normalizer.rb` and the format-independent `normalized_entry.rb` value object +under `lib/simple-rss/`. Preserve the existing raw parser and serialization +contracts when extending normalization. **Tag Syntax** (extend via `SimpleRSS.item_tags <<`): - `tag` - simple element extraction diff --git a/CHANGELOG.md b/CHANGELOG.md index 4816595..bb2a9ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ repository begins with a 1.1 import, so earlier releases are not reconstructed. ## Unreleased +- Add `normalized_entries`, an optional immutable RSS/Atom view with consistent + article URLs, separate publication/update dates, typed content and summaries, + category terms, authors, and associated attachment metadata. Preserve raw + items, XML, field sources, and normalization issues without changing existing + access or serialization. Support per-call content/keyword mappings such as + `full-text`, with no global tag changes. Resolve relative URLs using `xml:base` + and a supplied or fetched source URL; fetch now retains the final response URL + and resolves relative redirects. ([#59](https://github.com/cardmagic/simple-rss/issues/59)) - Parse Atom 1.0 category terms from attributes, so scalar category access and `array_tags: [:category]` work with `items_by_category`. Preserve source order and duplicates, skip missing or blank terms, and resolve namespace declarations diff --git a/README.md b/README.md index 4784a04..1fc3bf7 100644 --- a/README.md +++ b/README.md @@ -176,6 +176,126 @@ by the same date rules and keeps the newest entry for each identity. Equal dates keep the first occurrence. Entries without an identity remain at the end in input order, regardless of their dates. +### Normalized Entries + +Use `normalized_entries` when an importer or digest should handle RSS and Atom +through the same fields: + +```ruby +feed = SimpleRSS.parse(xml, source_url: "https://example.com/feed.xml") +entry = feed.normalized_entries.first + +entry.identifier # Publisher's opaque ID/GUID, or nil +entry.url # Article URL, preferring an Atom HTML alternate +entry.published_at # Time or nil; never filled from an update timestamp +entry.updated_at # Time or nil +entry.content_html # Full HTML content, when supplied +entry.content_text # Full plain text, when supplied +entry.summary # Separate synopsis; see summary_type (:html or :text) +entry.categories # Nonempty terms, unique in first-seen order +entry.attachments # Associated URL, media_type, size_in_bytes, duration_in_seconds +entry.authors # Name, email, URL, and source metadata +entry.issues # Inspect missing bases, invalid dates/numbers, unsupported content +entry.raw # Frozen copy of the original item hash +entry.raw_xml # Original entry XML, including unconfigured extensions +entry.field_sources # Which XML tag supplied each normalized scalar field +``` + +This is an optional, immutable view of the original XML. `items`, `entries`, +iteration, custom tags, `latest`, `merge`, `diff`, and all existing serialization +methods keep their current behavior. Normalization neither changes global tag +configuration nor mutates or freezes raw items. Each call returns fresh snapshots; +raw item edits are preserved in `raw` but do not rewrite the XML-derived fields. +Reordering or deduplicating `items` preserves the association with each item's +original XML. Inserting an unrelated hash into `items` cannot provide that source +and raises `SimpleRSSError` when normalized. + +| Normalized field | RSS mapping | Atom 1.0 mapping | +| --- | --- | --- | +| `identifier` | `guid`, without URL decoding or a generated fallback | `id`, without URL decoding or a generated fallback | +| `url` | Item `link`, then an Atom alternate extension | Alternate `link` (`rel` defaults to alternate); HTML/XHTML first, untyped second, other alternatives last; never a self/API fallback | +| `published_at` | First parseable `pubDate`, then Dublin Core `date` | `published` | +| `updated_at` | Atom `updated` extension, then `modified` | `updated` | +| `content_html` | Explicit mapping, then `content:encoded`, then an HTML/XHTML Atom content extension | Explicit mapping, then `content:encoded` if supplied, then HTML/XHTML `content` | +| `content_text` | Explicit mapping or a plain-text Atom content extension | Explicit mapping, then plain-text `content` | +| `summary` | `description`, treated as HTML-capable synopsis | `summary`, honoring text/HTML/XHTML type | +| `categories` | Repeated category text plus Dublin Core subjects and unsplit Media RSS/iTunes keywords | Category terms plus the same recognized extensions | +| `attachments` | Each `enclosure` and Media RSS `content` (direct or in a direct Media RSS group) | Each enclosure `link` and the same Media RSS elements | +| `authors` | Each `author` as its unparsed email value; Dublin Core `creator` as a name | Entry authors, otherwise source authors, otherwise feed authors | + +Namespaces are resolved by their declared URI; standard extension prefixes may +vary. Ordinary metadata must be a direct child of its entry. Nested article +markup, comments, and unrelated source metadata cannot replace entry fields. +Missing scalars are `nil`; missing collections are empty arrays. Dates use Ruby's +permissive `Time.parse`: invalid values produce `nil` and an issue, with the source +retained. `effective_at` returns `published_at || updated_at` for a normalized +entry; existing raw-item date ordering is unchanged. + +URLs use the applicable ancestor and element `xml:base` declarations, resolved +against `source_url`. `fetch` always supplies the final response URL after +redirects, even if its options include a different or nil `source_url`; +`normalized_entries(source_url: "https://example.com/feed.xml")` can override it +for one call. Relative redirects are resolved against the current request URL. +Without a usable base, relative values are preserved and reported in `issues`. +Invalid URI syntax is likewise preserved with an issue. Identifiers are never +resolved as URLs. Normalization performs no HTTP requests or article scraping. + +Content is not sanitized, and HTML is never implicitly stripped into plain text. +XML entities are decoded once outside CDATA; CDATA content is retained literally. +Atom XHTML drops its enclosing XHTML `div` and removes XHTML namespace prefixes +from element names, preserving the original in `raw_xml`. `content_base_url` +provides the effective base for the selected HTML content (or plain text when +HTML is absent); links inside content are not rewritten. External Atom content +is exposed as `content_url` without fetching it. Unsupported content types remain +in raw data with an issue. Render HTML only through your application's usual +sanitization policy. + +`category_details` preserves duplicate category records, labels, schemes/domains, +and their raw attributes/content even when `categories` removes repeated terms. +`links` preserves all entry link records with resolved `url`, `rel`, `media_type`, +and raw metadata. Each attachment includes `source` and `raw`; invalid numeric +values stay there and produce a `nil` normalized number plus an issue, rather +than becoming zero. An item-level iTunes duration applies only when there is one +attachment; its original element is then available as `raw_duration`. + +Custom content and keyword rules belong to an individual normalization call: + +```ruby +entries = feed.normalized_entries(mappings: { + content_html: "full-text", + content_text: "{urn:example:content}plain", + categories: [ + { tag: "dc:subject", separator: ";" }, + { tag: "media:keywords", separator: "," } + ] +}) +``` + +Content mappings select the first nonempty direct element and take precedence +over the defaults for that representation. A selector is an exact XML qualified +name or `{namespace-uri}local-name`, not XPath. Hyphenated names such as +`full-text` need no generated accessor or global tag registration. Category +mappings override the selected element's default term extraction; a separator +is literal and optional. Unmapped subject/keyword strings are never guessed to +be comma- or semicolon-delimited. Unknown mapping keys and malformed mapping +options raise `ArgumentError`. + +For migration, replace format-specific expressions such as +`item[:link_alternate] || item[:link]` with `entry.url`, while retaining +`entry.raw` for existing custom fields. The runnable [digest example](examples/digest.rb) +reads either format without testing which one it received: + +```bash +ruby -Ilib examples/digest.rb test/data/normalized_rss.xml +ruby -Ilib examples/digest.rb test/data/normalized_atom.xml +``` + +The mapping follows the [Atom specification](https://www.rfc-editor.org/rfc/rfc4287.html), +[RSS specification](https://www.rssboard.org/rss-specification), and +[XML Base rules](https://www.w3.org/TR/xmlbase/). It is a tolerant extraction view, +not a standards validator. JSON Feed parsing is tracked separately in +[#60](https://github.com/cardmagic/simple-rss/issues/60). + ### JSON Serialization ```ruby diff --git a/Rakefile b/Rakefile index cf4315d..4825949 100644 --- a/Rakefile +++ b/Rakefile @@ -14,7 +14,7 @@ RUBY_FORGE_USER = ENV["RUBY_FORGE_USER"] || "cardmagic" RELEASE_NAME = "#{PKG_NAME}-#{PKG_VERSION}".freeze PKG_FILES = FileList[ - "lib/*", "bin/*", "test/**/*", "[A-Z]*", "Rakefile", "html/**/*" + "lib/**/*", "bin/*", "examples/**/*", "test/**/*", "[A-Z]*", "Rakefile", "html/**/*" ] desc "Default Task" diff --git a/examples/digest.rb b/examples/digest.rb new file mode 100644 index 0000000..f9af0de --- /dev/null +++ b/examples/digest.rb @@ -0,0 +1,19 @@ +require "simple-rss" +require "optparse" + +options = {} +arguments = OptionParser.new do |parser| + parser.banner = "Usage: ruby -Ilib examples/digest.rb [--source-url URL] FEED_FILE..." + parser.on("--source-url URL", "Base URL for relative feed references") { |url| options[:source_url] = url } +end +arguments.parse! +abort arguments.to_s if ARGV.empty? + +entries = ARGV.flat_map do |path| + File.open(path) { |source| SimpleRSS.parse(source, options).normalized_entries } +end + +entries.sort_by { |entry| entry.effective_at ? -entry.effective_at.to_r : Float::INFINITY }.each do |entry| + puts [entry.title || entry.identifier || "Untitled", entry.effective_at&.getutc&.iso8601, entry.url, entry.content_text].compact.join("\n") + puts +end diff --git a/lib/simple-rss.rb b/lib/simple-rss.rb index 8255d95..21719f9 100644 --- a/lib/simple-rss.rb +++ b/lib/simple-rss.rb @@ -17,11 +17,13 @@ class SimpleRSS # rubocop:disable Metrics/ClassLength # @rbs @options: Hash[Symbol, untyped] # @rbs @etag: String? # @rbs @last_modified: String? + # @rbs @entry_contexts: Hash[Hash[Symbol, untyped], Hash[Symbol, untyped]] attr_reader :items #: Array[Hash[Symbol, untyped]] attr_reader :source #: String attr_reader :etag #: String? attr_reader :last_modified #: String? + attr_reader :source_url #: String? alias entries items #: Array[Hash[Symbol, untyped]] @@feed_tags = %i[ @@ -63,6 +65,9 @@ def initialize(source, options = {}) @items = [] #: Array[Hash[Symbol, untyped]] @options = {} #: Hash[Symbol, untyped] @options.update(options) + @source_url = options[:source_url] + @entry_contexts = {} #: Hash[Hash[Symbol, untyped], Hash[Symbol, untyped]] + @entry_contexts.compare_by_identity parse end @@ -73,6 +78,18 @@ def channel end alias feed channel + # @rbs (?source_url: String?, ?mappings: Hash[Symbol, untyped]) -> Array[NormalizedEntry] + def normalized_entries(source_url: nil, mappings: {}) + EntryNormalizer.validate_mappings(mappings) + feed_authors = normalized_feed_authors + items.map do |item| + context = @entry_contexts[item] || raise(SimpleRSSError, "Cannot normalize an item without its original XML source") + element = XmlElement.new(context[:name], context[:attributes], context[:content], context[:parent]) + EntryNormalizer.new(element, item, source_url: source_url || @source_url, mappings: mappings, + raw_xml: context[:xml], feed_authors: feed_authors).entry + end + end + # Iterate over all items in the feed # # @rbs () { (Hash[Symbol, untyped]) -> void } -> self @@ -280,14 +297,14 @@ def fetch(url, options = {}) require "uri" uri = URI.parse(url) - response = perform_fetch(uri, options) + response, final_uri = perform_fetch(uri, options) return nil if response.is_a?(Net::HTTPNotModified) raise SimpleRSSError, "HTTP #{response.code}: #{response.message}" unless response.is_a?(Net::HTTPSuccess) body = response.body.force_encoding(Encoding::UTF_8) - feed = parse(body, options) + feed = parse(body, options.merge(source_url: final_uri.to_s)) feed.instance_variable_set(:@etag, response["ETag"]) feed.instance_variable_set(:@last_modified, response["Last-Modified"]) feed @@ -301,7 +318,7 @@ def perform_fetch(uri, options) request = build_request(uri, options) response = http.request(request) - handle_redirect(response, options) || response + handle_redirect(response, uri, options) || [response, uri] end # @rbs (untyped, Hash[Symbol, untyped]) -> untyped @@ -334,8 +351,8 @@ def build_request(uri, options) request end - # @rbs (untyped, Hash[Symbol, untyped]) -> untyped - def handle_redirect(response, options) + # @rbs (untyped, untyped, Hash[Symbol, untyped]) -> untyped + def handle_redirect(response, uri, options) return nil unless response.is_a?(Net::HTTPRedirection) return nil if options[:follow_redirects] == false @@ -346,7 +363,7 @@ def handle_redirect(response, options) raise SimpleRSSError, "Too many redirects" if redirects > 5 new_options = options.merge(_redirects: redirects) - perform_fetch(URI.parse(location), new_options) + perform_fetch(URI.join(uri.to_s, location), new_options) end end @@ -358,6 +375,15 @@ def handle_redirect(response, options) private + # @rbs () -> Array[XmlElement] + def normalized_feed_authors + document = XmlElement.new("", "", @source, { namespaces: {}, base_urls: [] }) + feed = document.children.find { |element| element.matches?("feed", ATOM_NAMESPACE) } + return [] unless feed + + feed.children.select { |element| element.matches?("author", ATOM_NAMESPACE) } + end + # @rbs () -> void def parse raise SimpleRSSError, "Poorly formatted feed" unless @source =~ %r{<(channel|feed).*?>.*?}mi @@ -400,13 +426,18 @@ def parse end # RSS items' title, link, and description - namespace_contexts = entry_namespaces + namespace_contexts = entry_contexts entry_pattern = %r{<(rss:|atom:)?(item|entry)([\s][^>]*)?>(.*?)}mi position = 0 while (match = entry_pattern.match(@source, position)) position = match.end(0) item = {} #: Hash[Symbol, untyped] - namespaces = namespace_contexts[match.begin(0)] + parent_context = namespace_contexts[match.begin(0)] + namespaces = parent_context && parent_context[:namespaces].merge(namespace_attributes(match[3].to_s)) + @entry_contexts[item] = { + name: "#{match[1]}#{match[2]}", attributes: match[3].to_s, content: match[4].to_s, + parent: parent_context || { namespaces: {}, base_urls: [] }, xml: match[0] + } @@item_tags.each do |tag| next if tag.to_s.strip.empty? @@ -498,37 +529,7 @@ def entry_link_attributes(content) # @rbs (String) -> Array[[String, String, String?]] def child_elements(content) - elements = [] #: Array[[String, String, String?]] - current_element = nil #: [String, String, String?]? - depth = 0 - body_start = 0 - position = 0 - - while (token = XML_TAG_PATTERN.match(content, position)) - position = token.end(0) - attributes = token[3] - next unless attributes - - if token[1] == "/" - depth = [depth - 1, 0].max - if depth.zero? && current_element - current_element[2] = content[body_start...token.begin(0)] - current_element = nil - end - next - end - - self_closing = attributes.rstrip.end_with?("/") - if depth.zero? - element = [token[2].to_s, attributes, nil] #: [String, String, String?] - elements << element - current_element = element unless self_closing - body_start = token.end(0) - end - depth += 1 unless self_closing - end - - elements + XmlElement.child_elements(content) end # @rbs (Hash[Symbol, untyped], String, Hash[String, String], String?) -> void @@ -556,10 +557,10 @@ def parse_category_tag(item, content, namespaces, entry_namespace) item[:category] = array_tag?(:category) ? values : values.first end - # @rbs () -> Hash[Integer, Hash[String, String]] - def entry_namespaces - contexts = {} #: Hash[Integer, Hash[String, String]] - scopes = [{}] #: Array[Hash[String, String]] + # @rbs () -> Hash[Integer, Hash[Symbol, untyped]] + def entry_contexts + contexts = {} #: Hash[Integer, Hash[Symbol, untyped]] + scopes = [{ namespaces: {}, base_urls: [] }] #: Array[Hash[Symbol, untyped]] position = 0 while (token = XML_TAG_PATTERN.match(@source, position)) @@ -572,10 +573,17 @@ def entry_namespaces next end - namespaces = scopes.fetch(-1).merge(namespace_attributes(attributes)) + parent = scopes.fetch(-1) tag = token[2].to_s.split(":").last - contexts[token.begin(0).to_i] = namespaces if %w[item entry].include?(tag&.downcase) - scopes << namespaces unless attributes.rstrip.end_with?("/") + contexts[token.begin(0).to_i] = parent if %w[item entry].include?(tag&.downcase) + base_url = xml_attributes(attributes)["xml:base"] + base_urls = parent[:base_urls].dup + base_urls << CGI.unescapeHTML(base_url) if base_url + context = { + namespaces: parent[:namespaces].merge(namespace_attributes(attributes)), + base_urls: base_urls + } + scopes << context unless attributes.rstrip.end_with?("/") end contexts @@ -596,13 +604,7 @@ def element_namespace(tag, namespaces) # @rbs (String) -> Hash[String, String] def xml_attributes(attributes) - values = {} #: Hash[String, String] - attributes.scan(/([\w:.-]+)\s*=\s*(?:"([^"]*)"|'([^']*)')/m) do - name = Regexp.last_match(1) - value = Regexp.last_match(2) || Regexp.last_match(3) - values[name] = value if name && value - end - values + XmlElement.attributes(attributes) end # @rbs (String, String?) -> void @@ -912,5 +914,9 @@ def unescape(content) end end +require_relative "simple-rss/xml_element" +require_relative "simple-rss/normalized_entry" +require_relative "simple-rss/entry_normalizer" + class SimpleRSSError < StandardError # rubocop:disable Style/OneClassPerFile end diff --git a/lib/simple-rss/entry_normalizer.rb b/lib/simple-rss/entry_normalizer.rb new file mode 100644 index 0000000..f67641e --- /dev/null +++ b/lib/simple-rss/entry_normalizer.rb @@ -0,0 +1,356 @@ +# rbs_inline: enabled + +require "uri" + +class SimpleRSS::EntryNormalizer + ATOM = SimpleRSS::ATOM_NAMESPACE + CONTENT = "http://purl.org/rss/1.0/modules/content/".freeze + DUBLIN_CORE = "http://purl.org/dc/elements/1.1/".freeze + MEDIA = "http://search.yahoo.com/mrss/".freeze + ITUNES = "http://www.itunes.com/dtds/podcast-1.0.dtd".freeze + XHTML = "http://www.w3.org/1999/xhtml".freeze + + # @rbs (Hash[Symbol, untyped]) -> void + def self.validate_mappings(mappings) + allowed = %i[content_html content_text categories] + raise ArgumentError, "mappings must be a Hash with keys #{allowed.join(", ")}" unless mappings.is_a?(Hash) && (mappings.keys - allowed).empty? + + %i[content_html content_text].each do |field| + next unless mappings.key?(field) + raise ArgumentError, "#{field} mapping must be a nonempty XML tag name" unless mappings[field].is_a?(String) && !mappings[field].strip.empty? + end + return unless mappings.key?(:categories) + + categories = mappings[:categories] + raise ArgumentError, "categories mapping must be an array of tag/separator records" unless categories.is_a?(Array) + + categories.each do |mapping| + valid = mapping.is_a?(Hash) && (mapping.keys - %i[tag separator]).empty? && mapping[:tag].is_a?(String) && !mapping[:tag].strip.empty? + valid &&= !mapping.key?(:separator) || (mapping[:separator].is_a?(String) && !mapping[:separator].empty?) + raise ArgumentError, "category mappings require a tag and an optional nonempty separator" unless valid + end + end + + # @rbs (SimpleRSS::XmlElement, Hash[Symbol, untyped], Hash[Symbol, untyped]) -> void + def initialize(element, raw, options) + @element = element + @children = element.children + @source_url = options[:source_url] #: String? + @mappings = options.fetch(:mappings) #: Hash[Symbol, untyped] + @feed_authors = options.fetch(:feed_authors) #: Array[SimpleRSS::XmlElement] + @values = { raw: raw, raw_xml: options[:raw_xml], field_sources: {}, issues: [] } #: Hash[Symbol, untyped] + @field_elements = {} #: Hash[Symbol, SimpleRSS::XmlElement] + end + + # @rbs () -> SimpleRSS::NormalizedEntry + def entry + read_identity + read_links + read_dates + read_content + read_categories + read_attachments + read_authors + SimpleRSS::NormalizedEntry.new(@values) + end + + private + + # @rbs () -> bool + def atom? + @element.namespace == ATOM + end + + # @rbs (String) -> Array[SimpleRSS::XmlElement] + def core_elements(name) + @children.select { |element| atom? ? element.matches?(name, ATOM) : element.rss?(name) } + end + + # @rbs (String, String) -> Array[SimpleRSS::XmlElement] + def extension_elements(name, namespace) + @children.select { |element| element.matches?(name, namespace) } + end + + # @rbs (Symbol, untyped, SimpleRSS::XmlElement) -> void + def assign(field, value, element) + return if value.nil? || value == "" + + @values[field] = value + @values[:field_sources][field] = element.name + @field_elements[field] = element + end + + # @rbs (Symbol, Symbol, untyped, SimpleRSS::XmlElement) -> nil + def issue(field, code, value, element) + @values[:issues] << { field: field, code: code, value: value, source: element.name } + nil + end + + # @rbs () -> void + def read_identity + identifier = core_elements(atom? ? "id" : "guid").find { |element| !element.text.empty? } + assign(:identifier, identifier.text, identifier) if identifier + title = core_elements("title").first + assign(:title, title.text, title) if title + end + + # @rbs () -> void + def read_links + elements = @children.select { |element| element.matches?("link", ATOM) || (!atom? && element.rss?("link")) } + links = elements.map do |element| + href = element.namespace == ATOM ? element.attributes["href"] : element.text + relation = element.namespace == ATOM ? element.attributes.fetch("rel", "alternate") : "alternate" + relation = relation.delete_prefix("http://www.iana.org/assignments/relation/") + { url: resolve_url(href, element, :url), rel: relation, media_type: element.attributes["type"], raw: element.raw } + end + @values[:links] = links + candidates = links.each_index.select { |index| links[index][:rel] == "alternate" && links[index][:url] } + selected = candidates.min_by do |index| + link = links[index] + media_type = link[:media_type].to_s.split(";").first.to_s.downcase.strip + [!atom? && elements.fetch(index).rss?("link") ? 0 : 1, media_type_priority(media_type), index] + end + assign(:url, links[selected][:url], elements.fetch(selected)) if selected + end + + # @rbs (String) -> Integer + def media_type_priority(media_type) + return 0 if %w[text/html application/xhtml+xml].include?(media_type) + return 1 if media_type.empty? + + 2 + end + + # @rbs (String?, SimpleRSS::XmlElement, Symbol) -> String? + def resolve_url(value, element, field) + return if value.nil? || value.strip.empty? + + reference = value.strip + return reference if URI.parse(reference).absolute? + + base = base_url(element) + unless base + issue(field, :relative_url_without_base, value, element) + return reference + end + URI.join(base, reference).to_s + rescue URI::Error + issue(field, :invalid_url, value, element) + value + end + + # @rbs (SimpleRSS::XmlElement) -> String? + def base_url(element) + candidates = [@source_url, *element.base_urls].compact + start = candidates.rindex { |value| URI.parse(value).absolute? } + return unless start + + base = candidates.fetch(start) + candidates.drop(start + 1).each { |value| base = URI.join(base, value).to_s } + base + rescue URI::Error + issue(:base_url, :invalid_url, candidates, element) + end + + # @rbs () -> void + def read_dates + published = atom? ? core_elements("published") : core_elements("pubDate") + extension_elements("date", DUBLIN_CORE) + updated = extension_elements("updated", ATOM) + updated += core_elements("modified") unless atom? + read_date(:published_at, published) + read_date(:updated_at, updated) + end + + # @rbs (Symbol, Array[SimpleRSS::XmlElement]) -> void + def read_date(field, elements) + elements.each do |element| + next if element.text.empty? + + begin + assign(field, Time.parse(element.text), element) + break + rescue ArgumentError, RangeError + issue(field, :invalid_date, element.text, element) + end + end + end + + # @rbs () -> void + def read_content + %i[content_html content_text].each do |field| + selector = @mappings[field] + element = selector && @children.find { |child| child.selected?(selector) && !child.text.empty? } + assign(field, element.text, element) if element + end + encoded = extension_elements("encoded", CONTENT).find { |element| !element.text.empty? } + assign(:content_html, encoded.text, encoded) if encoded && !@values[:content_html] + read_atom_content + full_content = @field_elements[:content_html] || @field_elements[:content_text] + @values[:content_base_url] = base_url(content_container(full_content)) if full_content + summary = core_elements(atom? ? "summary" : "description").first + return unless summary + + value, type = atom? ? text_construct(summary, :summary) : [summary.text, :html] + assign(:summary, value, summary) + @values[:summary_type] = type if @values[:summary] + end + + # @rbs () -> void + def read_atom_content + content = extension_elements("content", ATOM).first + return unless content + return assign(:content_url, resolve_url(content.attributes["src"], content, :content_url), content) if content.attributes["src"] + + value, type = text_construct(content, :content) + field = type == :html ? :content_html : :content_text + assign(field, value, content) unless @values[field] + end + + # @rbs (SimpleRSS::XmlElement) -> SimpleRSS::XmlElement + def content_container(element) + type = element.attributes["type"].to_s.split(";", 2).first.to_s.strip.downcase + return element unless %w[xhtml application/xhtml+xml].include?(type) + + element.children.find { |child| child.matches?("div", XHTML) } || element + end + + # @rbs (SimpleRSS::XmlElement, Symbol) -> [String?, Symbol?] + def text_construct(element, field) + type = element.attributes.fetch("type", "text").split(";", 2).first.to_s.strip.downcase + case type + when "text", "text/plain" + return [element.text, :text] if element.children.empty? + + issue(field, :unexpected_markup, element.content, element) + when "html", "text/html" + return [element.text, :html] + when "xhtml", "application/xhtml+xml" + children = element.children + return [children.first.xhtml_content, :html] if children.size == 1 && children.first.matches?("div", XHTML) + + issue(field, :invalid_xhtml, element.content, element) + else + issue(field, :unsupported_content_type, type, element) + end + [nil, nil] + end + + # @rbs () -> void + def read_categories + details = @children.flat_map do |element| + mappings = @mappings[:categories] || [] #: Array[Hash[Symbol, untyped]] + mapping = mappings.find { |candidate| element.selected?(candidate[:tag]) } + term = category_term(element) + next [] unless mapping || term + + terms = [term] + terms = mapping[:separator] ? element.text.split(mapping[:separator]) : [element.text] if mapping + terms.filter_map do |value| + next if value.nil? || value.strip.empty? + + { + term: value.strip, label: element.attributes["label"], scheme: element.attributes["scheme"] || element.attributes["domain"], + source: element.name, raw: element.raw + } + end + end + @values[:category_details] = details + @values[:categories] = details.map { |category| category[:term] }.uniq + end + + # @rbs (SimpleRSS::XmlElement) -> String? + def category_term(element) + return element.attributes["term"] if element.matches?("category", ATOM) + return element.text if !atom? && element.rss?("category") + return element.text if element.matches?("subject", DUBLIN_CORE) + return element.text if element.matches?("keywords", MEDIA) || element.matches?("keywords", ITUNES) + + nil + end + + # @rbs () -> void + def read_attachments + elements = @children.flat_map do |element| + next [element] unless element.matches?("group", MEDIA) + + element.children.select { |child| child.matches?("content", MEDIA) } + end + attachments = elements.filter_map do |element| + attributes = element.attributes + if element.matches?("content", MEDIA) + attachment(element, attributes["url"], attributes["fileSize"], attributes["duration"]) + elsif !atom? && element.rss?("enclosure") + attachment(element, attributes["url"], attributes["length"], nil) + elsif element.matches?("link", ATOM) && attributes["rel"].to_s.delete_prefix("http://www.iana.org/assignments/relation/") == "enclosure" + attachment(element, attributes["href"], attributes["length"], nil) + end + end + duration = extension_elements("duration", ITUNES).first + if attachments.size == 1 && duration && !attachments.first[:raw][:attributes].key?("duration") + attachments.first[:duration_in_seconds] = duration_seconds(duration.text, duration) + attachments.first[:raw_duration] = duration.raw + end + @values[:attachments] = attachments + end + + # @rbs (SimpleRSS::XmlElement, String?, String?, String?) -> Hash[Symbol, untyped]? + def attachment(element, url, size, duration) + resolved_url = resolve_url(url, element, :attachment_url) + return unless resolved_url + + { + url: resolved_url, media_type: element.attributes["type"], size_in_bytes: size_in_bytes(size, element), + duration_in_seconds: duration_seconds(duration, element), source: element.name, raw: element.raw + } + end + + # @rbs (String?, SimpleRSS::XmlElement) -> Integer? + def size_in_bytes(value, element) + return if value.nil? + return value.to_i if value.match?(/\A\d+\z/) + + issue(:size_in_bytes, :invalid_number, value, element) + end + + # @rbs (String?, SimpleRSS::XmlElement) -> (Integer | Float)? + def duration_seconds(value, element) + return if value.nil? + return value.to_f if value.match?(/\A\d+(?:\.\d+)?\z/) && value.to_f.finite? + + unless element.matches?("duration", ITUNES) && value.match?(/\A\d+:\d{2}(?::\d{2})?\z/) + return issue(:duration_in_seconds, :invalid_number, value, element) + end + + parts = value.split(":").map(&:to_i) + return issue(:duration_in_seconds, :invalid_number, value, element) unless parts.drop(1).all? { |part| part < 60 } + + parts.reduce(0) { |total, part| (total * 60) + part } + end + + # @rbs () -> void + def read_authors + @values[:authors] = author_elements.map do |element| + next { name: element.text, email: nil, url: nil, raw: element.raw } if element.matches?("creator", DUBLIN_CORE) + next { name: nil, email: element.text, url: nil, raw: element.raw } unless element.namespace == ATOM + + children = element.children + name = children.find { |child| child.matches?("name", ATOM) } + email = children.find { |child| child.matches?("email", ATOM) } + uri = children.find { |child| child.matches?("uri", ATOM) } + { name: name&.text, email: email&.text, url: uri && resolve_url(uri.text, uri, :author_url), raw: element.raw } + end + end + + # @rbs () -> Array[SimpleRSS::XmlElement] + def author_elements + elements = core_elements("author") + return elements + extension_elements("creator", DUBLIN_CORE) unless atom? + return elements unless elements.empty? + + source = core_elements("source").first + return @feed_authors unless source + + elements = source.children.select { |element| element.matches?("author", ATOM) } + elements.empty? ? @feed_authors : elements + end +end diff --git a/lib/simple-rss/normalized_entry.rb b/lib/simple-rss/normalized_entry.rb new file mode 100644 index 0000000..8717332 --- /dev/null +++ b/lib/simple-rss/normalized_entry.rb @@ -0,0 +1,72 @@ +# rbs_inline: enabled + +class SimpleRSS::NormalizedEntry + attr_reader :identifier, :url, :title #: String? + attr_reader :published_at, :updated_at #: Time? + attr_reader :content_html, :content_text, :content_url, :content_base_url, :summary #: String? + attr_reader :summary_type #: Symbol? + attr_reader :categories #: Array[String] + attr_reader :category_details, :attachments, :links, :authors, :issues #: Array[Hash[Symbol, untyped]] + attr_reader :raw, :field_sources #: Hash[Symbol, untyped] + attr_reader :raw_xml #: String? + + # @rbs (Hash[Symbol, untyped]) -> void + def initialize(attributes) + values = immutable_copy(attributes) + empty_categories = [] #: Array[String] + empty_records = [] #: Array[Hash[Symbol, untyped]] + empty_data = {} #: Hash[Symbol, untyped] + empty_categories.freeze + empty_records.freeze + empty_data.freeze + @identifier = values[:identifier] + @url = values[:url] + @title = values[:title] + @published_at = values[:published_at] + @updated_at = values[:updated_at] + @content_html = values[:content_html] + @content_text = values[:content_text] + @content_url = values[:content_url] + @content_base_url = values[:content_base_url] + @summary = values[:summary] + @summary_type = values[:summary_type] + @categories = values.fetch(:categories, empty_categories) + @category_details = values.fetch(:category_details, empty_records) + @attachments = values.fetch(:attachments, empty_records) + @links = values.fetch(:links, empty_records) + @authors = values.fetch(:authors, empty_records) + @issues = values.fetch(:issues, empty_records) + @raw = values.fetch(:raw, empty_data) + @raw_xml = values[:raw_xml] + @field_sources = values.fetch(:field_sources, empty_data) + freeze + end + + # @rbs () -> Time? + def effective_at + published_at || updated_at + end + + # @rbs () -> Hash[Symbol, untyped] + def to_h + { + identifier: identifier, url: url, title: title, published_at: published_at, updated_at: updated_at, + content_html: content_html, content_text: content_text, content_url: content_url, content_base_url: content_base_url, + summary: summary, summary_type: summary_type, categories: categories, category_details: category_details, + attachments: attachments, links: links, authors: authors, issues: issues, field_sources: field_sources, + raw: raw, raw_xml: raw_xml + } + end + + private + + # @rbs (untyped) -> untyped + def immutable_copy(value) + case value + when Hash then value.to_h { |key, child| [immutable_copy(key), immutable_copy(child)] }.freeze + when Array then value.map { |child| immutable_copy(child) }.freeze + when String, Time then value.dup.freeze + else value + end + end +end diff --git a/lib/simple-rss/xml_element.rb b/lib/simple-rss/xml_element.rb new file mode 100644 index 0000000..1244f4c --- /dev/null +++ b/lib/simple-rss/xml_element.rb @@ -0,0 +1,136 @@ +# rbs_inline: enabled + +class SimpleRSS::XmlElement + attr_reader :name #: String + attr_reader :attributes #: Hash[String, String] + attr_reader :content #: String + attr_reader :namespaces #: Hash[String, String] + attr_reader :base_urls #: Array[String] + + # @rbs (String, String, String?, Hash[Symbol, untyped]) -> void + def initialize(name, attributes, content, parent) + @name = name + @attributes = self.class.attributes(attributes).transform_values { |value| CGI.unescapeHTML(value) } + @content = content.to_s + @namespaces = parent.fetch(:namespaces).merge(@attributes.select { |key, _value| key == "xmlns" || key.start_with?("xmlns:") }) + base_url = @attributes["xml:base"] + @base_urls = parent.fetch(:base_urls).dup + @base_urls << base_url if base_url + end + + # @rbs () -> Array[SimpleRSS::XmlElement] + def children + self.class.child_elements(content).map do |name, attributes, body| + self.class.new(name, attributes, body, { namespaces: namespaces, base_urls: base_urls }) + end + end + + # @rbs () -> String? + def namespace + namespaces[name.include?(":") ? "xmlns:#{name.split(":").first}" : "xmlns"] + end + + # @rbs (String, String?) -> bool + def matches?(local_name, namespace) + name.split(":").last == local_name && self.namespace == namespace + end + + # @rbs (String) -> bool + def rss?(local_name) + return false if name.include?(":") && namespace.nil? + + name.split(":").last == local_name && SimpleRSS::RSS_NAMESPACES.include?(namespace) + end + + # @rbs (String) -> bool + def selected?(selector) + if selector.start_with?("{") + namespace, local_name = selector.delete_prefix("{").split("}", 2) + return local_name ? matches?(local_name, namespace) : false + end + + name == selector + end + + # @rbs () -> String + def text + content.split(/()/m).map do |part| + part.start_with?("|<\?.*?\?>/m, "")) + end.join.strip + end + + # @rbs () -> Hash[Symbol, untyped] + def raw + { name: name, namespace: namespace, attributes: attributes, content: content } + end + + # @rbs () -> String + def xhtml_content + scopes = [namespaces] + content.gsub(SimpleRSS::XML_TAG_PATTERN) do |token| + name = Regexp.last_match(2).to_s + attributes = Regexp.last_match(3) + closing = Regexp.last_match(1) == "/" + next token unless attributes + + if closing + context = scopes.last || namespaces + scopes.pop if scopes.size > 1 + else + declarations = self.class.attributes(attributes).select { |key, _value| key == "xmlns" || key.start_with?("xmlns:") } + context = (scopes.last || namespaces).merge(declarations.transform_values { |value| CGI.unescapeHTML(value) }) + scopes << context unless attributes.rstrip.end_with?("/") + end + namespace = context[name.include?(":") ? "xmlns:#{name.split(":").first}" : "xmlns"] + next token unless namespace == "http://www.w3.org/1999/xhtml" + + token.sub(name, name.split(":").last.to_s) + end.strip + end + + # @rbs (String) -> Hash[String, String] + def self.attributes(attributes) + values = {} #: Hash[String, String] + attributes.scan(/([\w:.-]+)\s*=\s*(?:"([^"]*)"|'([^']*)')/m) do + name = Regexp.last_match(1) + value = Regexp.last_match(2) || Regexp.last_match(3) + values[name] = value if name && value + end + values + end + + # @rbs (String) -> Array[[String, String, String?]] + def self.child_elements(content) + elements = [] #: Array[[String, String, String?]] + current_element = nil #: [String, String, String?]? + depth = 0 + body_start = 0 + position = 0 + + while (token = SimpleRSS::XML_TAG_PATTERN.match(content, position)) + position = token.end(0) + attributes = token[3] + next unless attributes + + if token[1] == "/" + depth = [depth - 1, 0].max + if depth.zero? && current_element + current_element[2] = content[body_start...token.begin(0)] + current_element = nil + end + next + end + + self_closing = attributes.rstrip.end_with?("/") + if depth.zero? + element = [token[2].to_s, attributes, nil] #: [String, String, String?] + elements << element + current_element = element unless self_closing + body_start = token.end(0) + end + depth += 1 unless self_closing + end + + elements + end +end diff --git a/simple-rss.gemspec b/simple-rss.gemspec index 5a97deb..e47f6db 100644 --- a/simple-rss.gemspec +++ b/simple-rss.gemspec @@ -8,7 +8,7 @@ Gem::Specification.new do |s| s.metadata["changelog_uri"] = "https://github.com/cardmagic/simple-rss/blob/master/CHANGELOG.md" s.description = "A simple, flexible, extensible, and liberal RSS and Atom reader for Ruby. It is designed to be backwards compatible with the standard RSS parser, but will never do RSS generation." s.authors = ["Lucas Carlson"] - s.files = Dir["lib/**/*", "test/**/*", "LICENSE", "README.md", "CHANGELOG.md", "Rakefile", "simple-rss.gemspec"] + s.files = Dir["lib/**/*", "examples/**/*", "test/**/*", "LICENSE", "README.md", "CHANGELOG.md", "Rakefile", "simple-rss.gemspec"] s.required_ruby_version = ">= 3.1" s.add_development_dependency "rake" s.add_development_dependency "rdoc" diff --git a/test/base/normalized_entries_test.rb b/test/base/normalized_entries_test.rb new file mode 100644 index 0000000..e0c5919 --- /dev/null +++ b/test/base/normalized_entries_test.rb @@ -0,0 +1,410 @@ +require "test_helper" + +class NormalizedEntriesTest < Test::Unit::TestCase + def test_equivalent_rss_and_atom_entries_share_one_interface + entries = %w[rss atom].map { |format| fixture(format).normalized_entries.first } + + entries.each do |entry| + assert_equal "opaque:%2F:001", entry.identifier + assert_equal "Ruby & feeds", entry.title + assert_equal "https://example.com/blog/article?one=1&two=2", entry.url + assert_equal Time.utc(2026, 9, 12, 10), entry.published_at + assert_equal Time.utc(2026, 9, 13, 11), entry.updated_at + assert_equal "

Full & complete.

", entry.content_html + assert_nil entry.content_text + assert_equal "

A summary.

", entry.summary + assert_equal :html, entry.summary_type + assert_equal %w[ruby feeds], entry.categories + assert_equal(["https://example.com/audio/one.mp3", "https://example.com/images/two.png"], entry.attachments.map { |attachment| attachment[:url] }) + assert_equal(%w[audio/mpeg image/png], entry.attachments.map { |attachment| attachment[:media_type] }) + assert_equal([1234, 4321], entry.attachments.map { |attachment| attachment[:size_in_bytes] }) + assert_empty entry.issues + end + end + + def test_normalization_preserves_raw_access_serialization_and_configuration + feed = fixture("atom") + before = [feed.source.dup, feed.to_hash, feed.to_json, feed.to_xml, SimpleRSS.item_tags.dup, SimpleRSS.feed_tags.dup] + + first = feed.normalized_entries.first + second = feed.normalized_entries.first + + assert_equal first.to_h, second.to_h + assert_equal feed.first, first.raw + assert_not_same feed.first, first.raw + assert_equal before, [feed.source, feed.to_hash, feed.to_json, feed.to_xml, SimpleRSS.item_tags, SimpleRSS.feed_tags] + assert_raise(FrozenError) { first.categories << "changed" } + assert_raise(FrozenError) { first.raw[:title].replace("changed") } + assert_equal "Ruby & feeds", feed.first.title + end + + def test_link_selection_honors_default_and_iana_relations_and_preserves_metadata + entry = atom_entry(<<~XML) + + + + + + XML + + assert_equal "https://example.com/article", entry.url + assert_equal 3, entry.links.size + assert_equal "Web & mobile", entry.links.last[:raw][:attributes]["title"] + assert_equal "en", entry.links.last[:raw][:attributes]["hreflang"] + assert_include entry.links.last[:raw][:content], "wrong.example.com" + assert_nil atom_entry('').url + assert_equal "https://example.com/default", atom_entry('').url + end + + def test_relative_urls_use_source_url_and_each_xml_base_without_changing_identifiers + source = <<~XML + + Base URLs + + ../opaque%2Fid + + + + + XML + feed = SimpleRSS.parse(source, source_url: "https://example.com/feeds/current.xml") + entry = feed.normalized_entries.first + + assert_equal "https://example.com/articles/one?tag=ruby&page=2#section", entry.url + assert_equal "https://cdn.example.com/one.mp3", entry.attachments.first[:url] + assert_equal "../opaque%2Fid", entry.identifier + assert_equal "https://override.example.com/articles/one?tag=ruby&page=2#section", + feed.normalized_entries(source_url: "https://override.example.com/feeds/current.xml").first.url + assert_equal "https://example.com/feeds/current.xml", feed.source_url + end + + def test_relative_and_invalid_urls_remain_inspectable_without_a_usable_base + entry = atom_entry('') + assert_equal "../article", entry.url + assert_include entry.issues, { field: :url, code: :relative_url_without_base, value: "../article", source: "link" } + + entry = atom_entry('') + assert_equal "http://[broken", entry.url + assert_equal :invalid_url, entry.issues.first[:code] + + entry = atom_entry('', source_url: "https://example.com/") + assert_equal "article", entry.url + assert_include entry.issues.map { |issue| issue[:field] }, :base_url + end + + def test_publication_and_update_dates_remain_distinct_and_invalid_dates_are_preserved + entry = atom_entry("not-a-date2026-09-13T11:00:00Z") + + assert_nil entry.published_at + assert_equal Time.utc(2026, 9, 13, 11), entry.updated_at + assert_equal entry.updated_at, entry.effective_at + assert_equal "not-a-date", entry.raw[:published] + assert_include entry.issues, { field: :published_at, code: :invalid_date, value: "not-a-date", source: "published" } + assert_nil atom_entry("broken").effective_at + + entry = rss_entry("broken2026-09-12T10:00:00Zbroken too") + assert_equal Time.utc(2026, 9, 12, 10), entry.published_at + assert_nil entry.updated_at + assert_equal "dc:date", entry.field_sources[:published_at] + assert_equal 2, entry.issues.size + end + + def test_atom_content_types_preserve_text_html_cdata_and_summary_boundaries + entry = atom_entry("<b>literal & text</b>Short version") + assert_equal "literal & text", entry.content_text + assert_nil entry.content_html + assert_equal "Short version", entry.summary + assert_equal :text, entry.summary_type + + entry = atom_entry('A & B

]]>') + assert_equal "

A & B

", entry.content_html + assert_nil entry.content_text + assert_equal "content", entry.field_sources[:content_html] + + entry = atom_entry('<p>Summary only</p>') + assert_nil entry.content_html + assert_nil entry.content_text + assert_equal "

Summary only

", entry.summary + assert_nil rss_entry("Summary only").content_text + end + + def test_external_unsupported_and_invalid_text_content_do_not_become_plain_text + entry = atom_entry('Ignored inline body') + assert_equal "https://example.com/full.txt", entry.content_url + assert_nil entry.content_text + assert_nil entry.content_html + + entry = atom_entry('YWJj') + assert_nil entry.content_text + assert_nil entry.content_html + assert_equal "YWJj", entry.raw[:content] + assert_equal :unsupported_content_type, entry.issues.first[:code] + + entry = atom_entry('Actual markup') + assert_nil entry.content_text + assert_equal :unexpected_markup, entry.issues.first[:code] + end + + def test_xhtml_content_excludes_its_container_and_retains_its_base + entry = atom_entry(<<~XML) + +

A & B link

+ + XML + assert_equal '

A & B link

', entry.content_html + assert_equal "https://example.com/articles/", entry.content_base_url + assert_nil entry.content_text + assert_empty entry.issues + + entry = atom_entry('
Wrong namespace
') + assert_nil entry.content_html + assert_equal :invalid_xhtml, entry.issues.first[:code] + end + + def test_custom_full_text_mappings_are_local_and_take_precedence_without_global_tags + source = <<~XML + + Custom content + Summary + Default body

]]> + Complete body

]]> + Complete text + Nested body + + + XML + tags = [SimpleRSS.feed_tags.dup, SimpleRSS.item_tags.dup] + feed = SimpleRSS.parse(source) + entry = feed.normalized_entries(mappings: { content_html: "full-text", content_text: "{urn:example:body}plain" }).first + + assert_equal "

Complete body

", entry.content_html + assert_equal "Complete text", entry.content_text + assert_equal "Summary", entry.summary + assert_equal "full-text", entry.field_sources[:content_html] + assert_equal "body:plain", entry.field_sources[:content_text] + assert_include entry.raw_xml, "" + assert_equal "

Default body

", feed.normalized_entries.first.content_html + assert_equal "

Default body

", SimpleRSS.parse(source).normalized_entries.first.content_html + assert_equal tags, [SimpleRSS.feed_tags, SimpleRSS.item_tags] + end + + def test_categories_preserve_labels_schemes_duplicates_and_source_aware_keywords + entry = fixture("atom").normalized_entries.first + assert_equal %w[ruby feeds], entry.categories + assert_equal(%w[ruby feeds ruby], entry.category_details.map { |category| category[:term] }) + assert_equal "Ruby language", entry.category_details.first[:label] + assert_equal "urn:topics", entry.category_details.first[:scheme] + assert_equal "ruby", entry.category_details.first[:raw][:attributes]["term"] + + source = <<~XML + + Keywords + ruby, railsruby; feeds + audio, videoone | two + + + XML + feed = SimpleRSS.parse(source) + assert_equal ["ruby, rails", "ruby; feeds", "audio, video"], feed.normalized_entries.first.categories + mappings = { categories: [{ tag: "dc:subject", separator: ";" }, { tag: "keywords", separator: "|" }] } + assert_equal ["ruby, rails", "ruby", "feeds", "audio, video", "one", "two"], feed.normalized_entries(mappings: mappings).first.categories + end + + def test_multiple_attachments_keep_attributes_and_invalid_numbers_with_their_urls + entry = rss_entry(<<~XML) + + + + + + ]]> + XML + first, second, third = entry.attachments + assert_equal 3, entry.attachments.size + assert_equal 0, first[:size_in_bytes] + assert_equal "https://example.com/media/two", second[:url] + assert_equal "video/mp4", second[:media_type] + assert_equal 123, second[:size_in_bytes] + assert_equal 12.5, second[:duration_in_seconds] + assert_equal "https://example.com/media/three", third[:url] + assert_nil third[:size_in_bytes] + assert_nil third[:duration_in_seconds] + assert_equal "no-size", third[:raw][:attributes]["fileSize"] + assert_equal "forever", third[:raw][:attributes]["duration"] + assert_equal(%i[size_in_bytes duration_in_seconds], entry.issues.map { |issue| issue[:field] }) + end + + def test_single_attachment_can_use_itunes_duration_without_applying_it_to_multiple_files + entry = rss_entry('1:02:03') + assert_equal 3723, entry.attachments.first[:duration_in_seconds] + assert_equal "itunes:duration", entry.attachments.first[:raw_duration][:name] + + entry = rss_entry('1:99') + assert_nil entry.attachments.first[:duration_in_seconds] + assert_equal :invalid_number, entry.issues.first[:code] + + entry = rss_entry('42') + assert_equal([nil, nil], entry.attachments.map { |attachment| attachment[:duration_in_seconds] }) + end + + def test_nested_foreign_and_commented_metadata_cannot_supply_normalized_fields + entry = atom_entry(<<~XML) + + Wrong body]]> + Wrong id +
+ Foreign id + + Reset + 2026-01-01T00:00:00Z + Actual id + XML + assert_equal "Actual id", entry.identifier + assert_nil entry.url + assert_nil entry.published_at + assert_equal ["actual"], entry.categories + assert_empty entry.attachments + assert_empty entry.links + end + + def test_entry_order_and_deduplication_keep_original_xml_bound_to_the_correct_hash + source = <<~XML + Identical raw items + sameFirst body + sameSecond body + + XML + feed = SimpleRSS.parse(source) + mappings = { content_text: "full-text" } + assert_equal ["First body", "Second body"], feed.normalized_entries(mappings: mappings).map(&:content_text) + feed.items.reverse! + assert_equal ["Second body", "First body"], feed.normalized_entries(mappings: mappings).map(&:content_text) + feed.dedupe + assert_equal ["Second body"], feed.normalized_entries(mappings: mappings).map(&:content_text) + end + + def test_empty_fields_do_not_invent_identity_content_dates_or_collections + entry = atom_entry('') + assert_nil entry.identifier + assert_nil entry.url + assert_nil entry.title + assert_nil entry.published_at + assert_nil entry.updated_at + assert_nil entry.content_html + assert_nil entry.content_text + assert_nil entry.summary + assert_empty entry.categories + assert_empty entry.attachments + end + + def test_invalid_mapping_configuration_fails_with_clear_errors + feed = fixture("rss") + [nil, { unknown: "tag" }, { content_html: [] }, { content_text: "" }, { categories: "keywords" }, + { categories: [{ tag: "keywords", separator: "" }] }, { categories: [{ tag: nil }] }].each do |mappings| + assert_raise(ArgumentError) { feed.normalized_entries(mappings: mappings) } + end + end + + def test_rdf_item_bases_and_namespaces_do_not_inherit_from_a_sibling_channel + source = <<~XML + + Example + Firstarticle2026-09-12T10:00:00Zruby + + XML + entry = SimpleRSS.parse(source).normalized_entries.first + assert_equal "https://example.com/article", entry.url + assert_equal Time.utc(2026, 9, 12, 10), entry.published_at + assert_equal ["ruby"], entry.categories + end + + def test_plain_xml_text_ignores_comments_and_processing_instructions + entry = atom_entry("opaqueidBeforeafter") + assert_equal "opaqueid", entry.identifier + assert_equal "Beforeafter", entry.content_text + end + + def test_atom_authors_inherit_from_source_then_feed_and_keep_uri_bases + source = <<~XML + + Authors + Editorial teamabout + one + twoSource teamabout + threeEntry teamentryGuest team + + XML + entries = SimpleRSS.parse(source).normalized_entries + assert_equal(["Editorial team"], entries[0].authors.map { |author| author[:name] }) + assert_equal "https://example.com/about", entries[0].authors.first[:url] + assert_equal(["Source team"], entries[1].authors.map { |author| author[:name] }) + assert_equal "https://example.com/source/about", entries[1].authors.first[:url] + assert_equal(["Entry team", "Guest team"], entries[2].authors.map { |author| author[:name] }) + end + + def test_prefixed_xhtml_preserves_content_and_the_div_base_without_prefixing_html_tags + entry = atom_entry(<<~XML) + + + Read this & more. + + + XML + assert_equal '

Read this & more.

', entry.content_html + assert_equal "https://example.com/articles/", entry.content_base_url + assert_equal "content", entry.field_sources[:content_html] + assert_include entry.raw_xml, "<p>Body</p>') + assert_equal "

Body

", entry.content_html + assert_nil entry.content_text + + entry = rss_entry('https://example.com/article') + assert_equal "https://example.com/article", entry.url + end + + def test_media_groups_only_contribute_media_content_attachments + entry = rss_entry(<<~XML) + + + + + + XML + assert_equal(["https://example.com/video"], entry.attachments.map { |attachment| attachment[:url] }) + end + + def test_invalid_attachment_duration_is_not_replaced_by_an_item_level_duration + entry = rss_entry(<<~XML) + + 42 + XML + assert_nil entry.attachments.first[:duration_in_seconds] + assert_equal "unknown", entry.attachments.first[:raw][:attributes]["duration"] + assert_equal :invalid_number, entry.issues.first[:code] + end + + private + + def fixture(format) + SimpleRSS.parse(File.read(File.join(__dir__, "../data/normalized_#{format}.xml"))) + end + + def atom_entry(content, options = {}) + SimpleRSS.parse('Example' + content + "", options).normalized_entries.first + end + + def rss_entry(content) + source = <<~XML + + Example#{content} + + XML + SimpleRSS.parse(source).normalized_entries.first + end +end diff --git a/test/base/normalized_fetch_test.rb b/test/base/normalized_fetch_test.rb new file mode 100644 index 0000000..81389c8 --- /dev/null +++ b/test/base/normalized_fetch_test.rb @@ -0,0 +1,92 @@ +require "test_helper" +require "socket" +require "stringio" + +class NormalizedFetchTest < Test::Unit::TestCase + def test_fetch_uses_the_final_response_url_after_relative_redirects + body = 'Example' \ + '' + responses = [[302, { "Location" => "../feeds/final.xml" }, ""], [200, { "Content-Type" => "application/atom+xml" }, body]] + with_server(responses) do |base_url, requests| + feed = SimpleRSS.fetch("#{base_url}/start/feed.xml", timeout: 1) + entry = feed.normalized_entries.first + + assert_equal "#{base_url}/feeds/final.xml", feed.source_url + assert_equal "#{base_url}/feeds/article", entry.url + assert_equal "#{base_url}/feeds/full.txt", entry.content_url + assert_nil entry.content_text + assert_equal ["GET /start/feed.xml HTTP/1.1", "GET /feeds/final.xml HTTP/1.1"], requests.map(&:first) + end + end + + def test_fetch_retains_conditional_get_and_parse_options + body = 'Examplearticleruby' \ + "feeds" + headers = { "ETag" => '"version-1"', "Last-Modified" => "Sat, 12 Sep 2026 10:00:00 GMT" } + with_server([[200, headers, body], [304, headers, ""]]) do |base_url, requests| + feed = SimpleRSS.fetch("#{base_url}/feed.xml", timeout: 1, array_tags: [:category]) + + assert_equal '"version-1"', feed.etag + assert_equal headers["Last-Modified"], feed.last_modified + assert_equal %w[ruby feeds], feed.first.category + assert_equal "#{base_url}/article", feed.normalized_entries.first.url + assert_nil SimpleRSS.fetch("#{base_url}/feed.xml", timeout: 1, etag: feed.etag, last_modified: feed.last_modified) + assert_include requests.last, 'If-None-Match: "version-1"' + assert_include requests.last, "If-Modified-Since: #{feed.last_modified}" + end + end + + def test_readable_io_and_explicit_source_url_work_without_fetching + source = StringIO.new('Examplearticle') + feed = SimpleRSS.parse(source, source_url: "https://example.com/feed.xml") + + assert_equal "https://example.com/article", feed.normalized_entries.first.url + assert_nil feed.etag + end + + def test_fetched_source_url_wins_over_shared_parse_options_without_mutating_them + body = 'Examplearticle' + responses = [[302, { "Location" => "/feeds/final.xml" }, ""], [200, {}, body]] * 2 + with_server(responses) do |base_url, requests| + [nil, "https://wrong.example.com/old.xml"].each do |source_url| + options = { source_url: source_url, timeout: 1 } + feed = SimpleRSS.fetch("#{base_url}/initial.xml", options) + + assert_equal "#{base_url}/feeds/final.xml", feed.source_url + assert_equal "#{base_url}/feeds/article", feed.normalized_entries.first.url + assert_equal source_url, options[:source_url] + assert_equal "https://override.example.com/article", feed.normalized_entries(source_url: "https://override.example.com/feed.xml").first.url + end + assert_equal 4, requests.size + end + end + + private + + def with_server(responses) + server = TCPServer.new("127.0.0.1", 0) + base_url = "http://127.0.0.1:#{server.addr[1]}" + requests = [] + worker = Thread.new do + responses.each do |status, headers, body| + client = server.accept + request = [] + while (line = client.gets) + break if line == "\r\n" + + request << line.strip + end + requests << request + response_headers = headers.merge("Content-Length" => body.bytesize.to_s, "Connection" => "close") + client.write("HTTP/1.1 #{status} Test\r\n" + response_headers.map { |name, value| "#{name}: #{value}\r\n" }.join + "\r\n" + body) + client.close + end + end + yield base_url, requests + worker.value + ensure + worker&.kill + worker&.join + server&.close + end +end diff --git a/test/data/normalized_atom.xml b/test/data/normalized_atom.xml new file mode 100644 index 0000000..2707208 --- /dev/null +++ b/test/data/normalized_atom.xml @@ -0,0 +1,22 @@ + + Example feed + + opaque:%2F:001 + Ruby & feeds + + + + + + 2026-09-12T10:00:00Z + 2026-09-13T11:00:00Z + <p>A summary.</p> + <p>Full &amp; complete.</p> + + + + + + + + diff --git a/test/data/normalized_rss.xml b/test/data/normalized_rss.xml new file mode 100644 index 0000000..f877975 --- /dev/null +++ b/test/data/normalized_rss.xml @@ -0,0 +1,21 @@ + + + Example feed + + opaque:%2F:001 + Ruby & feeds + article?one=1&two=2 + Sat, 12 Sep 2026 10:00:00 GMT + 2026-09-13T11:00:00Z + A summary.

]]> + Full & complete.

]]> + ruby + feeds + ruby + + + + + +