Skip to content

creating-custom-feeds.mdx

Latest commit

 

History

History
230 lines (156 loc) · 7.78 KB

creating-custom-feeds.mdx

File metadata and controls

230 lines (156 loc) · 7.78 KB
title Creating Custom Feeds
description Learn to write custom YAML configurations for RSS feeds when auto-sourcing isn't enough.
sidebar
order
2

import { Aside, Code } from "@astrojs/starlight/components";

When auto-sourcing isn't enough, you can write your own configuration files to create custom RSS feeds for any website. This guide shows you how to take full control with YAML configs.

Prerequisites: You should be familiar with the Getting Started guide before diving into custom configurations.

Start with included feeds first. If your site is not covered, try [automatic feed generation](/web-application/how-to/use-automatic-feed-generation/) next. Reach for a custom config when you need a stable, reviewable setup or the generated feed misses important content.

When to Use Custom Configs

Use custom configs when:

  • Auto-sourcing doesn't work for the website you want to follow
  • Existing feeds are incomplete or missing important content
  • You need specific formatting or data extraction
  • The website has complex structure that requires custom selectors
  • You want to combine data from multiple sources

Don't need custom configs? Check the Feed Directory first - there might already be a working feed for your website.

Recommended Workflow

  1. Inspect the live page in your browser developer tools
  2. Write the smallest useful config that extracts items, titles, and links
  3. Validate the config with html2rss validate your-config.yml
  4. Render the feed with html2rss feed your-config.yml
  5. Add it to html2rss-web so you can use it through your normal instance
  6. Escalate request strategy when needed: use a browser-based rendering strategy only when troubleshooting requires it

This order keeps iteration fast and makes it easier to see whether the problem is the page structure, your selectors, or the fetch strategy.

How It Works

A config file is a simple "recipe" that tells html2rss:

  1. Which website to look at
  2. What content to find
  3. How to organize it into an RSS feed

The channel Block

This tells html2rss basic information about your feed - like giving it a name and telling it which website to look at.

Example:

<Code code={channel: url: https://example.com/blog title: My Awesome Blog} lang="yaml" />

This says: "Look at this website and call the feed 'My Awesome Blog'"

The selectors Block

This is where you tell the html2rss engine exactly what to find on the page. You use CSS selectors (like you might use in web design) to point to specific parts of the webpage.

Example:

<Code code={selectors: items: selector: "article.post" title: selector: "h2 a" link: selector: "h2 a" attribute: href} lang="yaml" />

This says: "Find each article, get the title from the h2 anchor, and get the link from the same h2 anchor's href attribute"

Need more details? Check our complete guide to selectors for all the options.

Your First Config

Step 1: Inspect the website you want to create a feed for. Start with your browser's developer tools to inspect the live DOM. "View Page Source" can still help, but it may miss JavaScript-rendered content.

Step 2: Create a file called example.com.yml with this basic structure:

<Code code={ "channel:\n" + " url: https://example.com/blog\n" + " title: My Blog\n" + "\n" + "selectors:\n" + " items:\n" + ' selector: "article.post"\n' + " title:\n" + ' selector: "h2 a"\n' + " link:\n" + ' selector: "h2 a"\n' + " attribute: href" } lang="yaml" />

Step 3: Test it with your html2rss-web instance or the Ruby gem.

Need help? See our troubleshooting guide for common issues.

Configuration Options

html2rss supports many configuration options:

  • Basic selectors for title, description, and links
  • Advanced features like custom headers and dynamic parameters
  • Multiple strategies for different types of websites
  • Post-processing to clean up extracted content

See our Ruby Gem Reference for complete documentation.

Testing Your Config

Before sharing your config, test it:

  1. Validate the config first:

<Code code={html2rss validate your-config.yml} lang="bash" />

  1. Then render the feed with the Ruby gem:

<Code code={html2rss feed your-config.yml} lang="bash" />

  1. Test with html2rss-web: Add your config to the feeds.yml file and restart your instance

  2. Check the output: Make sure all items have titles, links, and descriptions

Useful CLI flags when a site is difficult

Some sites need a little more request budget than the defaults.

  • Use --max-redirects when the site bounces through several canonicalization or tracking redirects before the real page loads.
  • Use --max-requests when your config needs more than one request, for example pagination or other follow-up fetches.

<Code code={html2rss feed your-config.yml --max-redirects 10 && \ html2rss feed your-config.yml --max-requests 5 && \ html2rss auto https://example.com/blog --max-redirects 10 --max-requests 5} lang="bash" />

Keep these values tight. Raise them only when the site proves it needs more.

Add It To html2rss-web

Once the config works locally, add it to your feeds.yml or shared config repository and restart your instance. Then open the feed through your normal html2rss-web URL and confirm it behaves the same way there.

Sharing Your Config

Help the community by sharing your config:

  1. Go to html2rss-configs on GitHub
  2. Click "Fork" → "Add file" → Create domain.com.yml
  3. Paste your config → "Commit new file" → "Open pull request"

Need help? See our contribution guide for detailed instructions.

Troubleshooting

Common issues when writing configs:

  • No items found? Check your selectors with browser tools (F12) - the items.selector might not match the page structure
  • Invalid YAML? Use spaces, not tabs, and ensure proper indentation
  • Website not loading? Check the URL and try accessing it in your browser
  • Missing content? Try a browser-based rendering strategy during troubleshooting
  • Wrong data extracted? Verify your selectors are pointing to the right elements

Need more help? See our comprehensive troubleshooting guide or ask in GitHub Discussions.

Next Steps

🎉 Congratulations! You've learned the basics of creating html2rss configuration files.

What's Next?

For Beginners:

For Contributors: