Sources

A source is one feed of content: an RSS feed, a subreddit, an arXiv search, and so on. Each source has a source_type and a config object whose shape depends on the type. This page documents every supported type.

All sources share two common fields regardless of type:

Field	Default	Notes
fetch_interval	3600 seconds	Minimum 300 (5 minutes). The settings UI only offers fixed choices (15m / 30m / 1h / 6h / 24h), but the API accepts any integer ≥ 300.
is_active	true	Inactive sources are skipped by the fetch scheduler.

Sources with an identical (source_type, config) pair across different users are fetched once per cycle and fanned out to every subscriber — see Architecture.

RSS / Atom

{"url": "https://example.com/feed.xml"}

url is the only field. If it points at an HTML page rather than a raw feed, Shoebill tries to autodiscover a feed link on that page. atom is handled by the same fetcher as rss — there’s no configuration difference between the two types.

Reddit

{
  "subreddit": "MachineLearning",
  "sort": "hot",
  "limit": "25"
}

Field	Required	Default	Notes
subreddit	yes	—	No r/ prefix
sort	no	hot
limit	no	25
client_id / client_secret	no	—	Per-source Reddit API credentials. Falls back to the global REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET env vars if omitted; a per-source value takes priority over the global one.
username / password	no	—	Only needed for the OAuth “password” grant (script-type Reddit apps); without them, Shoebill authenticates as client_credentials (app-only).

Adding a Reddit source in the UI auto-fills the credential fields from any Reddit source you’ve already configured, so you only enter your API credentials once even when adding several subreddits.

Email (IMAP newsletters)

{
  "imap_host": "imap.example.com",
  "imap_port": "993",
  "username": "you@example.com",
  "password": "...",
  "folder": "INBOX",
  "subject_filter": "Weekly Digest"
}

Field	Required	Default	Notes
imap_host	yes	—
username / password	yes	—	Plain IMAP auth over IMAP4_SSL — no OAuth2 support
imap_port	no	993
folder	no	INBOX
subject_filter	no	—	Plain IMAP SUBJECT search string, not a filter language

Each matching email is treated as a “newsletter wrapper” — the LLM extracts the individual articles it contains and creates one NewsItem per article (see LLM Providers).

Warning

The password field takes any password, including your real account password. There’s no built-in distinction between a full account password and an app-specific password. Use an app password (most providers — Gmail, Fastmail, etc. — support generating one) rather than your real account password, since it’s stored so the fetcher can use it on every poll.

Mastodon

{"instance_url": "mastodon.social", "feed_type": "hashtag", "name": "opensource"}

Field	Required	Default	Notes
instance_url	yes	—	Scheme is added automatically if omitted
feed_type	no	hashtag	user, hashtag, or local
name	required unless feed_type=local	—	Account name (for user) or hashtag (for hashtag), without @/#

Uses each instance’s built-in RSS endpoints (/@user.rss, /tags/x.rss, /public/local.rss) rather than the Mastodon API — no auth needed.

Bluesky

{"feed_type": "user", "handle": "bsky.app", "limit": "20"}

Field	Required	Default	Notes
feed_type	no	user	user or search
handle	required for feed_type=user	—	Leading @ is stripped automatically
query	required for feed_type=search	—
limit	no	20	Capped at 100

Uses the public AT Protocol AppView API — no authentication required.

Lemmy

{"instance_url": "lemmy.world", "community": "technology", "sort": "Hot", "limit": "25"}

Field	Required	Default
instance_url	yes	—
community	yes	—
sort	no	Hot (Hot, New, TopDay, TopWeek)
limit	no	25

GitHub

Two modes, chosen via mode:

{"mode": "releases", "repo": "openai/openai-python", "limit": "20"}

{"mode": "trending", "language": "python", "since": "daily"}

Field	Required	Default	Notes
mode	no	releases	releases or trending
repo	required for releases	—	"owner/repo"
limit	no	20	Capped at 100
token	no	—	A GitHub PAT, for higher API rate limits
prerelease	no	false	Include prereleases; "true"/"false" string
language	no	all languages	For trending mode
since	no	daily	For trending mode: daily, weekly, monthly

arXiv

{"query": "transformer architecture"}

query is the only field — searches arXiv’s official Atom API directly (not Google Scholar, despite this fetcher historically also being reachable under a "scholar" type name — that legacy alias has since been removed; any existing sources that used it were migrated to "arxiv" automatically).

arXiv papers also keep their own subject categories. arXiv tags every paper with a curated taxonomy (cs.LG, quant-ph, …), and those tags are translated to readable labels — cs.LG becomes “machine learning” — and merged into the paper’s keywords alongside the ones the LLM infers. This gives clustering a much steadier signal than inferred wording alone: two papers both tagged cs.CL and cs.LG are reliably related, where two generated keyword sets describing the same idea can disagree on phrasing. Categories outside the curated list fall back to their archive name (math.AT → “mathematics”), so nothing is silently dropped.

Telegram

{"channel": "durov"}

channel (no @ prefix) is the only field. Scrapes the public t.me/s/{channel} web preview; channels without a public preview fail gracefully (empty result, logged warning) rather than erroring.

Generic scraper

For sites with no feed at all:

{
  "url": "https://example.com/news",
  "item_selector": "article.post",
  "title_selector": "h2",
  "link_selector": "a",
  "content_selector": ".post-body",
  "fetch_full_articles": "true"
}

Field	Required	Default	Notes
url	yes	—
item_selector	yes	—	CSS selector matching each article’s container element
title_selector	no	h1,h2,h3
link_selector	no	a
content_selector	no	—	Falls back to the full container’s text if omitted
base_url	no	= url	For resolving relative links
fetch_full_articles	no	true	Follow each article’s link to fetch the full text when the listing excerpt is under 300 characters

robots.txt is checked both when the source is created and on every subsequent fetch — a scraper source can’t be created against a site whose robots.txt disallows it.

Auto-detect selectors: rather than writing CSS selectors by hand, the “Auto-detect” button in the source form calls the configured LLM to propose them — it fetches the page, strips it to a simplified tag/class/href skeleton, asks the LLM for item_selector / title_selector / link_selector / content_selector, validates the result by actually running the selectors against the page, and retries once if the first attempt matches zero items.