FreeBSD Manual Pages
SFEEDRC(5) File Formats Manual SFEEDRC(5) NAME sfeedrc -- sfeed_update(1) configuration file DESCRIPTION sfeedrc is the configuration file for sfeed_update(1) and is evaluated as a shellscript. VARIABLES sfeedpath can be set for the directory to store the TAB-separated feed files. The default is $HOME/.sfeed/feeds. maxjobs can be used to change the amount of concurrent feed() jobs. The default is 16. FUNCTIONS feeds() This function is the required "main" entry-point function called from sfeed_update(1). feed(name, feedurl, basesiteurl, encoding) Inside the feeds() function feeds can be defined by calling the feed() function. Its arguments are: name Name of the feed, this is also used as the filename for the TAB-separated feed file. The feed name cannot con- tain the '/' character because it is a path separator, they will be replaced with '_'. Each name should be unique. feedurl URL to fetch the RSS/Atom data from. This is usually a HTTP or HTTPS URL. [basesiteurl] Base URL of the feed links. This argument allows fix- ing relative item links. According to the RSS and Atom specification, feeds should always have absolute URLs, but this is not al- ways the case in practice. [encoding] Feeds are converted from this encoding to UTF-8. The encoding should be a usable character-set name for the iconv(1) tool. OVERRIDE FUNCTIONS Because sfeed_update(1) is a shellscript each function can be overrid- den to change its behaviour. Notable functions are: fetch(name, url, feedfile) Fetch feed from URL and write the data to stdout. Its argu- ments are: name Feed name. url URL to fetch. feedfile Used feedfile (useful for comparing modification times). By default the tool curl(1) is used. convertencoding(name, from, to) Convert data from stdin from one text-encoding to another and write it to stdout. Its arguments are: name Feed name. from From text-encoding. to To text-encoding. By default the tool iconv(1) is used. parse(name, feedurl, basesiteurl) Read RSS/Atom XML data from stdin, convert and write it as sfeed(5) data to stdout. Its arguments are: name Feed name. feedurl URL of the feed. basesiteurl Base URL of the feed links. This argument allows to fix relative item links. filter(name, url) Filter sfeed(5) data from stdin and write it to stdout. Its arguments are: name Feed name. url URL of the feed. merge(name, oldfile, newfile) Merge sfeed(5) data of oldfile with newfile and write it to stdout. Its arguments are: name Feed name. oldfile Old file. newfile New file. order(name, url) Sort sfeed(5) data from stdin and write it to stdout. Its ar- guments are: name Feed name. url URL of the feed. EXAMPLES An example configuration file is included named sfeedrc.example and also shown below: #sfeedpath="$HOME/.sfeed/feeds" # list of feeds to fetch: feeds() { # feed <name> <feedurl> [basesiteurl] [encoding] feed "codemadness" "https://www.codemadness.org/atom_content.xml" feed "explosm" "http://feeds.feedburner.com/Explosm" feed "golang github releases" "https://github.com/golang/go/releases.atom" feed "linux kernel" "https://www.kernel.org/feeds/kdist.xml" "https://www.kernel.org" feed "reddit openbsd" "https://old.reddit.com/r/openbsd/.rss" feed "slashdot" "http://rss.slashdot.org/Slashdot/slashdot" "http://slashdot.org" feed "tweakers" "http://feeds.feedburner.com/tweakers/mixed" "http://tweakers.net" "iso-8859-1" # get youtube Atom feed: curl -s -L 'https://www.youtube.com/user/gocoding/videos' | sfeed_web | cut -f 1 feed "youtube golang" "https://www.youtube.com/feeds/videos.xml?channel_id=UCO3LEtymiLrgvpb59cNsb8A" feed "xkcd" "https://xkcd.com/atom.xml" "https://xkcd.com" } To change the default curl(1) options for fetching the data, the fetch() function can be overridden and added at the top of the sfeedrc file, for example: # fetch(name, url, feedfile) fetch() { # allow for 1 redirect, set User-Agent, timeout is 15 seconds. curl -L --max-redirs 1 -H "User-Agent: 007" -f -s -m 15 \ "$2" 2>/dev/null } Caching, incremental data updates and bandwidth saving For HTTP servers that support it some bandwidth saving can be done by changing some of the default curl options. These options can come at a cost of some privacy, because it exposes additional metadata from the previous request. • The curl ETag options (--etag-save and --etag-compare) can be used to store and send the previous ETag header value. curl version 7.73+ is recommended for it to work properly. • The curl -z option can be used to send the modification date of a local file as a HTTP If-Modified-Since request header. The server can then respond if the data is modified or not or respond with only the incremental data. • The curl --compressed option can be used to indicate the client supports decompression. Because RSS/Atom feeds are textual XML data this generally compresses very well. • The example below also sets the User-Agent to sfeed, because some CDNs block HTTP clients based on the User-Agent request header. Example: mkdir -p "$HOME/.sfeed/etags" "$HOME/.sfeed/lastmod" # fetch(name, url, feedfile) fetch() { basename="$(basename "$3")" etag="$HOME/.sfeed/etags/${basename}" lastmod="$HOME/.sfeed/lastmod/${basename}" output="${sfeedtmpdir}/feeds/${filename}.xml" curl \ -f -s -m 15 \ -L --max-redirs 0 \ -H "User-Agent: sfeed" \ --compressed \ --etag-save "${etag}" --etag-compare "${etag}" \ -R -o "${output}" \ -z "${lastmod}" \ "$2" 2>/dev/null || return 1 # succesful, but no file written: assume it is OK and Not Modified. [ -e "${output}" ] || return 0 # use server timestamp from curl -R to set Last-Modified. touch -r "${output}" "${lastmod}" 2>/dev/null cat "${output}" 2>/dev/null # use write output status, other errors are ignored here. fetchstatus="$?" rm -f "${output}" 2>/dev/null return "${fetchstatus}" } The README file has more examples. SEE ALSO curl(1), iconv(1), sfeed_update(1), sh(1) AUTHORS Hiltjo Posthuma <hiltjo@codemadness.org> FreeBSD ports 15.0 February 9, 2025 SFEEDRC(5)
NAME | DESCRIPTION | VARIABLES | FUNCTIONS | OVERRIDE FUNCTIONS | EXAMPLES | SEE ALSO | AUTHORS
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=sfeedrc&sektion=5&manpath=FreeBSD+Ports+15.0>