I’ve been on a big “there’s nothing new under the sun” kick lately, so let’s take a look at Infoseek’s sitelist.txt proposal from 1997, and deepleap.xml, from DeepLeap back in 2000. Or hey, there’s HTML’s LINK, which lets you describe relationships that can encompass a lot of a site’s content. And hey, there might even be formats for describing a feed of a site’s content and when it gets updated.