User-Agent: * Disallow: /directory.bml Sitemap: http://www.livejournal.com/sitemap.xml # # Blocked journals aren't listed here because robots.txt files # can't be above 50k or so, depending on the spider. # # Instead, blocked journals have HTML inserted in them which # should prevent behaved spiders from indexing it. # # Note that http://username.livejournal.com journals have an # autogenerated robots.txt, since it can be small. #