1. Forum
  2. USENET
  3. alt.bbs.synchronet

  • webv4 and web crawlers

    From Al@al@TRMB.remove-kc1-this to echicken on Mon Aug 10 13:12:31 2020
    From Newsgroup: alt.bbs.synchronet

    To: echicken
    Hello echicken,

    I have been messing around with the stock web interface and webv4 and I am just wondering if there is a way to block web crawlers from the message areas the way the stock interface does it with the robots.txt?

    Ttyl :-),
    Al

    ... Cursor: An expert in four-letter words

    ---
    þ Synchronet þ The Rusty MailBox - Penticton, BC Canada
    --- Synchronet 3.18a-Win32 NewsLink 1.113
    * Vertrauen - Riverside County, California - telnet://vert.synchro.net
    --- Synchronet 3.19c-Linux NewsLink 1.113
  • From echicken@echicken@ECBBS.remove-656-this to Al on Mon Aug 10 16:39:24 2020
    From Newsgroup: alt.bbs.synchronet

    To: Al
    Re: webv4 and web crawlers
    By: Al to echicken on Mon Aug 10 2020 13:12:31

    I have been messing around with the stock web interface and webv4 and I am just wondering if there
    is a way to block web crawlers from the message areas the way the stock interface does it with the
    robots.txt?

    You can put a robots.txt file inside the document root (as on any webserver); I don't remember the exact path offhand, but the directory name is 'root' and it has an 'index.xjs' file in it.

    You may also want to configure a web 'guest' user and limit its access to message areas, giving unauthenticated bots less surface area to crawl over. (In modopts.ini->[web], set 'guest' to the username of whatever account you want guests to be treated as.)

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com
    þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com
    --- Synchronet 3.18a-Win32 NewsLink 1.113
    * Vertrauen - Riverside County, California - telnet://vert.synchro.net
    --- Synchronet 3.19c-Linux NewsLink 1.113