xenforo robots.txt erstellen und optimieren

Dieses Thema im Forum "XenForo Tipps & Tricks" wurde erstellt von Maestro2k5, 7. Juli 2015.

  1. Maestro2k5

    Maestro2k5 Administrator Lizenznehmer

    892
    361
    7. Juli 2015
    Erfurt
    Hallo @ All,

    vielen haben sicher schon eine robots.txt aber für alle anderen hier ein kleiner Ansatz. Dies ist meine robots.txt die ich für meine Projekte verwende.

    Die robots.txt sollte im Root-Verzeichnis eures Projektes liegen.

    Code:
    User-agent: *
    Disallow: /
    
    User-agent: Googlebot
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-Agent: Googlebot-Mobile
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-Agent: Googlebot-Image
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-Agent: Googlebot-News
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-Agent: facebookexternalhit
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: msnbot
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: msnbot-media
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: bingbot
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: Slurp
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: ia_archiver
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: pagespeedbot
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: Websnapr
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    User-agent: Twitterbot
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /beobachtete/
    Disallow: /chat/
    Disallow: /conversations/
    Disallow: /find-new/
    Disallow: /finde-neue/
    Disallow: /help/
    Disallow: /hilfe/
    Disallow: /letzte-aktivitaeten/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /members/
    Disallow: /misc/
    Disallow: /mitglieder/
    Disallow: /mobiquo/
    Disallow: /online/
    Disallow: /profil/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /unterhaltungen/
    Disallow: /watched/
    Disallow: /info/
    
    
    Sitemap: https://www.android-port.de/sitemap.php
    Wie sieht eure robots.txt aus?
     
  2. au lait

    au lait Administrator Lizenznehmer

    1.904
    940
    7. Juli 2015
    Singen
    hier eine Sitemap wie ich sie zB bei macuser.de nutze


    Code:
    User-agent: Mediapartners-Google*
    Allow: /
    
    User-agent: Slurp
    Crawl-delay: 60
    
    User-agent: Teoma
    Crawl-delay: 60
    
    User-agent: *
    Disallow: /misc
    Disallow: /help
    Disallow: /search
    Disallow: /members
    Disallow: /register
    Disallow: /login
    Disallow: /online
    Disallow: /attachments
    Disallow: /lost-password
    Disallow: /recent-activity
    
    
    Sitemap: http://www.macuser.de/sitemap.php
     
    Maestro2k5 gefällt das.
OK Mehr Informationen

Diese Seite verwendet Cookies. Mit Deinem Klick auf OK, stimmst Du dem setzen von Cookies zu. Andernfalls werden keine gesetzt, was die Funktionalität einschränkt.