(332) 331-8677

Sitemap.xml and robots.txt for Bitrix Regions

Create a file robots.php in the root folder of the site.

<?php

$host = $_SERVER["HTTP_HOST"];
$host = preg_replace("/\:\d+/is", "", $host);
if (isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == "on"){
    $http = "https";
}
else{
    $http = "http";
}
header("Content-Type: text/plain");?>
User-Agent: *
Disallow: /admin/
Disallow: /ajax/
Disallow: /personal/
Disallow: /filter/
Disallow: /catalog/compare/
Disallow: *ADD_TO_COMPARE
Disallow: *bx_bogus_href
Disallow: *?q=
Disallow: *?captcha
Disallow: *?manufacturers
Disallow: *index.php
Disallow: *?action
Disallow: *?login
Disallow: *?clear_cache
Disallow: */index.php
Disallow: /bitrix/
Disallow: /*show_include_exec_time=
Disallow: /*show_page_exec_time=
Disallow: /*show_sql_stat=
Disallow: /*bitrix_include_areas=
Disallow: /*clear_cache=
Disallow: /*clear_cache_session=
Disallow: /*ADD_TO_COMPARE_LIST
Disallow: /*ORDER_BY
Disallow: /*PAGEN
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*print_course=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*register=
Disallow: /*forgot_password=
Disallow: /*change_password=
Disallow: /*login=
Disallow: /*logout=
Disallow: /*auth=
Disallow: /*backurl=
Disallow: /*back_url=
Disallow: /*BACKURL=
Disallow: /*BACK_URL=
Disallow: /*back_url_admin=
Disallow: /*?utm_source=
Disallow: /*?bxajaxid=
Disallow: /*&bxajaxid=
Disallow: /*?view_result=
Disallow: /*&view_result=
Allow: /bitrix/components/
Allow: /bitrix/cache/
Allow: /bitrix/js/
Allow: /bitrix/templates/
Allow: /bitrix/panel/
Host: <?=$http?>://<?=$host.PHP_EOL;?>
Sitemap: <?=$http?>://<?=$host;?>/sitemap.xml

Next, create a file sitemap.php in the root folder of the site.

<?php

function sitemap_gen($sitemap_path, $site_url, $new_path){
    if (substr($sitemap_path, 0, 1) != '/'){
        $sitemap_path = '/'.$sitemap_path;
    }
    $sitemap_path = $_SERVER["DOCUMENT_ROOT"].$sitemap_path;
    if (substr($new_path, 0, 1) != '/'){
        $new_path = '/'.$new_path;
    }
    $new_path = $_SERVER["DOCUMENT_ROOT"].$new_path;

    $dyn_sitemap = '<?'.PHP_EOL.'$host = preg_replace("/\:\d+/is", "", $_SERVER["HTTP_HOST"]);'.PHP_EOL.
        'if (isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == "on"){'.PHP_EOL.
        '	$http = "https";'.PHP_EOL.
        '}'.PHP_EOL.
        'else{'.PHP_EOL.
        '	$http = "http";'.PHP_EOL.
        '}'.PHP_EOL.
        'header("Content-Type: text/xml");'.PHP_EOL;

    $sitemap = file_get_contents($sitemap_path);
    if (!$sitemap){
        return false;
    }

    // замены
    $search = Array(
        $site_url,
        'http:',
        'https:',
    );
    $replace = Array(
        '<?=$host?>',
        '<?=$http?>:',
        '<?=$http?>:'
    );

    $sitemap = str_replace($search, $replace, $sitemap);

    $sitemap = preg_replace('/(\<\?xml[^\>]+\>)/i', "echo '$1';?>".PHP_EOL, $sitemap);

    $dyn_sitemap .= $sitemap;

    if (!file_put_contents($new_path, $dyn_sitemap)){
        return false;
    }
    return true;
}
sitemap_gen('sitemap.xml', 'okean-kv.ru', 'sitemap_dyn.php');
sitemap_gen('sitemap-files.xml', 'okean-kv.ru', 'sitemap-files.php');
sitemap_gen('sitemap-iblock-49.xml', 'okean-kv.ru', 'sitemap-iblock-49.php');
sitemap_gen('sitemap-iblock-50.xml', 'okean-kv.ru', 'sitemap-iblock-50.php');
sitemap_gen('sitemap-iblock-51.xml', 'okean-kv.ru', 'sitemap-iblock-51.php');
sitemap_gen('sitemap-iblock-52.xml', 'okean-kv.ru', 'sitemap-iblock-52.php');
sitemap_gen('sitemap-iblock-53.xml', 'okean-kv.ru', 'sitemap-iblock-53.php');
sitemap_gen('sitemap-iblock-54.xml', 'okean-kv.ru', 'sitemap-iblock-54.php');
sitemap_gen('sitemap-iblock-55.xml', 'okean-kv.ru', 'sitemap-iblock-55.php');
sitemap_gen('sitemap-iblock-56.xml', 'okean-kv.ru', 'sitemap-iblock-56.php');
sitemap_gen('sitemap-iblock-57.xml', 'okean-kv.ru', 'sitemap-iblock-57.php');
sitemap_gen('sitemap-iblock-58.xml', 'okean-kv.ru', 'sitemap-iblock-58.php');
sitemap_gen('sitemap-iblock-60.xml', 'okean-kv.ru', 'sitemap-iblock-60.php');
sitemap_gen('sitemap-iblock-61.xml', 'okean-kv.ru', 'sitemap-iblock-61.php');
sitemap_gen('sitemap-iblock-62.xml', 'okean-kv.ru', 'sitemap-iblock-62.php');
sitemap_gen('sitemap-iblock-63.xml', 'okean-kv.ru', 'sitemap-iblock-63.php');
sitemap_gen('sitemap-iblock-64.xml', 'okean-kv.ru', 'sitemap-iblock-64.php');
sitemap_gen('sitemap-iblock-65.xml', 'okean-kv.ru', 'sitemap-iblock-65.php');
sitemap_gen('sitemap-iblock-71.xml', 'okean-kv.ru', 'sitemap-iblock-71.php');
sitemap_gen('sitemap-iblock-72.xml', 'okean-kv.ru', 'sitemap-iblock-72.php');
sitemap_gen('sitemap-iblock-73.xml', 'okean-kv.ru', 'sitemap-iblock-73.php');
sitemap_gen('sitemap-iblock-74.xml', 'okean-kv.ru', 'sitemap-iblock-74.php');

Now you need to generate a standard site map in the Bitrix admin panel. Go to Marketing-Search engine optimization-settings sitemap.xml -Click Add or Start generating the sitemap.

After creating the sitemap, you need to run the sitemap.php script. To do this, you need to enter in the browser's address bar https://yoursite.com/sitemap.php

The script will generate a file with a dynamic site map in the site root sitemap_dyn.php

Now you need to set up a redirect to the files we created robots.php and sitemap_dyn.php, to open them instead of the original files robots.txt and sitemap.xml. To do this, open the .htaccess file in the site root and add the following code to it:

RewriteRule ^robots\.txt$ /robots.php [L]
RewriteRule ^sitemap\.xml$ /sitemap_dyn.php [L]
RewriteRule ^sitemap-files\.xml$ /sitemap-files.php [L]
RewriteRule ^sitemap-iblock-49\.xml$ /sitemap-iblock-49.php [L]
RewriteRule ^sitemap-iblock-50\.xml$ /sitemap-iblock-50.php [L]
RewriteRule ^sitemap-iblock-51\.xml$ /sitemap-iblock-51.php [L]
RewriteRule ^sitemap-iblock-52\.xml$ /sitemap-iblock-52.php [L]
RewriteRule ^sitemap-iblock-53\.xml$ /sitemap-iblock-53.php [L]
RewriteRule ^sitemap-iblock-54\.xml$ /sitemap-iblock-54.php [L]
RewriteRule ^sitemap-iblock-55\.xml$ /sitemap-iblock-55.php [L]
RewriteRule ^sitemap-iblock-56\.xml$ /sitemap-iblock-56.php [L]
RewriteRule ^sitemap-iblock-57\.xml$ /sitemap-iblock-57.php [L]
RewriteRule ^sitemap-iblock-58\.xml$ /sitemap-iblock-58.php [L]
RewriteRule ^sitemap-iblock-60\.xml$ /sitemap-iblock-60.php [L]
RewriteRule ^sitemap-iblock-61\.xml$ /sitemap-iblock-61.php [L]
RewriteRule ^sitemap-iblock-62\.xml$ /sitemap-iblock-62.php [L]
RewriteRule ^sitemap-iblock-63\.xml$ /sitemap-iblock-63.php [L]
RewriteRule ^sitemap-iblock-64\.xml$ /sitemap-iblock-64.php [L]
RewriteRule ^sitemap-iblock-65\.xml$ /sitemap-iblock-65.php [L]
RewriteRule ^sitemap-iblock-71\.xml$ /sitemap-iblock-71.php [L]
RewriteRule ^sitemap-iblock-72\.xml$ /sitemap-iblock-72.php [L]
RewriteRule ^sitemap-iblock-73\.xml$ /sitemap-iblock-73.php [L]
RewriteRule ^sitemap-iblock-74\.xml$ /sitemap-iblock-74.php [L]

This is the end of creating dynamic robots.txt and sitemap.xml finished.