Google XML Sitemap 404

[denialdesign]

I've generated an xml sitemap for 3 stores in a multi store set up (Magento 1.9.4.0)

sitemap_store1.xml

sitemap_store2.xml

sitemap_store3.xml

The files are generated successfully and I can see and view the files on the server.

When I try to view the files in my browser I get a Magento 404 error.

I've also tried with the default sitemap.xml name and tried different folders for each sitemap and still get 404 page.

When I remove the root .htaccess file the sitemap displays in my browser.

Here's my .htaccess file contents:

############################################
## uncomment these lines for CGI mode
## make sure to specify the correct cgi php binary file name
## it might be /cgi-bin/php-cgi

#    Action php5-cgi /cgi-bin/php5-cgi
#    AddHandler php5-cgi .php

############################################
## GoDaddy specific options

#   Options -MultiViews

## you might also need to add this line to php.ini
##     cgi.fix_pathinfo = 1
## if it still doesn't work, rename php.ini to php5.ini

############################################
## this line is specific for 1and1 hosting

    #AddType x-mapp-php5 .php
    #AddHandler x-mapp-php5 .php

############################################
## default index file

    DirectoryIndex index.php

<IfModule mod_php5.c>

############################################
## adjust memory limit

#    php_value memory_limit 64M
    php_value memory_limit 256M
    php_value max_execution_time 18000

############################################
## disable magic quotes for php request vars

    php_flag magic_quotes_gpc off

############################################
## disable automatic session start
## before autoload was initialized

    php_flag session.auto_start off

############################################
## enable resulting html compression

    #php_flag zlib.output_compression on

###########################################
# disable user agent verification to not break multiple image upload

    php_flag suhosin.session.cryptua off

###########################################
# turn off compatibility with PHP4 when dealing with objects

    php_flag zend.ze1_compatibility_mode Off

</IfModule>

<IfModule mod_security.c>
###########################################
# disable POST processing to not break multiple image upload

    SecFilterEngine Off
    SecFilterScanPOST Off
</IfModule>

<IfModule mod_deflate.c>

############################################
## enable apache served files compression
## http://developer.yahoo.com/performance/rules.html#gzip

    # Insert filter on all content
    SetOutputFilter DEFLATE
    # Insert filter on selected content types only
    AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript

    # Netscape 4.x has some problems...
    BrowserMatch ^Mozilla/4 gzip-only-text/html

    # Netscape 4.06-4.08 have some more problems
    BrowserMatch ^Mozilla/4\.0[678] no-gzip

    # MSIE masquerades as Netscape, but it is fine
    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    # Don't compress images
    #SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary

    # Make sure proxies don't deliver the wrong content
    #Header append Vary User-Agent env=!dont-vary

</IfModule>

<IfModule mod_ssl.c>

############################################
## make HTTPS env vars available for CGI mode

    SSLOptions StdEnvVars

</IfModule>

<IfModule mod_rewrite.c>

############################################
## enable rewrites

    Options +FollowSymLinks
    RewriteEngine on

############################################
## you can put here your magento root folder
## path relative to web root

    #RewriteBase /magento/

############################################
## uncomment next line to enable light API calls processing

#    RewriteRule ^api/([a-z][0-9a-z_]+)/?$ api.php?type=$1 [QSA,L]

############################################
## rewrite API2 calls to api.php (by now it is REST only)

    RewriteRule ^api/rest api.php?type=rest [QSA,L]

############################################
## workaround for HTTP authorization
## in CGI environment

    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

############################################
## TRACE and TRACK HTTP methods disabled to prevent XSS attacks

    RewriteCond %{REQUEST_METHOD} ^TRAC[EK]
    RewriteRule .* - [L,R=405]

<IfModule mod_setenvif.c>
    <IfModule mod_headers.c>

        ############################################
        # X-Content-Type-Options: nosniff disable content-type sniffing on some browsers.
        Header set X-Content-Type-Options: nosniff

        ############################################
        # This header forces to enables the Cross-site scripting (XSS) filter in browsers (if disabled)
        BrowserMatch \bMSIE\s8 ie8
        Header set X-XSS-Protection: "1; mode=block" env=!ie8

    </IfModule>
</IfModule>

############################################
## redirect for mobile user agents

    #RewriteCond %{REQUEST_URI} !^/mobiledirectoryhere/.*$
    #RewriteCond %{HTTP_USER_AGENT} "android|blackberry|ipad|iphone|ipod|iemobile|opera mobile|palmos|webos|googlebot-mobile" [NC]
    #RewriteRule ^(.*)$ /mobiledirectoryhere/ [L,R=302]

############################################
## always send 404 on missing files in these folders

    RewriteCond %{REQUEST_URI} !^/(media|skin|js)/

############################################
## never rewrite for existing files, directories and links

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-l

RewriteCond %{HTTP_HOST} ^.*?site1\.co\.uk$ [NC]
RewriteRule ^robots\.txt$ robots-site1.txt [NC, L]
RewriteCond %{HTTP_HOST} ^.*?site2\.co\.uk$ [NC]
RewriteRule ^robots\.txt$ robots-site2.txt [NC, L]
RewriteCond %{HTTP_HOST} ^.*?site3\.co\.uk$ [NC]
RewriteRule ^robots\.txt$ robots-site3.txt [NC, L]

############################################
## rewrite everything else to index.php

    
RewriteRule .* index.php [L]

</IfModule>


############################################
## Prevent character encoding issues from server overrides
## If you still have problems, use the second line instead

    AddDefaultCharset Off
    #AddDefaultCharset UTF-8

<IfModule mod_expires.c>

############################################
## Add default Expires header
## http://developer.yahoo.com/performance/rules.html#expires

    ExpiresDefault "access plus 1 year"

</IfModule>

############################################
## By default allow all access

    Order allow,deny
    Allow from all

###########################################
## Deny access to release notes to prevent disclosure of the installed Magento version

    <Files RELEASE_NOTES.txt>
        order allow,deny
        deny from all
    </Files>

############################################
## If running in cluster environment, uncomment this
## http://developer.yahoo.com/performance/rules.html#etags

    #FileETag none

###########################################
## Deny access to cron.php
    <Files cron.php>

############################################
## uncomment next lines to enable cron access with base HTTP authorization
## http://httpd.apache.org/docs/2.2/howto/auth.html
##
## Warning: .htpasswd file should be placed somewhere not accessible from the web.
## This is so that folks cannot download the password file.
## For example, if your documents are served out of /usr/local/apache/htdocs
## you might want to put the password file(s) in /usr/local/apache/.

        #AuthName "Cron auth"
        #AuthUserFile ../.htpasswd
        #AuthType basic
        #Require valid-user

############################################

        Order allow,deny
        Deny from all

    </Files>

[syedhasan]

Hi @denialdesign 

I think you give permission to your folder or follow steps to manual creation of Google Sitemap

1. Sitemap File Configuration

      First, we need to create a sitemap file in Magento. This is basically a pointer to tell     

      Magento where to create a sitemap file and how to name it. Log in to Magento

      Administration and go to:

    Catalog -> Google Sitemap

1. FTP to the server
2. create sitemap folder
3. chmod the folder to 777

On some servers, previous steps will not be required, but it is better to create the file first so that Magento can edit it.

Let’s go to Magento administration now and click on “Add Sitemap” button. Just insert the default values.

After this step, we just told Magento where to create a sitemap file. It is not yet created.

 2. Configure Sitemap

 Sitemap can be configured from the interface System -> Configuration -> Catalog->Google Sitemap. If you are not sure what does it all mean, check XML Tag definitions section from “Sitemaps XML format” document.

3. Create Sitemap Manually

Looks like Magento folks were thinking that sitemap should only be created by a cron job. Yes, it can be done that way also. Be sure to read Wiki article How to Set Up a Cron Job to get familiar with it. If you are looking for manual creation, quit your search for “Create Sitemap” button. You will not find it. But there is a workaround. You can enter a manual URL to execute sitemap creation:

http://www.yourstore.com/index.php/admin/sitemap/generate/sitemap_id/[ID of Sitemap from step 1]

Of course, replace the content with square brackets with actual Sitemap ID.

------------------------------------------------
If you've found one of my answers useful, please give"Kudos" or "Accept as Solution"

[denialdesign]

@syedhasantry reading my post again. The sitemap is generating correctly. I just can't see it in the browser.

[denialdesign · Accepted solution]

I got the sitemap to display by adding the following lines to my .htaccess file:

RewriteCond %{REQUEST_URI} !^/sitemap_site1\.xml$ 
RewriteCond %{REQUEST_URI} !^/sitemap_site2\.xml$ 
RewriteCond %{REQUEST_URI} !^/sitemap_site3\.xml$ 

[maritalaru7a0d]

On this site see the travis merch sitemap is not clickable. Is that problem for Googlebot or not? Suggest options for other sitemap plugins.

[maritalaru7a0d]

On this site see the travis merch sitemap is not clickable. Is that problem for Googlebot or not? Suggest options for other sitemap plugins.