How can I put my robots.txt file to the path www.domain/robots.txt? No server is used, its only frontend with react router.

robots.txt --> in root folder ./

app.js --> in src folder ./src/

(...)

export class App extends React.Component {
 render() {
  return (
   <div>
    <Switch>
      <Route exact path='/stuff' ponent={Stuff}/>
      <Route exact path='/' ponent={HomePage}/>
    </Switch>
   </div>
  )
 }
}

If I test it locally, it works OK, localhost:4000/robots.txt opens the file properly in the browser. However I am afraid it is not case in production environment? Any ideas? Thank you in advance

How can I put my robots.txt file to the path www.domain./robots.txt? No server is used, its only frontend with react router.

robots.txt --> in root folder ./

app.js --> in src folder ./src/

(...)

export class App extends React.Component {
 render() {
  return (
   <div>
    <Switch>
      <Route exact path='/stuff' ponent={Stuff}/>
      <Route exact path='/' ponent={HomePage}/>
    </Switch>
   </div>
  )
 }
}

If I test it locally, it works OK, localhost:4000/robots.txt opens the file properly in the browser. However I am afraid it is not case in production environment? Any ideas? Thank you in advance

• javascript
• reactjs
• react-router
• robots.txt

• Is your prod environment an npm build? If so, did robots.txt end up in the build? I would also remend using hash routing. This removes ambiguity with static files mixed in. – F. P. Freely Commented Mar 13, 2019 at 17:54

2 Answers 2

Just put robots.txt (and sitemap.xml if any) in build folder.

I did this and deployed it to firebase and I can access robots.txt through https://my-site.firebaseapp./robots.txt and sitemap.xml through https://my-site.firebaseapp./sitemap.xml

You should follow like mine and access robots.txt through yoursite./robots.txt

Note: If your build folder gets deleted before performing npm run build, then you need to again put robots.txt in the build folder.

UPDATE:

if you put robots.txt and sitemap.xml in the public folder, then they will auto get copied to the build folder every time you perform npm run build

If you are using CRA (React) or NextJS, you should place the sitemap.xml & robots.txt file in the app's public folder.

Official NextJS Documentaion

After you push it to production, you can verify it by hitting the url : you-example-domain/sitemap.xml or you-example-domain/robots.txt