There are various TXT files used for different purposes like informing crawler robots, domain name verification or identity verification. One of them is a robots.txt file. Sooner or later you will need to serve a robots.txt file with Django dynamically, in order to tell web crawler robots which parts of your website is open to search engines. Let's see how we can do it.
robots.txt: A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.
1. With HttpResponse in urlconf
This is simple and easy to understand. We define a string variable that provides the content of the TXT file and when the target URL accessed we serve that content in response with the content type text/plain.
from django.urls import path
from django.http import HttpResponse
ADS_TXT = 'Publisher ID: e98a0dce21'
ROBOTS_TXT_LINES = [
'User-agent: Googlebot',
'Disallow: /nogooglebot/',
'\n',
'User-agent: *',
'Allow: /',
]
ROBOTS_TXT = '\n'.join(ROBOTS_TXT_LINES)
urlpatterns = [
path('ads.txt', lambda r: HttpResponse(ADS_TXT, content_type="text/plain")),
path('robots.txt', lambda r: HttpResponse(ROBOTS_TXT, content_type="text/plain")),
]
2. As a template
With this method, we'll create a robots.txt file and serve it as a template.
User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Allow: /
from django.urls import path
from django.views.generic.base import TemplateView
urlpatterns = [
path(
'robots.txt',
TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]
3. With a custom view function
We can use function-based or class based views.
from django.http import HttpResponse
from django.views.decorators.http import require_GET
@require_GET # only allow GET requests
def ads_txt(request):
content = 'Publisher ID: e98a0dce21 \n'
return HttpResponse(content, content_type="text/plain")
from django.urls import path
from main.views import ads_txt
urlpatterns = [
path('robots.txt', ads_txt),
]
4. Via NGINX
As we will serve static files via NGINX (or something similar), this method looks more accurate to prefer. However, each of these methods in this post has advantages and disadvantages. For instance, while you can write tests in other methods, you can't with this one in Django.
Just add a location /robots.txt ... line to your NGINX configuration:
upstream server_http {
server backend:8000;
}
server {
listen 80;
# ...
location / {
proxy_pass http://server_http;
# ...
}
location /robots.txt {
alias /static/robots.txt;
}
# ...
}