[Django]-How can I check if a URL exists with Django’s validators?

Problem

from django.core.validators import URLValidator says that www.google.ro is invalid. Which is wrong in my point of view. Or at least not enough.

How to solve it?

The clue Is to look at the source code for models.URLField, you will see that it uses forms.FormField as a validator. Which does more than URLValidator from above

Solution

If I want to validate a url like http://www.google.com or like www.google.ro, I would do the following:

from django.forms import URLField

def validate_url(url):
    url_form_field = URLField()
    try:
        url = url_form_field.clean(url)
    except ValidationError:
        return False
    return True

I found this useful. Maybe it helps someone else.

Anything based on the verify_exists parameter to django.core.validators.URLValidator will stop working with Django 1.5 — the documentation helpfully says nothing about this, but the source code reveals that using that mechanism in 1.4 (the latest stable version) leads to a DeprecationWarning (you’ll see it has been removed completely in the development version):

if self.verify_exists:
    import warnings
    warnings.warn(
        "The URLField verify_exists argument has intractable security "
        "and performance issues. Accordingly, it has been deprecated.",
        DeprecationWarning
        )

There are also some odd quirks with this method related to the fact that it uses a HEAD request to check URLs — bandwidth-efficient, sure, but some sites (like Amazon) respond with an error (to HEAD, where the equivalent GET would have been fine), and this leads to false negative results from the validator.

I would also (a lot has changed in two years) recommend against doing anything with urllib2 in a template — this is completely the wrong part of the request/response cycle to be triggering potentially long-running operations: consider what happens if the URL does exist, but a DNS problem causes urllib2 to take 10 seconds to work that out. BAM! Instant 10 extra seconds on your page load.

I would say the current best practice for making possibly-long-running tasks like this asynchronous (and thus not blocking page load) is using django-celery; there’s a basic tutorial which covers using pycurl to check a website, or you could look into how Simon Willison implemented celery tasks (slides 32-41) for a similar purpose on Lanyrd.

It took an additional:

from django.core.exceptions import ValidationError

for it to work for me. Just saying ;0)

from django.core.validators import URLValidator
from django.core.exceptions import ValidationError

validate = URLValidator(verify_exists=True)    
value = request.GET.get('url', None)

if value:        
    try:
        validate(value)
    except ValidationError, e:
        print e

validate(value) fails if the url is not preceeded with a schema like http://. I wonder if that is by design.

I have not seen the answer here. It might helpful to someone else.

from django import forms
f = forms.URLField()
try:
    f.clean(http://example.com)
    print "valid url"
except:
    print "invalid url"

See:
http://www.agmweb.ca/2009-04-19-django-urlpatterns—its-more-than-just-urls/

In django 1.10 i now use:

from django.core.urlresolvers import RegexURLResolver, Resolver404

if 'next' in request.GET.keys():
    n = request.GET["next"].strip('/') + "/"
    resolver = RegexURLResolver(r'', urls)
    try:
        callback, callback_args, callback_kwargs = resolver.resolve(n)
        return HttpResponseRedirect(str(request.GET["next"]))
    except Resolver404:
        raise PermissionDenied("This page is not available")

For new version of Django, you should just when a non valid url is given it raises a Validation exception django.core.exceptionValidationError

from django.core.validators import URLValidator                                                                                                                             
validate = URLValidator()                                                                                                                                                                  
validate('http://www.google.com')                                                                                                                                                                
validate('http://www2.google.com')                                                                                                                                                               
validate('httpAA://www2.google.com')                                                                                                                                                             
     Traceback (most recent call last):                                                                                                                                                                     
     File "<console>", line 1, in <module>                                                                                                                                                                
     File "/etl/.venv/lib/python3.8/site- 
     packages/django/core/validators.py", line 110, in __call__                                                                                                         
     raise ValidationError(self.message, code=self.code, params={'value': value})                                                                                                                     
     django.core.exceptions.ValidationError: ['Enter a valid URL.']