Limiter

Bot protection / IP rate limitation. The intention of rate limitation is to limit suspicious requests from an IP. The motivation behind this is the fact that SearXNG passes through requests from bots and is thus classified as a bot itself. As a result, the SearXNG engine then receives a CAPTCHA or is blocked by the search engine (the origin) in some other way.

To avoid blocking, the requests from bots to SearXNG must also be blocked, this is the task of the limiter. To perform this task, the limiter uses the methods from the Bot Detection:

  • Analysis of the HTTP header in the request / Probe HTTP headers can be easily bypassed.

  • Block and pass lists in which IPs are listed / IP lists are hard to maintain, since the IPs of bots are not all known and change over the time.

  • Detection & dynamically Rate limit of bots based on the behavior of the requests. For dynamically changeable IP lists a Redis database is needed.

The prerequisite for IP based methods is the correct determination of the IP of the client. The IP of the client is determined via the X-Forwarded-For HTTP header.

Attention

A correct setup of the HTTP request headers X-Forwarded-For and X-Real-IP is essential to be able to assign a request to an IP correctly:

Enable Limiter

To enable the limiter activate:

server:
  ...
  limiter: true  # rate limit the number of request on the instance, block some bots

and set the redis-url connection. Check the value, it depends on your redis DB (see redis:), by example:

redis:
  url: unix:///usr/local/searxng-redis/run/redis.sock?db=0

Configure Limiter

The methods of Bot Detection the limiter uses are configured in a local file /etc/searxng/limiter.toml. The defaults are shown in limiter.toml / Don’t copy all values to your local configuration, just enable what you need by overwriting the defaults. For instance to activate the link_token method in the Method ip_limit you only need to set this option to true:

[botdetection.ip_limit]
link_token = true

limiter.toml

In this file the limiter finds the configuration of the Bot Detection:

[real_ip]

# Number of values to trust for X-Forwarded-For.

x_for = 1

# The prefix defines the number of leading bits in an address that are compared
# to determine whether or not an address is part of a (client) network.

ipv4_prefix = 32
ipv6_prefix = 48

[botdetection.ip_limit]

# To get unlimited access in a local network, by default link-lokal addresses
# (networks) are not monitored by the ip_limit
filter_link_local = false

# activate link_token method in the ip_limit method
link_token = false

[botdetection.ip_lists]

# In the limiter, the ip_lists method has priority over all other methods -> if
# an IP is in the pass_ip list, it has unrestricted access and it is also not
# checked if e.g. the "user agent" suggests a bot (e.g. curl).

block_ip = [
  # '93.184.216.34',  # IPv4 of example.org
  # '257.1.1.1',      # invalid IP --> will be ignored, logged in ERROR class
]

pass_ip = [
  # '192.168.0.0/16',      # IPv4 private network
  # 'fe80::/10'            # IPv6 linklocal / wins over botdetection.ip_limit.filter_link_local
]

# Activate passlist of (hardcoded) IPs from the SearXNG organization,
# e.g. `check.searx.space`.
pass_searxng_org = true

Implementation

searx.limiter.initialize(app: Flask, settings)[source]

Install the limiter

searx.limiter.is_installed()[source]

Returns True if limiter is active and a redis DB is available.

searx.limiter.pre_request()[source]

See flask.Flask.before_request

searx.limiter.LIMITER_CFG = PosixPath('/etc/searxng/limiter.toml')

Local Limiter configuration.

searx.limiter.LIMITER_CFG_SCHEMA = PosixPath('/home/user/Desktop/shared/documentation/docs/searxng/repository/searx/limiter.toml')

Base configuration (schema) of the botdetection.