Makefile & ./manage
¶
All relevant build and development tasks are implemented in the ./manage script and for CI or IDE integration a small git://Makefile wrapper is available. If you are not familiar with Makefiles, we recommend to read gnu-make introduction.
The usage is simple, just type make {target-name}
to build a target.
Calling the help
target gives a first overview (make help
):
nvm.: use nvm (without dot) to execute nvm commands directly
install : install NVM locally at /home/user/Desktop/shared/documentation/docs/searxng/repository/.nvm
clean : remove NVM installation
status : prompt some status informations about nvm & node
nodejs : install Node.js latest LTS
cmd ... : run command ... in NVM environment
bash : start bash interpreter with NVM environment sourced
buildenv:
rebuild ./utils/brand.env
webapp.:
run : run developer instance
docs.:
html : build HTML documentation
live : autobuild HTML documentation while editing
gh-pages : deploy on gh-pages branch
prebuild : build reST include files (./build/docs/includes)
clean : clean documentation build
docker.:
build : build docker image
push : build and push docker image
gecko.driver:
download & install geckodriver if not already installed (required for
robot_tests)
redis:
build : build redis binaries at /home/user/Desktop/shared/documentation/docs/searxng/repository/dist/redis/6.2.6/amd64
install : create user (searxng-redis) and install systemd service (searxng-redis)
help : show more redis commands
py.:
build : Build python packages at ./dist
clean : delete virtualenv and intermediate py files
pyenv.:
install : developer install of SearXNG into virtualenv
uninstall : uninstall developer installation
cmd ... : run command ... in virtualenv
OK : test if virtualenv is OK
pypi.upload:
Upload python packages to PyPi (to test use pypi.upload.test)
format.:
python : format Python code source using black
pygments.:
less : build LESS files for pygments
go.:
ls : list golang binary archives (stable)
golang : (re-) install golang binary in user's $HOME/local folder
install : install go package in user's $HOME/go-apps folder
bash : start bash interpreter with golang environment sourced
node.:
env : download & install SearXNG's npm dependencies locally
env.dev : download & install developer and CI tools
clean : drop locally npm installations
weblate.:
push.translations: push translation changes from SearXNG to Weblate's counterpart
to.translations: Update 'translations' branch with last additions from Weblate.
data.:
all : update searx/sxng_locales.py and searx/data/*
traits : update searx/data/engine_traits.json & searx/sxng_locales.py
useragents: update searx/data/useragents.json with the most recent versions of Firefox
test.:
yamllint : lint YAML files (YAMLLINT_FILES)
pylint : lint PYLINT_FILES, searx/engines, searx & tests
pyright : static type check of python sources
black : check black code format
unit : run unit tests
coverage : run unit tests with coverage
robot : run robot test
rst : test .rst files incl. README.rst
clean : clean intermediate test stuff
themes.:
all : build all themes
live : to get live builds of CSS & JS use 'LIVE_THEME=simple make run'
simple.:
build : build simple theme
test : test simple theme
static.build.: [build] /static
commit : build & commit /static folder
drop : drop last commit if it was previously done by static.build.commit
restore : git restore of the /static folder (after themes.all)
environment ...
SEARXNG_REDIS_URL :
----
run - run developer instance
install - developer install of SearxNG into virtualenv
uninstall - uninstall developer installation
clean - clean up working tree
search.checker - check search engines
test - run shell & CI tests
test.shell - test shell scripts
ci.test - run CI tests
The Makefile targets are implemented for comfort, if you can do without tab-completion and need to have a more granular control, use git://manage without the Makefile wrappers.
$ ./manage help
Python environment (make install
)¶
We do no longer need to build up the virtualenv manually. Jump into your git
working tree and release a make install
to get a virtualenv with a
developer install of SearXNG (git://setup.py).
$ cd ~/searxng-clone
$ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
If you release make install
multiple times the installation will only
rebuild if the sha256 sum of the requirement files fails. With other words:
the check fails if you edit the requirements listed in
git://requirements-dev.txt and git://requirements.txt).
$ make install
PYENV OK
PYENV [virtualenv] requirements.sha256 failed
[virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt
[virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt
...
PYENV [virtualenv] installing ./requirements*.txt into local/py3
...
PYENV OK
PYENV [install] pip install -e 'searx[test]'
...
Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
If you think, something goes wrong with your ./local environment or you change the git://setup.py file, you have to call make clean.
make buildenv
¶
Rebuild instance’s environment with the modified settings from the brand: and server: section of your settings.yml.
What is the git://utils/brand.env needed for and why do you need to rebuild it if necessary?
Short answer: installation and maintenance scripts are running outside of instance’s runtime environment and need some values defined in the runtime environment.
All the SearXNG setups are centralized in the settings.yml file. This
setup is available as long we are in a installed instance. E.g. the
installed instance on the server or the installed developer instance at
./local
(the later one is created by a make install or
make run).
Tasks running outside of an installed instance, especially installation and maintenance tasks running at (pre-) installation time do not have access to the SearXNG setup (from a installed instance). Those tasks need a build environment.
The make buildenv
target will update the build environment in:
Tasks running outside of an installed instance, need the following settings from the YAML configuration:
SEARXNG_URL
from server.base_url (akaPUBLIC_URL
)SEARXNG_BIND_ADDRESS
from server.bind_addressSEARXNG_PORT
from server.port
The GIT_URL
and GIT_BRANCH
in the origin:utils/brand.env file, are
read from the git VCS and the branch that is checked out when make
buildenv
command runs.
I would like to create my own brand, how should I proceed?
Create a remote branch (example.org
), checkout the remote branch (on your
local developer desktop) and in the git://searx/settings.yml file in the
server: section set base_url
. Run make buildenv
and
create a commit for your brand.
On your server you clone the branch (example.org
) into your HOME folder
~
from where you run the installation and
maintenance task.
To upgrade you brand, rebase on SearXNG’s master branch (on your local developer desktop), force push it to your remote branch. Go to your server, do a force pull and run sudo -H ./utils/searxng.sh instance update.
Node.js environment (make node.env
)¶
Node.js version 16.13.0 or higher is required to build the themes. If the requirement is not met, the build chain uses nvm (Node Version Manager) to install latest LTS of Node.js locally: there is no need to install nvm or npm on your system.
To install NVM and Node.js in once you can use make nvm.nodejs.
NVM make nvm.install nvm.status
¶
Use make nvm.status
to get the current status of your Node.js and nvm
setup.
$ LANG=C make nvm.install
INFO: install (update) NVM at ./searxng/.nvm
INFO: clone: https://github.com/nvm-sh/nvm.git
|| Cloning into './searxng/.nvm'...
INFO: checkout v0.39.4
|| HEAD is now at 8fbf8ab v0.39.4
Here is the output you will typically get on a Ubuntu 20.04 system which serves only a no longer active Release Node.js v10.19.0.
$ make nvm.status
INFO: Node.js is installed at /usr/bin/node
INFO: Node.js is version v10.19.0
WARN: minimal Node.js version is 16.13.0
INFO: npm is installed at /usr/bin/npm
INFO: npm is version 6.14.4
WARN: NVM is not installed
make nvm.nodejs
¶
Install latest Node.js LTS locally (uses nvm):
$ make nvm.nodejs
INFO: install (update) NVM at /share/searxng/.nvm
INFO: clone: https://github.com/nvm-sh/nvm.git
...
Downloading and installing node v16.13.0...
...
INFO: Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node
INFO: Node.js is version v16.13.0
INFO: npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm
INFO: npm is version 8.1.0
INFO: NVM is installed at searxng/.nvm
make run
¶
To get up a running a developer instance simply call make run
. This enables
debug option in git://searx/settings.yml, starts a ./searx/webapp.py
instance and opens the URL in your favorite WEB browser (xdg-open):
$ make run
Changes to theme’s HTML templates (jinja2) are instant. Changes to the CSS & JS sources of the theme need to be rebuild. You can do that by running:
$ make themes.all
Alternatively to themes.all
you can run live builds of the theme you are
modify (make themes.*):
$ LIVE_THEME=simple make run
make format.python
¶
Format Python source code using Black code style. See $BLACK_OPTIONS
and $BLACK_TARGETS
in git://Makefile.
Attention
We stuck at Black 22.12.0, please read comment in PR Bump black from 22.12.0 to 23.1.0
make clean
¶
Drops all intermediate files, all builds, but keep sources untouched. Before
calling make clean
stop all processes using the Python environment (make install) or
Node.js environment (make node.env).
$ make clean
CLEAN pyenv
PYENV [virtualenv] drop local/py3
CLEAN docs -- build/docs dist/docs
CLEAN themes -- locally installed npm dependencies
...
CLEAN test stuff
CLEAN common files
make docs
¶
Target docs
builds the documentation:
$ make docs
HTML ./docs --> file://
DOCS build build/docs/includes
...
The HTML pages are in dist/docs.
make docs.clean docs.live
¶
We describe the usage of the doc.*
targets in the How to contribute /
Documentation section. If you want to edit the documentation
read our live build section. If you are working in your own brand,
adjust your brand:.
make docs.gh-pages
¶
To deploy on github.io first adjust your brand:. For any further read deploy on github.io.
make test
¶
Runs a series of tests: make test.pylint, test.pep8
, test.unit
and test.robot
. You can run tests selective, e.g.:
$ make test.pep8 test.unit test.shell
TEST test.pep8 OK
...
TEST test.unit OK
...
TEST test.shell OK
make test.shell
¶
Lint shell scripts / if you have changed some bash scripting run this test before commit.
make test.pylint
¶
Pylint is known as one of the best source-code, bug and quality checker for the Python programming language. The pylint profile used in the SearXNG project is found in project’s root folder git://.pylintrc.
make search.checker.{engine name}
¶
To check all engines:
make search.checker
To check a engine with whitespace in the name like google news replace space by underline:
make search.checker.google_news
To see HTTP requests and more use SEARXNG_DEBUG:
make SEARXNG_DEBUG=1 search.checker.google_news
To filter out HTTP redirects (3xx):
make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
make themes.*
¶
The git://Makefile targets make theme.*
cover common tasks to build the
theme(s). The ./manage themes.*
command line can be used to convenient run
common theme build tasks.
themes.:
all : build all themes
live : to get live builds of CSS & JS use 'LIVE_THEME=simple make run'
simple.:
build : build simple theme
test : test simple theme
To get live builds while modifying CSS & JS use (make run):
$ LIVE_THEME=simple make run
make static.build.*
¶
The git://Makefile targets static.build.*
cover common tasks to build (a
commit of) the static files. The ./manage static.build..*
command line
can be used to convenient run common build tasks of the static files.
static.build.: [build] /static
commit : build & commit /static folder
drop : drop last commit if it was previously done by static.build.commit
restore : git restore of the /static folder (after themes.all)
./manage redis.help
¶
The ./manage redis.*
command line can be used to convenient run common Redis
tasks (Redis Developer Notes).
redis.:
devpkg : install essential packages to compile redis
build : build redis binaries at /home/user/Desktop/shared/documentation/docs/searxng/repository/dist/redis/6.2.6/amd64
install : create user (searxng-redis) and install systemd service (searxng-redis)
remove : delete user (searxng-redis) and remove service (searxng-redis)
shell : start bash interpreter from user searxng-redis
src : clone redis source code to <path> and checkput 6.2.6
useradd : create user (searxng-redis) at /usr/local/searxng-redis
userdel : delete user (searxng-redis)
addgrp : add <user> to group (searxng-redis)
rmgrp : remove <user> from group (searxng-redis)
./manage go.help
¶
The ./manage go.*
command line can be used to convenient run common go
(wiki) tasks.
go.:
ls : list golang binary archives (stable)
golang : (re-) install golang binary in user's $HOME/local folder
install : install go package in user's $HOME/go-apps folder
bash : start bash interpreter with golang environment sourced