Search without being tracked.

Searx is a free internet metasearch engine which aggregates results from more than 70 search services. Users are neither tracked nor profiled. Additionally, searx can be used over Tor for online anonymity.

Get started with searx by using one of the public instances. If you don’t trust anyone, you can set up your own, see Installation.
Features

  • Self hosted
  • No user tracking
  • No user profiling
  • About 70 supported search engines
  • Easy integration with any search engine
  • Cookies are not used by default
  • Secure, encrypted connections (HTTPS/SSL)
  • Hosted by organisations, such as La Quadrature du Net, which promote digital rights
    Source: https://asciimoo.github.io/searx/index.html

We will host and install searx on our Raspberry Pi, the OS is debian based Raspbian.
Install all needed programs:

$ sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-babel zlib1g-dev libffi-dev libssl-dev tmux uwsgi uwsgi-plugin-python golang  libapache2-mod-uwsgi

Tmux is useful if we are suddenly disconnected from our SSH session we can resume what we were doing before. Configure system for searx, clone the repo, create a user for searx, it will be isolated from our own account. We need to setup a virtual env:

$ tmux
$ sudo su -
# cd /usr/local
# git clone https://github.com/asciimoo/searx.git
# useradd searx -d /usr/local/searx
# chown searx:searx -R /usr/local/searx
# exit
$ sudo -u searx -i
$ cd /usr/local/searx
$ virtualenv searx-ve
$ . ./searx-ve/bin/activate
$ ./manage.sh update_packages
$ sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
Edit the searx/settings.yml configuration file to suit our needs.

$ nano searx/settings.yml

general:
    debug : True # Debug mode, only for development
    instance_name : "SEARX" # displayed name

search:
    safe_search : 0 # Filter results. 0: None, 1: Moderate, 2: Strict
    autocomplete : "startpage" # Existing autocomplete backends: "dbpedia", "duckduckgo", "google", "startpage", "wikipedia" - leave b$
    language : "en-US"

server:
    port : 8888
    bind_address : "127.0.0.1" # address to listen on
    secret_key : "XXXXXXXXXXXXXXXXXXXXXx" # change this!
    base_url : "https://mysite.mooo.com:8445/" # Set custom base_url. Possible values: False or "https://your.custom.host/location/"
    image_proxy : True # Proxying image results through searx
    http_protocol_version : "1.0"  # 1.0 and 1.1 are supported

ui:
    static_path : "" # Custom static path - leave it blank if you didn't change
    templates_path : "" # Custom templates path - leave it blank if you didn't change
    default_theme : simple # simple ui theme
    default_locale : "" # Default interface locale - leave blank to detect from browser information or use codes from the 'locales' co$

# searx supports result proxification using an external service: https://github.com/asciimoo/morty
# uncomment below section if you have running morty proxy
result_proxy:
#    url : http://127.0.0.1:3001/
    url : https://mysite.mooo.com:8445/morty/
    key : MYSECRETMORTYKEY

outgoing: # communication with search engines
    request_timeout : 2 # seconds INCREASE THIS VALUE IF YOU GET TIMEOUTS OFTEN
    useragent_suffix : "" # suffix of searx_useragent, could contain informations like an email address to the administrator
    pool_connections : 100 # Number of different hosts
    pool_maxsize : 10 # Number of simultaneous requests by host

.
.
.
.
.

Run searx to test:

$ python searx/webapp.py

Check if searx is working properly

$ elinks http://localhost:8888

Let us configure uwsgi to daemonize searx and install morty to provide proxy searches.


$  sudo nano /etc/uwsgi/apps-available/searx.ini
[uwsgi]
# Who will run the code
uid = searx
gid = searx

# disable logging for privacy
disable-logging = true

# Number of workers (usually CPU count)
workers = 4

# The right granted on the created socket
chmod-socket = 666

# Plugin to use and interpretor config
single-interpreter = true
master = true
plugin = python
lazy-apps = true
enable-threads = true

# Module to import
module = searx.webapp

# Virtualenv and python path
virtualenv = /usr/local/searx/searx-ve/
pythonpath = /usr/local/searx/
chdir = /usr/local/searx/searx/

#filtron
http = 127.0.0.1:8888

Setup apache with uwsgi


$ sudo su -
# cd /etc/uwsgi/apps-enabled
# ln -s ../apps-available/searx.ini
# /etc/init.d/uwsgi restart
# a2enmod uwsgi

Create your site configuration I named mine as searx.conf
I have other service on https 443 so I used 8445 as an alternative port.


$ sudo nano /etc/apache2/sites-available/searx.conf
<VirtualHost *:8445>

ServerAdmin webmaster@mysite.mooo.com
Servername mysite.mooo.com
ServerAlias mysite.mooo.com
ServerSignature   Off
HostnameLookups   Off


SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/mysite.mooo.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mysite.mooo.com/privkey.pem

#for morty
ProxyPass "/morty/" "http://127.0.0.1:3001/"
ProxyPassReverse "/morty/" "http://127.0.0.1:3001/"
#for filtron
ProxyPass / http://127.0.0.1:4004/


CustomLog /dev/null combined
# disable log to improve privacy and save disk space
</VirtualHost>
#do not forget to enable HSTS and use mozilla's ssl config generator for apache/nginx etc.
#https://mozilla.github.io/server-side-tls/ssl-config-generator/

Restart apache2 using init.d or systemd


Set golang path
$ go env go env
GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/srv/golang"
GORACE=""
GOROOT="/usr/lib/go-1.7"
GOTOOLDIR="/usr/lib/go-1.7/pkg/tool/linux_arm"

Setup up golang


$ export GOPATH="/srv/morty"
$ export GOROOT="/usr/lib/go-1.7"

Configure morty to enable proxy searches.


$ sudo useradd --disabled-login morty
$ cd /srv
$ sudo -s
# mkdir morty
# go get github.com/asciimoo/morty
# chown -Rv morty:morty morty
# exit
$ sudo su morty
$ cd /srv/
$ export GOPATH="/srv/morty"
$ "$GOPATH/bin/morty" --help

Test morty


$ cd "$GOPATH/src/github.com/asciimoo/morty"
$ go test

Benchmark


$ cd "$GOPATH/src/github.com/asciimoo/morty"
$ go test -benchmem -bench .

Create systemd service for morty


$ sudo nano /etc/systemd/system/morty.service
[Unit]
Description=morty proxy
After=network.target

[Service]
User=morty
Group=morty
Environment="GOROOT=/usr/lib/go-1.7"
Environment="GOPATH=/srv/golang"
ExecStart=/srv/golang/bin/morty -key "randomchars" -listen "127.0.0.1:3000" -timeout "5"

[Install]
WantedBy=multi-user.target Enable and start morty $ sudo systemctl enable morty.service
$ sudo systemctl start morty.service

Updating searx


$ tmux
$ cd /usr/local/searx
$ sudo -u searx -i
$ . ./searx-ve/bin/activate
$ git stash
$ git pull origin master
$ git stash apply
$ ./manage.sh update_packages
$ sudo service uwsgi restart

FILTRON (APP FIREWALL)


$ sudo su morty
$ nano /srv/morty/bin/rules.json
[
    {
        "name": "search request",
        "filters": ["Param:q", "Path=^(/|/search)$"],
        "interval": 60,
        "limit": 10,
        "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ],
        "subrules": [
            {
                "name": "roboagent limit",
                "interval": 60,
                "limit": 10,
                "filters": ["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"],
                "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ]
            },
            {
                "name": "botlimit",
                "limit": 0,
                "stop": true,
                "filters": ["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"],
                "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ]
            },
            {
                "name": "IP limit",
                "interval": 60,
                "limit": 10,
                "stop": true,
                "aggregations": ["Header:X-Forwarded-For"],
                "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ]
            },
            {
                "name": "rss/json limit",
                "interval": 60,
                "limit": 10,
                "stop": true,
                "filters": ["Param:format=(csv|json|rss)"],
                "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ]
            },
            {
                "name": "useragent limit",
                "interval": 60,
                "limit": 10,
                "aggregations": ["Header:User-Agent"],
                "actions": [
                    {"name": "block",
                     "params": {"message": "Rate limit exceeded"}}
                ]
            }
        ]

    }
]

Download flitron


$ go get github.com/asciimoo/filtron
$ "$GOPATH/bin/filtron" --help
$ "$GOPATH/bin/filtron" -rules rules.json
2018/01/27 20:35:04 6 rules loaded from rules.json
2018/01/27 20:35:04 API listens on 127.0.0.1:4005
2018/01/27 20:35:04 Proxy listens on 127.0.0.1:4004 . . .

Test if filtron is working the localhost:

$ curl 127.0.0.1:4044

Create filtron systemd service

$ sudo nano /etc/systemd/system/morty.service

[Unit]
Description=filtron app proxy
After=network.target

[Service]
User=morty
Group=morty
Environment="GOROOT=/usr/lib/go-1.7"
Environment="GOPATH=/srv/golang"
ExecStart=/srv/golang/bin/filtron -rules /path/to/rules.json

[Install]
WantedBy=multi-user.target

Enable apache site, filtron and morty.


$ sudo a2ensite searx
$ sudo systemctl reload apache2
$ sudo systemctl enable morty.service $ sudo systemctl enable filtron.service
$ sudo systemctl start morty.service
$ sudo systemctl start filtron.service

Test if the site works.

 

Except where otherwise noted, this work is licensed under Creative Commons Attribution-ShareAlike 4.0 International License (http://creativecommons.org/licenses/by-sa/4.0/).
I hope that this post is useful to you, if you liked this post you may support me via liberapay. Thank you for your support.

Donate using Liberapay

References:
https://asciimoo.github.io/searx/dev/install/installation.html
https://github.com/asciimoo/searx/issues/1038
https://github.com/asciimoo/morty
https://asciimoo.github.io/searx/admin/filtron.html
https://github.com/asciimoo/filtron