Search without being tracked.
Searx is a free internet metasearch engine which aggregates results from more than 70 search services. Users are neither tracked nor profiled. Additionally, searx can be used over Tor for online anonymity.
Get started with searx by using one of the public instances. If you don’t trust anyone, you can set up your own, see Installation.
Features
- Self hosted
- No user tracking
- No user profiling
- About 70 supported search engines
- Easy integration with any search engine
- Cookies are not used by default
- Secure, encrypted connections (HTTPS/SSL)
- Hosted by organisations, such as La Quadrature du Net, which promote digital rights
Source: https://asciimoo.github.io/searx/index.html
We will host and install searx on our Raspberry Pi, the OS is debian based Raspbian.
Install all needed programs:
$ sudo apt-get install git build-essential libxslt-dev python-dev python-virtualenv python-babel zlib1g-dev libffi-dev libssl-dev tmux uwsgi uwsgi-plugin-python golang libapache2-mod-uwsgiTmux is useful if we are suddenly disconnected from our SSH session we can resume what we were doing before. Configure system for searx, clone the repo, create a user for searx, it will be isolated from our own account. We need to setup a virtual env:
$ tmux
$ sudo su -
# cd /usr/local
# git clone https://github.com/asciimoo/searx.git
# useradd searx -d /usr/local/searx
# chown searx:searx -R /usr/local/searx
# exit
$ sudo -u searx -i
$ cd /usr/local/searx
$ virtualenv searx-ve
$ . ./searx-ve/bin/activate
$ ./manage.sh update_packages
$ sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
Edit the searx/settings.yml configuration file to suit our needs.$ nano searx/settings.yml
general:
debug : True # Debug mode, only for development
instance_name : "SEARX" # displayed name
search:
safe_search : 0 # Filter results. 0: None, 1: Moderate, 2: Strict
autocomplete : "startpage" # Existing autocomplete backends: "dbpedia", "duckduckgo", "google", "startpage", "wikipedia" - leave b$
language : "en-US"
server:
port : 8888
bind_address : "127.0.0.1" # address to listen on
secret_key : "XXXXXXXXXXXXXXXXXXXXXx" # change this!
base_url : "https://mysite.mooo.com:8445/" # Set custom base_url. Possible values: False or "https://your.custom.host/location/"
image_proxy : True # Proxying image results through searx
http_protocol_version : "1.0" # 1.0 and 1.1 are supported
ui:
static_path : "" # Custom static path - leave it blank if you didn't change
templates_path : "" # Custom templates path - leave it blank if you didn't change
default_theme : simple # simple ui theme
default_locale : "" # Default interface locale - leave blank to detect from browser information or use codes from the 'locales' co$
# searx supports result proxification using an external service: https://github.com/asciimoo/morty
# uncomment below section if you have running morty proxy
result_proxy:
# url : http://127.0.0.1:3001/
url : https://mysite.mooo.com:8445/morty/
key : MYSECRETMORTYKEY
outgoing: # communication with search engines
request_timeout : 2 # seconds INCREASE THIS VALUE IF YOU GET TIMEOUTS OFTEN
useragent_suffix : "" # suffix of searx_useragent, could contain informations like an email address to the administrator
pool_connections : 100 # Number of different hosts
pool_maxsize : 10 # Number of simultaneous requests by host
.
.
.
.
.Run searx to test:
$ python searx/webapp.pyCheck if searx is working properly
$ elinks http://localhost:8888Let us configure uwsgi to daemonize searx and install morty to provide proxy searches.
$ sudo nano /etc/uwsgi/apps-available/searx.ini
[uwsgi]
# Who will run the code
uid = searx
gid = searx
# disable logging for privacy
disable-logging = true
# Number of workers (usually CPU count)
workers = 4
# The right granted on the created socket
chmod-socket = 666
# Plugin to use and interpretor config
single-interpreter = true
master = true
plugin = python
lazy-apps = true
enable-threads = true
# Module to import
module = searx.webapp
# Virtualenv and python path
virtualenv = /usr/local/searx/searx-ve/
pythonpath = /usr/local/searx/
chdir = /usr/local/searx/searx/
#filtron
http = 127.0.0.1:8888
Setup apache with uwsgi
$ sudo su -
# cd /etc/uwsgi/apps-enabled
# ln -s ../apps-available/searx.ini
# /etc/init.d/uwsgi restart
# a2enmod uwsgi
Create your site configuration I named mine as searx.conf
I have other service on https 443 so I used 8445 as an alternative port.
$ sudo nano /etc/apache2/sites-available/searx.conf
<VirtualHost *:8445>
ServerAdmin webmaster@mysite.mooo.com
Servername mysite.mooo.com
ServerAlias mysite.mooo.com
ServerSignature Off
HostnameLookups Off
SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/mysite.mooo.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mysite.mooo.com/privkey.pem
#for morty
ProxyPass "/morty/" "http://127.0.0.1:3001/"
ProxyPassReverse "/morty/" "http://127.0.0.1:3001/"
#for filtron
ProxyPass / http://127.0.0.1:4004/
CustomLog /dev/null combined
# disable log to improve privacy and save disk space
</VirtualHost>
#do not forget to enable HSTS and use mozilla's ssl config generator for apache/nginx etc.
#https://mozilla.github.io/server-side-tls/ssl-config-generator/
Restart apache2 using init.d or systemd
Set golang path
$ go env go env
GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/srv/golang"
GORACE=""
GOROOT="/usr/lib/go-1.7"
GOTOOLDIR="/usr/lib/go-1.7/pkg/tool/linux_arm"Setup up golang
$ export GOPATH="/srv/morty"
$ export GOROOT="/usr/lib/go-1.7"Configure morty to enable proxy searches.
$ sudo useradd --disabled-login morty
$ cd /srv
$ sudo -s
# mkdir morty
# go get github.com/asciimoo/morty
# chown -Rv morty:morty morty
# exit
$ sudo su morty
$ cd /srv/
$ export GOPATH="/srv/morty"
$ "$GOPATH/bin/morty" --help
Test morty
$ cd "$GOPATH/src/github.com/asciimoo/morty"
$ go testBenchmark
$ cd "$GOPATH/src/github.com/asciimoo/morty"
$ go test -benchmem -bench .Create systemd service for morty
$ sudo nano /etc/systemd/system/morty.service [Unit] Description=morty proxy After=network.target [Service] User=morty Group=morty Environment="GOROOT=/usr/lib/go-1.7" Environment="GOPATH=/srv/golang" ExecStart=/srv/golang/bin/morty -key "randomchars" -listen "127.0.0.1:3000" -timeout "5" [Install] WantedBy=multi-user.targetEnable and start morty$ sudo systemctl enable morty.service $ sudo systemctl start morty.service
Updating searx
$ tmux
$ cd /usr/local/searx
$ sudo -u searx -i
$ . ./searx-ve/bin/activate
$ git stash
$ git pull origin master
$ git stash apply
$ ./manage.sh update_packages
$ sudo service uwsgi restartFILTRON (APP FIREWALL)
$ sudo su morty
$ nano /srv/morty/bin/rules.json
[
{
"name": "search request",
"filters": ["Param:q", "Path=^(/|/search)$"],
"interval": 60,
"limit": 10,
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
],
"subrules": [
{
"name": "roboagent limit",
"interval": 60,
"limit": 10,
"filters": ["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "botlimit",
"limit": 0,
"stop": true,
"filters": ["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "IP limit",
"interval": 60,
"limit": 10,
"stop": true,
"aggregations": ["Header:X-Forwarded-For"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "rss/json limit",
"interval": 60,
"limit": 10,
"stop": true,
"filters": ["Param:format=(csv|json|rss)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "useragent limit",
"interval": 60,
"limit": 10,
"aggregations": ["Header:User-Agent"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
}
]
}
]Download flitron
$ go get github.com/asciimoo/filtron
$ "$GOPATH/bin/filtron" --help
$ "$GOPATH/bin/filtron" -rules rules.json
2018/01/27 20:35:04 6 rules loaded from rules.json
2018/01/27 20:35:04 API listens on 127.0.0.1:4005
2018/01/27 20:35:04 Proxy listens on 127.0.0.1:4004 . . .Test if filtron is working the localhost:
$ curl 127.0.0.1:4044
Create filtron systemd service
$ sudo nano /etc/systemd/system/morty.service[Unit] Description=filtron app proxy After=network.target [Service] User=morty Group=morty Environment="GOROOT=/usr/lib/go-1.7" Environment="GOPATH=/srv/golang" ExecStart=/srv/golang/bin/filtron -rules /path/to/rules.json [Install] WantedBy=multi-user.target
Enable apache site, filtron and morty.
$ sudo a2ensite searx $ sudo systemctl reload apache2 $ sudosystemctl enable morty.service $ sudo systemctl enable filtron.service $ sudo systemctl start morty.service $ sudo systemctl start filtron.service
Test if the site works.
Except where otherwise noted, this work is licensed under Creative Commons Attribution-ShareAlike 4.0 International License (http://creativecommons.org/licenses/by-sa/4.0/).
I hope that this post is useful to you, if you liked this post you may support me via liberapay. Thank you for your support.
References:
https://asciimoo.github.io/searx/dev/install/installation.html
https://github.com/asciimoo/searx/issues/1038
https://github.com/asciimoo/morty
https://asciimoo.github.io/searx/admin/filtron.html
https://github.com/asciimoo/filtron
