VestaCP Reverse Proxy Problems

I did approach @Harambe with this, but he wasn’t sure either.

So, I’ve been messing around with different panels, and despite the recent exploit, I’ve settled with VestaCP. I’m using the configuration with Apache behind Nginx. I setup a reverse proxy via Apache so that you’re able to access the panel via a subdomain, rather than using an IP with a port number. I don’t use Apache anywhere near as much as I use Nginx, so I’m not as familiar with its configuration. While everything seemed successful, I’ve encountered the following issues.

When creating a domain as a user, it reports that the domain already exists. Not a huge issue, the domain still gets created etc. However, when deleting a domain, the connection to the panel closes and all of that user’s domains go offline (usually with a 520 error). No other domains are affected, just that user’s domains.

I’ve tried looking into logs to no avail. If you access the panel via the IP with the port number, the problem is gone. So I’m pretty sure that there is an issue with the reverse proxy configuration that I’ve setup.

Hopefully, someone here will have an idea haha. Sorry if it’s something simple that I’m missing!

I’d ask how to replicate your setup so I could observe it first hand but I’m still a bit reluctant to install it lol

Just spin up a fresh drople… nvm

Have you tried using nginx instead of Apache for the reverse proxy? Because in a default nginx + Apache Vesta install, every request hits nginx before apache.

I think I mentioned in the DM that I could see potential issues with disconnection because after you add/remove a domain it’s reloading the Apache and nginx conf.

I’ve also seen Vesta issues if you don’t have a mail or DNS server installed and those options are selected at creation.

Just random thoughts that come to mind as I sit on the toilet, haven’t tested anything.

Might be good to get @Falzo’s take on this as he’s got more Vesta experience than me.

hey! thx for tagging @harambe :wink:

from my pov apache (if available) is the easier way, though it’s like double proxying. but in the result you’ll end up with a ssl-secured connection to the panel without additional port, can use the inbuilt letsencrypt functionality and can even catch breakin attempts via iptables very early.
I’ve also added http auth for the API as additional layer.

@wolveix of course I can help and share my way of doing exactly that, I am just busy at this very moment, so need you to wait a few more hours. I’ll do a write up in this thread then…

1 Like

Hahaha, no doubt.

Yeah, I believe the issue still persisted. I did a lot of messing around to get PHPMyAdmin to work with the proxy, so I was switching back and forth between using Apache and using Nginx as the reverse proxy for the panel. I ended up getting it working by creating a symlink to the PMA install folder from Vesta’s own web directory.

Though I do have them both installed, I actually uncheck both of those boxes when creating a domain, so that shouldn’t be the issue. I believe that I did try once with them enabled which didn’t seem to make a difference.

Sweet, thank you so much! @Harambe did recommend tagging you and it slipped my mind when I created the thread.

sorry for the delay, here is how I go about it…

I only use debian, so my starting point would be a regular vesta-install which includes apache on stretch, I didn’t check with jessie yet, but there shouldn’t be any differences though.

first step, I’m using the main hostname/domain which should be already existant in the admin account.
(you probably can adapt all steps to another (sub)domain or even another account)
however lets call this domain my.host.name - of course that domain needs to be connected to your server properly :slight_smile:

  • log into vesta and activate ssl for that domain and let letsencrypt create the cert.

  • change to /home/admin/conf/web and open/create a file called sapache2.my.host.name.conf

  • put in the following content:

    SSLProxyEngine On
    ProxyPass /.well-known !
    ProxyPass /webmail !
    ProxyPass /roundcube !
    ProxyPass /phpmyadmin !
    ProxyPass / https://my.host.name:8083/
    ProxyPassReverse / https://my.host.name:8083/

    <Location /api>
        Authtype Basic
        AuthName "restricted"
        AuthUserFile /home/admin/conf/web/.htpasswd
        Require valid-user
    </Location>
  • save file and create .htpasswd with user and password to your likings
    htpasswd -c .htpasswd <user>

  • make sure mods proxy and proxy_http are enabled for apache
    a2enmod proxy proxy_http

  • restart apache
    service apache2 restart

if everything went well you now should be able to access vesta directly via https://my.host.name without port number and also see a nice green padlock :wink:

  • inside vesta go to the firewall-settings and change the IP for the Vesta port to 127.0.0.1 which should cut off external access to that port (verify!)

  • check api access&restriction via https://my.host.name/api - you should be greeted with a basic auth prompt and after putting your credentials get to a blank page without any errors.

that’s it for the proxy-thing, as there are no changes to any vesta-files this config should survive vesta updates easily.

if you want to block potential offenders early you can now use the blocked port 8083 as trap with this rules:

iptables -I INPUT -m recent --name vestablock --reap --update --seconds 36000 -j DROP
iptables -I INPUT -i ens18 -p tcp --dport 8083 -m recent --name vestablock --set -j LOG

careful! read first: what does this do?

the second rule listens on the ethernet interface (!!change ens18 accordingly!!), port 8083 and if there is any packet incoming it adds the IP to an internal recent-list named vestablock and logs it (default /var/log/messages, adjust to your needs).
because of the default DROP for the INPUT chain, the packet then will be dropped. also there simply is no service listening on 8083 anyway (see above)

now for the first rule, all requests coming from an IP that’s been put on that recent list will also be DROPed regardless of the port! entries are pruned only after 10 hours (adjustable) as long as they haven’t hit the first rule again, which instead would reset that counter.

again please be CAREFUL with iptables, you can easily block yourself out of your server with this, so try to understand how this works and have a fallback ready from were you can login via SSH to remove the rules if mistyped or wrong or you need to reboot.

to make the rules permanent, get some admin-basics :wink: :wink:

on a sidenote: apart from that I use elmo’s GitHub - ifaist0s/vesta-server-ssl-cert: A Linux shell script for VESTA Control Panel to assign and update SSL certificate to all services (vesta, exim, etc). script to copy the letsencrypt cert automatically over to vesta itself, so this and the mail-service have a valid one when using the hosts domain :wink:

as usual without warranties, use at your own risk and adopt to your likings. feel free to ask questions or point out mistakes.

3 Likes

Thank you so much for the detailed response! I’m using Debian Stretch as well, so no issues there. The only things that I was missing from my Apache configuration were the API location and the proxy pass for PMA etc.

I’ll go and check whether this fixes it or not, though I’m sure it will :slight_smile:

as I missed that part earlier: of course proxying with the inbuild apache/nginx carries the risk of dropping out of the panel if vesta reloads those services and something isn’t right.
yet I haven’t encountered similar problems like what you described, so I’d suggest to check if there is any orphaned config-file or something like that left, which might interfere during the reload.

1 Like

Well, I must be doing something wrong. This is all that I’m getting now:

If I change the ProxyPass and ProxyPassReverse hostname to 127.0.0.1, then I get this:

Manually going to domain:port did flag up a security exception on the SSL certificate as it was using the Vesta self-signed one, but I’ve replaced the certificate and key with the domain’s ones now which hasn’t made a difference to the ‘Service Unavailable’ error. Interestingly though, I can still access PMA and webmail.

The Apache log reads:
[proxy_http:error] [pid 771] [client IP:PORT] AH01114: HTTP: failed to make connection to backend: DOMAIN/HOSTNAME

Any ideas @Falzo?

hmm, sounds weird, I am using the same setup on multiple VMs and don’t get any errors.

I assume you did not change anything on the firewall or iptables yet?
are you using some additional firewall or piece of security that might interfere with the proxying?
are you using the main domain and admin account or is that behaviour with a different sub/user?
did you install apache separately or was it installed with the initial vesta-setup all the way?

Weird. Nope, I haven’t touched iptables or any form of firewall. I’m using a subdomain within the admin user (sub.domain.tld). And Apache was installed via the VestaCP installer. I’ll respond to your PM, and hopefully we’ll be able to get to the bottom of this :slight_smile:

I have almost no Apache experience (I’m a nginx guy), but have you tried disabling peer certificate verification?

SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyVerify optional

Source: mod_ssl - Apache HTTP Server Version 2.4

That was what I used originally (source: "I hate web interface at port 8083" No problem, I have solution for this - Vesta Control Panel - Forum). I didn’t try mixing those directly with @Falzo’s configuration though.

EDIT: Adding the following lines to Falzo’s configuration fixed the issue.

SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

However, we’re now back to square one. Adding and deleting domains as a user causes errors. It would also seem that the issue only occurs when running Apache behind Nginx.

to add to @Wolveix comment above:

for the main proxy setup it seems to depend if one uses the real hostname in the config vs an IP/localhost.
the latter will require turning of the SSL checks like mentioned above.

using the real hostname works in my setup, though I am only on KVM guest with dedicated IPs while @Wolveix box was on LXC with a routed setup/private IP. for unknown reasons the real domain didn’t work for the proxy with his setup, I suspect some routing related things we’re missing there…

the problem with creating/deleting users seems to not be related to proxying the panel at all. on a system with only apache running, there are no such issues, while having nginx<-apache<-vesta the connection drops each time you do an action that involves restarting nginx/apache.
I can’t tell yet if it is nginx dropping the connection when apache restart or the other way round. already tried to change the command issued to a reload instead of restart without much success :frowning:

will try to figure how to deal with that, that’s sadly a bummer I didn’t recognize before…

1 Like

I had similar issues with VestaCP. Effectively the install didn’t go properly. Try getting a new VM, or wiping the current VM. Install a fresh Debian 9, and install it again.

Also do not install anything out of the ordinary on the server. Just basic bread and butter.

If all else fails, I’ll try it. I don’t think that that’s the issue, as both @Falzo and myself have tried several separate installs to no avail. I setup two seperate installs on one of my other servers, and the issue still persisted. And I know that @Falzo has several VM’s running VestaCP which share the same issue(s). As long as Apache and Nginx are both there, the issue is there.

Additionally, I haven’t installed anything other than VestaCP on any of the installs. I just installed Debian 9, ran apt-get update and apt-get upgrade, then installed curl, and then ran the install script.

it’s nginx that’s the culprit here. it gets restartet on action like creating/deleting domains while apache2 only gets reloaded.

that said using nginx directly as proxy instead of apache2 most likely won’t help, as it still will be restarted instead of reloaded.

one can change that behaviour by editing line 53 in /usr/local/vesta/bin/v-restart-proxy replacing the restart with reload
at least in my tests that made it possible to add and delete domains without errors popping up :wink:

sadly any change on those files might be overwritten with the next update of vesta, so not a preferable solution anyway :confused:

edit: worth to mention that the restart-script for the webserver (could be either apache or also nginx, if stand-alone) more correctly does a reload and only if that fails also tries a restart, while the proxy restart script simply issues a restart.
so, if editing this, it probably would be better at least to replicate the behaviour done with the web-system, do a reload and if that fails try a restart…

1 Like

Fantastic, this works beautifully. Thank you so much for taking the time to figure this out today :smiley: Unfortunately, updating VestaCP does indeed overwrite this fix.

For anyone else that might be wanting to do the same thing:

Change the following code in /usr/local/vesta/bin/v-restart-proxy like so (the block starting on line 52):

From this:

# Restart system
service $PROXY_SYSTEM restart >/dev/null 2>&1
if [ $? -ne 0 ]; then
    send_email_report
    check_result $E_RESTART "$PROXY_SYSTEM restart failed"
fi

To this:

# Reload proxy system
service $PROXY_SYSTEM reload >/dev/null 2>&1
rc=$?

# Restart proxy system if reload didn't work
if [ "$rc" -ne 0 ]; then
    service $PROXY_SYSTEM restart >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        send_email_report
        check_result $E_RESTART "$PROXY_SYSTEM restart failed"
    fi
fi

Massive thank you to @Falzo for dedicating the time to figuring this out!

You only need to do this if you have Apache running behind Nginx

1 Like

you’re welcome, thanks for drawing attention to that problem :wink:

1 Like