Any Proxmox Gurus Present? noVNC Constant Reconnection + SSL Issue

Proxmox is driving me fucking nuts last night and today…

noVNC isn’t working correctly and it has something to do with the certs. I was pretty sure that everything was kosher before I deployed a LE cert in place of the self generated ones. But ever since then, I’ve had a issue where any VNC console you open will refresh and reconnect every few seconds, making anything you’re doing infeasible.

After deploying the LE certs, I get this error in my logs –

pveproxy[1775]: problem with client xxx.xxx.xxx.xxx; ssl3_read_bytes: sslv3 alert bad certificate

But if I delete my LE certs and use the self-generated certs (via pvecm updatecerts -f), I still have the VNC issue and see this error instead.

pveproxy[3585]: problem with client xxx.xxx.xxx.xxx; ssl3_read_bytes: tlsv1 alert unknown ca

Proxmox help forums are failing me. Some people just re-gen their certs and it works again, some people try a different browser and clear cache and it works again, some topics don’t have any solutions. I’m 10 seconds away from just saying fuck it and reinstalling the OS again. I have a feeling it’s something to do with Proxmox being installed over top of Debian and some underlying incompatibility at play here. But that was my only option since you can’t use RAID in the Proxmox ISO installation without ZFS.

Any ideas for me to try?

1 Like

I can’t imagine that’s the issue, I always install Proxmox over Debian. With that being said, Proxmox’ web interface can be a real pain. If there’s nothing to lose, I would try doing a fresh install.

1 Like

Just tedious to reinstall and reconfigure everything, plus I have 4 client VMs that I’ll have to remigrate. But in the end it’ll probably save me more time by just doing a reinstall rather than fighting with this any more.

1 Like

If you’re unable to find much through the forums etc. then I I would just reinstall. Given how much time you’ve already sunk into it, it might just be easier haha.

1 Like

Yeah, I’m thinking the same.

I’ve found plenty of people with the same issue on the Proxmox forum, but no solutions worked. Tried a dozen times to remove the LE certs and use Proxmox generated ones and vice-versa. I’ve confirmed the cert itself is valid. I’ve tried a plethora of different browsers w/ incognito sessions. The node itself has been rebooted. Services have been restarted a hundred times. Nothing has worked.

I guess reinstall it is! :stuck_out_tongue:

1 Like

Are you using the fullchain certificate?

2 Likes

Yep, I’ve always used this script to deploy and renew the LE cert:

#!/bin/bash
service nginx stop
certbot certonly --standalone --agree-tos --email {redacted.my.email} -d {redacted.my.domain}
rm -rf /etc/pve/local/pve-ssl.pem
rm -rf /etc/pve/local/pve-ssl.key
rm -rf /etc/pve/pve-root-ca.pem
cp /etc/letsencrypt/live/{redacted.my.domain}/fullchain.pem /etc/pve/local/pve-ssl.pem
cp /etc/letsencrypt/live/{redacted.my.domain}/chain.pem /etc/pve/pve-root-ca.pem
cp /etc/letsencrypt/live/{redacted.my.domain}/privkey.pem /etc/pve/local/pve-ssl.key
service pveproxy restart
service pvedaemon restart
service nginx start

I also tried to use the new built-in ACME cert manager as well to “order” and auto-deploy the certs. End result is always the same (one of the two errors above).

Yeah, looks like it’s incomplete.

mrowe@us1:~$ sudo openssl s_client -connect {redacted.my.domain}:8006
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = {redacted.my.domain}
verify return:1
---
Certificate chain
 0 s:/CN={redacted.my.domain}
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGXjCCBUagAwIBAgISA6HfwwMPEzdsIpH9xfWBLt/IMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTA1MjkxNTU2NTlaFw0x
OTA4MjcxNTU2NTlaMB8xHTAbBgNVBAMTFHVzMS5mcmVlbWFjaC5yb3dlLnNoMIIC
IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAyqATDtYZCTZPtKPs2A0QlUpP
C30kIH9mnHMkI6R+BCuGsb3HBn/2QeMEivPGo1okXaKrXpFO7Bdasm3GYz0kyOMm
bzJpGRwiajrl3ULP9SbzbROd0G3ENZnwYhweCJnM4uku3TZXlHW/MF/3shUlu+Eg
vmWWFGDKM5AcPpP4eBXw41r/tj2kEhHYt54m5+C70ZV56o3Pv+JdpFmtNZRb8L1Q
y1ST3xsEi1PC5w7Hqj8TiMOIXo10ZqFp6gvWR8BoGnqZBSamhnbip1KQP8rswxNU
iX37Z8gErJuKkodwIgDjJTdMwP9pCwF6hDhkZu/Kzaeg9q+eguvrhARKv4k3RFl9
rFNn1JxjxVdOLKO98o61p53oXySSTBXYQ57lNZQAMM/DRwbACqgqtKdnNmqDZqIt
k9S0upwK8gsln++a/eKU0R16+a3vvN3ku1K6RvC7fC6THyWmioSptKaLd00QgjI3
tltiMbozl6mVP2CCADX6U1BOGRLDhJYxDIEUSvJxB41GnJJknLcdoIEv5n1X/8zq
QNr6WenMRSZW5iyE9bsS9AC5DUxaXo6H+R8isLtqfQHbWXr4U8YnfLFuYuAFSq3A
n31kdMOfgRSzdJiMjLH1Owe8FyzM6f7+hN+MaZsIJUfqsbV70nXGBjG9WY/S+Gea
xvMgcp/zQPseXS1z00UCAwEAAaOCAmcwggJjMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUqXMfJSE9RdDZJiyLUxjvOunDPgMwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3
pkVl7/Oo7KEwbwYIKwYBBQUHAQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2Nz
cC5pbnQteDMubGV0c2VuY3J5cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2Vy
dC5pbnQteDMubGV0c2VuY3J5cHQub3JnLzAfBgNVHREEGDAWghR1czEuZnJlZW1h
Y2gucm93ZS5zaDBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAo
MCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQIGCisG
AQQB1nkCBAIEgfMEgfAA7gB1AHR+2oMxrTMQkSGcziVPQnDCv/1eQiAIxjc1eeYQ
e8xWAAABawSGAUUAAAQDAEYwRAIgFwAJNtzrU7QzDf9C4Zp9OXXDMG2cIy11d+0v
vkDbSt0CIHlJ1encEtAWKybc/gjq5tPap0d+hgPqgs2+qoUC18/mAHUAY/Lbzeg7
zCzPC3KEJ1drM6SNYXePvXWmOLHHaFRL2I0AAAFrBIYDTwAABAMARjBEAiBG6UTc
uUnEfK68WKV/Ce8mcaCU5pLb1A8da6ZdsZM51AIgNPRPFspkTOIB0ahpba3dN4sU
Vmg1ISGolVCk9Kgbaf8wDQYJKoZIhvcNAQELBQADggEBAFCoo5+Sw1sCh6O0ovLy
5bA+MCjdkvunE7eqxtlPo4PqggvuHs/fPgrEhhgRgZMk2LUm+OzcgjMkLCNLx1eS
xvMa5/xk/FqLLwKpdLFGn1c+sSkwsIT7nM/ct25aqSp9Sr8Tlvtz4ATW+0ntJt9D
keLQb5Bu1o7LYdv4/XeFIczXQ5tmg5ttKSmGLmoMdQ7OpARIE+dib5iU9dkVvZUq
C61Qr3a/wQ0IBu07hNjH5qPCF2UrUV8afFZoemYU5HkrpsVIlXFf30tC2VrJRLTB
NTQ0iQkfvF/GQRt+41OHyA2zT1/xg7YRwjqeKf8sE1My9VYnpOBMKV1YqJbHQsd6
q/M=
-----END CERTIFICATE-----
subject=/CN={redacted.my.domain}
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3692 bytes and written 269 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: FFB2218B6A7AC5FD02DD80313EDB64B182318FC37EF83B1987AB830E32D01588
    Session-ID-ctx:
    Master-Key: BDDB78BEC4E3017A9CC45FA7D9E976331B97CBDF6EED3B51E6CADC532456F5DF315DC52EDEB49C1097DEA81A49D2FF2F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - ba 69 24 b3 71 d1 d3 42-3e 04 1f 71 c2 92 95 8b   .i$.q..B>..q....
    0010 - f6 1b 5e 82 57 ce 2d a6-35 ed 3e cd 83 12 97 67   ..^.W.-.5.>....g
    0020 - c6 06 8b 3a b8 3e ba 89-b5 0b eb a1 49 ce 9b f2   ...:.>......I...
    0030 - 2c 28 45 c7 42 7d 42 98-1f 55 2a b3 db 70 fd c6   ,(E.B}B..U*..p..
    0040 - 01 d8 82 ee 0d bb 8d b8-86 67 d4 b9 8d b5 bd 90   .........g......
    0050 - 03 f5 7e 85 c5 87 dd 1c-5e 61 fc 05 28 3a 58 cd   ..~.....^a..(:X.
    0060 - 23 1b 2c 83 f5 c5 d6 60-07 c5 2b 18 9d 4e 64 37   #.,....`..+..Nd7
    0070 - 60 e5 a0 c2 37 f6 6a 1b-d9 ea 47 09 39 12 85 ad   `...7.j...G.9...
    0080 - 7d 4b a9 0a 32 7c 99 37-ec 69 ec bb 24 93 1d 4b   }K..2|.7.i..$..K
    0090 - 66 11 07 67 c1 2a a7 40-7e 6a 0e 77 54 c1 47 55   f..g.*.@~j.wT.GU

    Start Time: 1559156608
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Cert looks fine to me. SSL handshake succeeds.

Wait a second, I had a brainfart. Fix your damn settings. You don’t want SSLv3.

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
2 Likes

And how do I do that in Proxmox settings? Not using any reverse proxies at the moment.

I gave up on Proxmox and went raw KVM years ago because I didn’t like it. It’s a 5 year old post, but start here:

https://pve.proxmox.com/pipermail/pve-user/2014-December/164522.html

1 Like

That’s what I’m seeing in their git too, but on both my proxmox installs, /etc/default/pveproxy doesn’t exist.

Edit: and /usr/bin/pveproxy doesn’t have anything relating to SSL.

I also installed Proxmox on top of Debian.

2 Likes

Looks like they moved it to:
/usr/share/perl5/PVE/Service/pveproxy.pm

Added “sslv3 => 0” and the new “cipher_list” to the ssl block. But the issue persists :frowning: Cleared cookies/browser data, tried incognito mode, tried completely different browser. Still getting the sslv3 error.

1 Like

Did you do systemctl restart pveproxy?

1 Like

Yep. Restart both pvedaemon & pveproxy for safe measure after every config change.

1 Like

Seems I may have spoke too soon. VNC is currently stable and I’m not seeing the errors anymore. Going to monitor it for the next couple days, but fingers crossed that something I did fixed it or it just needed time to sort itself out.

3 Likes

Use nginx as proxy, do not bother with the proxmox webserver.

4 Likes

Spoken like a true Python dev.

2 Likes

The Python programmer’s creed when shit magically starts working again.

Especially for machine learning shit because… here be dragons

1 Like