XMPP Connectivity & Security
moparisthebest
July 13, 2023
Discovering XMPP
- Early 2012, had first baby, needed job for health insurance
- Needed to communicate with wife throughout the day about baby
- Nothing
to hide, but also no one else’s business, what color was the baby’s
poop this morning, how many times did she go, how much is she eating and
how often etc etc
- SMS: can only type on phone, tried a few through-computer things
that were terrible, not private
- Google Talk: in web browser, terrible from phone, not private
- at some point, Google announced it was cancelling Talk, and the
Snowden revelations came out. I then resolved to host everything myself
from now on so I only had to change anything when I wanted, not at
anyone else’s whim, somehow, I found XMPP and that I could easily host
my own server with Prosody
- Only had 100mb mobile data per month, but work allowed connecting
through WiFi, problem solved until…
- attack of the evil stupid firewall…
- suddenly my XMPP clients (gajim on desktop, Conversations on phone)
would no longer connect, why?
Connectivity
- Work installed a new firewall “for security” -.-
- investigation showed it did not MITM TLS, so it was still
private to connect over
- it allowed plain HTTP over port 80, or TLS over port 443, but
only TLS over port 443, listening for plain XMPP (with
STARTTLS) over 443 was also refused
- it was sniffing for a TLS header and only allowing connections
preceded by that and only that
- What good is a self-hosted, private chat system if I couldn’t
connect?
Connectivity (cont.)
- Realized I could just pipe most of my stuff over TLS on
port 443, smtps/imaps/sieve (email), https, ssh(over tls)
- this required multiplexing, nothing complete existed, hence a patch
for sslh to multiplex on TLS SNI
- but XMPP used STARTTLS, which is rejected flat out
- why can’t XMPP connect over TLS directly like these other
protocols?
- I submitted a patch for
Conversations to implement looking up a different set of SRV records
and connecting directly with TLS
- happily ignorant of the XSF or XEPs
- Daniel (Conversations developer) said he wouldn’t merge until I
submitted and had a XEP accepted
- the rest is history XEP-0368: SRV records
for XMPP over TLS
- aside: I had never written a standard or participated in a standards
organization before, but everyone was really helpful and great at
pretending not to mind when I asked stupid questions. The XSF is
definitely an example of a great organization when it comes to welcoming
newcomers in my opinion.
Security Background
- I really got into programming in 2005, writing cheats for a game
called RuneScape
- So from the beginning, I’ve always been looking for things I could
exploit
- Always try to think like an attacker
- We’ll go through some recent XMPP exploits, not in order
Security - httppppppppppp-upload: full drive exploit
- XEP-0363: HTTP
File Upload is a popular way to share files in XMPP, and really the
only one used for MUCs, multiple clients, or sending/recieving when the
other party may be offline
- clients with an HTTP uploaded link sent to them will generally do 2
things:
- HEAD request to determine size of file (returned in Content-Length:
header)
- If user clicks download, or if under some set size, GET request to
download the file
- if your client automatically downloads files from strangers or in
public MUCs, GET A NEW CLIENT! (or at least complain to the dev)
- An idea struck, what if an HTTP server lied about the length in a
HEAD request, and then sent an unlimited-length file in the GET request
- write a little rust code, call it evil-http
(clever right?)
- for the HEAD request return that it’s a jpg that’s 11kb
- for the GET request return that’s it’s a jpg with
Transfer-Encoding: chunked, send the picture first, and
then just send unlimited \0’s after, which is a valid jpg
by the way
- turns out, nearly every XMPP client will just fill up your hard
drive :’(
Security - httppppppppppp-upload: full drive exploit (cont.)
- Another idea, many HTTP libraries store headers in a
HashMap<String, String> or similar, what if I just
never stopped sending (randomly named) headers? would they just run out
of memory and crash or?
- add another mode to evil-http
- turns out some libraries have sane limits already and abort the
connection, but others like the version of libsoup Dino was using at the
time, have no mitigations at all, oops… (sure hope it’s fixed now)
Security - httppppppppppp-upload: full drive exploit (cont.)
- So I put both of these up at public (but secret) URLs so client devs
can test against them (please don’t use these for evil, only testing
your own stuff)
- https://evpic.moparisthe.best/test.jpg - evil-http in unlimited GET
mode, uses a lot of my bandwidth
- https://gzevpic.moparisthe.best/test.jpg - evil-http in unlimited
GET mode, but nginx is compressing it with gzip, uses almost no
bandwidth but will still fill up your drive
- https://evheader.moparisthe.best/test.jpg - evil-http in unlimited
header mode
- The hard part: coordinated disclosure
- Impossible in practice, all of these are open source, once one
commits a fix the cat is out of the bag, not to mention Google and
Apple’s store review lead time etc etc
- Tell who I know, they tell who they know, then I send a message to
jdev inviting devs to message me for more details
- After awhile, when hopefully everyone has updated, publish a blog
post
Security - httppppppppppp-upload: full drive exploit (cont.)
- Key takeaways:
- You have no guarantee headers will end, limit these to something
sane, maybe 16k of headers or something
- You have no guarantee data returned by the HEAD request will match
that returned by the GET request.
- Beware transfer-chunked encoding.
- Always have a way to cancel or a sane limit.
- Make sure your “sane limit” is after decompression is applied, not
before (ie zip-bomb )
- Beware old attacks like slow loris so require a minimum speed
- For XMPP client devs specifically, this advice applies to
downloading HTTP Uploaded files, POSH files, host-meta files, and
anything else you might grab over HTTP.
- Honestly just beware any stream that may be unlimited.
- full
writeup
Security - eatxmempp: CVE-2021-32918
- Lightning intro to XMPP, if you
are experienced with this feel free to check your phone or something :)
- XMPP is basically individual chunks of XML sent over a stream,
called stanzas
- an example might be
<message to='romeo@example.net'><body>Wherefore art thou?</body></message>
- the X stands for eXtensible so adding an element like this is also
valid:
<message to='romeo@example.net'><somethingelse attributeone='1' attribute2='aoeuaoaeu'/><body>Wherefore art thou?</body></message>
- You might have noticed that nothing up front says how long a stanza
should be
- hmm common theme? (you caught me!)
- where do we stop reading from the stream?
- well, where the stanza ends of course
- so just read until
</message> right?
- well, this is a single valid stanza:
<message to='romeo@example.net'><body><![CDATA[</message>]]></body></message>
- most clients and servers use an XML parsing library to detect stanza
boundaries
Security - eatxmempp: CVE-2021-32918 (cont.)
- So if you are throwing bytes from the network that you need to keep
into a generic XML library, and you don’t know how many might get sent,
what happens when…
- the stanza never ends?!?!?!?
- coded up what I later named eatxmempp, it simply
connects, opens a stream, and starts sending unlimited XML prior to
auth, when it is disconnected, it reconnects and does it again, simple
in virtually any language
- my prosody server hosts about 10 accounts and I’m joined to
many busy MUCs and IRC channels, it had been up for a few weeks
and was using 61mb of ram, pretty nice..
- I pointed eatxmempp at my server, and within seconds it was using
4.4gb of ram! I rapidly spammed ctrl+c, killing eatxmempp
- I expected the garbage collector to kick in and right things
(recover memory), but it never did
Security - eatxmempp: CVE-2021-32918 (cont.)
- Now what? I now have the power to take down services (usually
prosody, but because of how the linux oom killer works, I could kill
random other processes too!) on any server running prosody on the
internet in a few seconds!
Security - eatxmempp: CVE-2021-32918 (cont.)
- Contact prosody devs, let them know, send them POC
- MattJ responded immediately, thanked me, and promised to look into
it
- Too lazy to set up other servers to test, so contacted developers of
openfire, tigase, and m-link, and a friend that ran an ejabberd, sent
them my POC with an explanation, and kindly asked them to test it. They
all reported back that they had no problems with this, which honestly
surprised me especially in the case of the Java servers (openfire &
tigase)
Security - eatxmempp: CVE-2021-32918 (cont.)
- Prosody
0.11.7 released with configurable stanza limits!
- eventually I test this and instead of 4.4gb of memory in seconds…
- it takes about a minute :’(
- briefly talk in prosody MUC about memory use (likely cause was
fragmentation, which is a problem with the GC), and come to the
conclusion this wasn’t fixable in Lua, at least not easily, and
certainly not by me.
- An idea forms, what if I write a reverse proxy to sit in front of
prosody and limit stanza sizes before anything even reaches the
bad GC of Lua?
- Rust is an obvious choice for a language, fast, no GC, and a joy to
work in
- start development on xmpp-proxy
- I ran xmpp-proxy in front of my own server for a few weeks for
testing, and planned to publish a blog post explaining the issue and
recommending prosody users also run xmpp-proxy as a mitigation
- but something came up, check full writeup for details
- long story short MattJ did some more tests and realized that prosody
was fine on it’s original Lua target, 5.1, and on the latest, 5.4, but
there were major regressions in GC performance in 5.2 and 5.3
which is what most distros were shipping at the time
Security - eatxmempp: CVE-2021-32918 (cont.)
- Prosody
0.11.9 released with fix for eatxmempp! It was a herculean effort by
the prosody team and a combination was the key
- increased GC speed
- smaller default stanza size limits (bonus: now matches ejabberd’s
defaults)
- bandwidth limits (without these, GC uses too much CPU)
- now I’m left with this cool (that’s just like, your opinion, man)
xmpp-proxy, what can I do with it?
- Key takeaways:
- beware unlimited streams
- if you depend on a GC, check for regressions
- full
writeup
Security - XEP-0156 _xmppconnect is vulnerable to MITM
- While adding WebSocket support to xmpp-proxy, I’m implementing
discovery via XEP-0156: Discovering
Alternative XMPP Connection Methods
- Realize when you find a websocket URL via the DNS TXT record,
there’s really no correct way to validate the TLS certificate that is
both secure and works
- say you are trying to connect to example.org, and the TXT record
tells you to connect to wss://evil.com/xmpp
- all WebSocket libraries out of the box will check the certificate
for evil.com, but this is meaningless, because absent DNSSEC, an
attacker could have sent us this URL, you can’t trust that it is
legitimate
- it would be secure if checked for example.org (which is how
certificate validation for normal XMPP via SRV records work), but no
HTTPS servers actually support this
- note this issue is identical for the BOSH connection method in
addition to WebSocket
- Being a issue with the spec, I decided the best place to start would
be an email to the standards@ list explaining it
- the authors of the spec responded quickly and agreed with my proposed changes
- removed TXT record and added Security Considerations
Security - XEP-0156 _xmppconnect is vulnerable to MITM (cont.)
- sadly I have to admit this is the one time I found centralization on
(proprietary) github helpful, I used their search for
_xmppconnect, tried to filter out duplicates (forks), and
created issues on each one
- https://github.com/processone/docs.ejabberd.im/issues/113
- https://github.com/JustOxlamon/TwoRatChat/issues/2
- https://github.com/poVoq/converse_wp/issues/2
- https://github.com/BombusMod/BombusMod/issues/130
- https://github.com/hesa2020/Twitch-To-League-by-Hesa/issues/1
- https://github.com/xmppjs/xmpp.js/issues/933
- https://github.com/tigase/tigase-http-api/issues/8
- https://github.com/tigase/tigase-extras/issues/3
- https://dev.gajim.org/gajim/python-nbxmpp/-/issues/124
- pidgin devs I talked to in their channel, and they created CVE-2022-26491
Security - XEP-0156 _xmppconnect is vulnerable to MITM (cont.)
- Key takeaways:
- DNS is not secure (absent DNSSEC, which sadly isn’t very widely
deployed)
- TLS certificate validation is useless if you are checking the wrong
thing!
- is there any “global” code search for all of us hosting our own code
on our own FOSS ? :’(
- full
writeup
xmpp-proxy high-level today
- bytes in via one transport, bytes out via another (plain TCP, plain
WebSocket, STARTTLS, Direct TLS, TLS WebSocket, QUIC)
- Rust crate (library) for all things needed for making or receiving
an incoming or outgoing XMPP connection
- SRV, host-meta, POSH lookups
- proper SRV fallback behavior, trying each in the right order until
one is found with a proper certificate that speaks XMPP
- STARTTLS, Direct TLS, QUIC, WebSocket protocol support
- splitting the stream on stanza boundaries without an XML parser
- any c2s/s2s/protocol support can be compiled out
xmpp-proxy high-level today
(cont.)
- program using said library that:
- listens for outgoing plaintext TCP XMPP / WebSocket XMPP on
localhost, doing lookups and connecting to the proper server
securely
- listens for incoming encrypted XMPP and connecting to a plaintext
TCP XMPP port locally
- c2s and s2s are enabled in all directions (and s2s supports/requires
SASL EXTERNAL in both directions)
- all this allows clients/servers to be developed that can only speak
plain XMPP/WebSocket and still have full featured connection support
over the best in class TLS library (rustls)
splitting the stream
- so the main goal of xmpp-proxy in the beginning was to limit stanza
size, which required splitting the stream on stanza boundaries
- we don’t want to allocate, because that could lead to the same
fragmented memory problem we are trying to avoid
- eventually land on allocating a fixed sized buffer per connection,
and implement a state machine that knows just enough about XMPP’s
restricted subset of XML to know when a stanza ends
- let’s look at some code…
testing all this
- early in development I found out with this kind of stuff unit tests
don’t help that much, really need to test against real servers in the
wild
- for example ejabberd won’t accept
<features xmlns="http://etherx.jabber.org/streams">
only <stream:features>
- also things like DNS lookups, certificate validation etc etc really
need real world tests
- podman to the rescue!
- I wrote an integration test framework (in bash) to start up a podman
network per folder, where each folder is a scenario
- each network runs a bind9 for DNS, nginx for http(s), up to 2
prosodys, 2 ejabberds, and 3 xmpp-proxys, and then runs scansion to send
messages between accounts on the various servers
- this gives me comprehensive real-world testing of all possible
combinations of protocols and servers, and real confidence when
refactoring or making changes
testing all this
- feel free to steal this for your XMPP testing use
xmpp-bench-proxy
- someone came to me with a problem, we need to benchmark our XMPP
server, but it’s only accessible over WebSocket
- rtb and tsung are good mature XMPP benchmark tools written in
erlang, but neither support XMPP over WebSocket
- about 100 lines of rust later, depending on xmpp-proxy, we have a
very fast Plain TCP -> WebSocket XMPP converter
- harcoded to accept connections on 1 port and connect to another,
skips TLS certificate validation
- I was actually nervous that rust+tokio might not be able to keep up
with erlang, but…
- testing against my own prosody running on a very old Xeon box
- rtb was able to get to 10k connections against prosody using normal
STARTTLS
- rtb was able to get to 10k connections through xmpp-bench-proxy to
prosody TLS WebSocket, with xmpp-bench-proxy running on the same machine
as rtb, using about 20% CPU of a laptop
- success!
- xmpp-bench-proxy
Converse-Tauri
- Converse.js is a very nice web XMPP client which supports connecting
via WebSocket or BOSH
- Tauri is a very nice way to run web applications on the desktop
without the bloat or security problems that come with electron, and it’s
written in Rust!
- run the command line tool to create a new Tauri app, write a few
lines of code to start xmpp-proxy and let converse know the port, and
now we have a full featured desktop XMPP client, the first I know of
that supports QUIC !
- let’s look at some code…
Java/Android bindings
- Written quickly using the JNI crate
- works with desktop openjdk fine
- hacked into Cheogram android, installed on phone, segfaults
- need to install and configure an android VM and debugger but what a
PITA
- therefore I haven’t pushed up any code yet
- let’s look at some code…
What’s next for xmpp-proxy?
- Documentation is sorely lacking, this is my current priority
- Supporting unix sockets both incoming and outgoing as well as TCP
sockets are currently supported
- Native Tor support, connecting to/from and properly validating
.onion addresses
- either via socks5 proxy (which would be good by itself)
- or via Arti (new official Rust Tor library) that is nearing
maturity
- DNSSEC + DANE support
(not so) secret project
- Goal: enable anyone to run their own XMPP server, even if they don’t
have a domain name or know what NAT or port forwarding is
- even if behind CGNAT
- also support this if they do own their own domain (which is
the ideal)
- 1000 foot view
- XMPP server generates a TLS key (random of course)
- follows established algorithm to turn the private key material into
a domain
- thsnthoantheunthoaue.example.org
- self-signs a certificate for this domain
- creates a QUIC connection to my server (or any ran by volunteers,
due to SRV records these will be geographically diverse and preferably
diverse in actual operators)
- server will verify the certificate and private key matches, and
then, based on SNI (for Direct TLS, QUIC, and WebSocket) or domain name
from STARTTLS, start routing new incoming connections back over a new
QUIC stream to this server
- server also will accept valid certificates like “bob.com” etc and
route traffic to them, this enables ownership of your own domain
(not so) secret project (cont.)
- ideally this enables someone with app/front-end experience to do
cool things like make a “Snikket Server for Android TV” app etc, if no
one does it, I’ll probably bundle it on top of Snikket first
- got pretty far writing code, then got distracted by life (as per
usual)
Questions?
- please join xmpp:xmpp-proxy@code.moparisthebest.com?join to discuss
anything remotely related to xmpp-proxy or this presentation
- FYI: all my code hosted at
https://github.com/moparisthebest/project-name is also hosted at
https://code.moparisthebest.com/moparisthebest/project-name , if you
don’t want to use github to contribute, great! join the above MUC, email
me patches, or ask for an account on code.moparisthebest.com to
contribute
- how did I make these awesome slides?
https://www.moparisthebest.com/slides/slides.sh
- thanks!