The discovery and mitigation of CVE-2021-32918, code to exploit it, and some thoughts about XMPP security.

A good library exposes an API that makes it easy to do the right thing, and hard to impossible to do the incorrect thing. OpenSSL is a classic example of a bad library. On the other hand, all XMPP libraries I've seen are of very high quality. XMPP is a message passing protocol, individual messages are nicely contained in individual stanzas and sent across a TLS/TCP stream. This means XMPP library APIs universally expose a way to let you send a stanza, usually by sending objects serialized to proper XMPP stanzas, or raw XML that is always checked for correctness/completeness before sending. This is great for writing tools/clients/servers using XMPP, but this is TERRIBLE for testing security etc.

Thus was born the simplest attack ever, what if I just sent an unlimited-length stanza? I wrote up some quick java code, pointed it to my prosody, and watched the normal 60 MB memory usage climb to 4.4 GB within seconds, I quickly turned it off, hoping the GC (Garbage Collector) would catch up, but it never did. I proceeded to email developers@prosody.im about this security issue (pre-auth memory exhaustion DOS), as well as contact other server developers to ask if it might affect them. I recall contacting openfire, tigase, and m-link devs, and an ejabberd user, all of which I sent my test program and all reported no problems on their end, I never verified this personally however.

MattJ (prosody dev) was very responsive, thanked me for my report and the POC, and promised to look into it. The obvious solution was simple stanza size limits, but when Prosody 0.11.7 was released allowing for configurable stanza sizes (but not changing the default of 10 MB), I eventually got around to testing it and realized it didn't actually fix things at all. Rather than 4.4 GB in seconds, it was more like a minute, not good. After another informal talk in the prosody MUC I was convinced this wasn't fixable in Lua, at least not easily, and certainly not by me. I decided to solve the problem myself with a reverse proxy written in Rust to limit stanza sizes before they even reached prosody, and Lua's bad GC. xmpp-proxy (mirror) was created. I ran it in front of my prosody for a couple weeks and prepared to go public with the POC and encourage people to fix it by running xmpp-proxy too, until I read about the awesome stuff vaxbot was doing. The admin of yaxim.org was already going through herculean efforts putting up with the load of vaxbot, and I couldn't risk dropping a POC and some script kiddie destroying his server with it. I contacted him and he invited me to discuss it with the prosody devs once again. Long story short, MattJ ended up discovering massive regressions in the Lua garbage collector in versions 5.2 and 5.3, and, through much effort, put together a series of mitigations that adequately solved this issue on those versions, a combination of:

  1. increased GC speed
  2. smaller default stanza size limits (bonus: now matches ejabberd's defaults)
  3. bandwidth limits (without these, GC uses too much CPU)

There has been some pushback from some operators over the last two mitigations, but hopefully this additional information and the release of the POC will change their mind. There is nothing special or magical about this code, it simply sends an unlimited-size stanza, and when it gets disconnected, re-connects and sends another unlimited-size stanza again, over and over, until you kill it. Anyone could do it in a few lines of code in any language. Needless to say, you should only point this to servers you control or have permission to run it against. It should run with any version of Java 11+. I call it eatxmempp.

I think the lesson here is libraries should enforce correctness, but should have an escape hatch for doing bad things, to enable testing other things. I put in a PR for xmpp-rs to do this, and implemented sending an unbounded stream of data in sendxmpp-rs (mirror) so it can be used to test these kind of things too. (though, currently, only after authentication)

Where does xmpp-proxy go from here? Well I'm still running it in front of my servers, and am in the process of prototyping XMPP-over-QUIC with it before writing a XEP and/or RFC. More to come soon.

For questions/comments, please comment on the fediverse post here