`-XX:MaxDirectMemorySize=512M` appears to be the solution to my Java process's greed for RAM.
Some things use Direct Buffers. Those live outside of the JVM heap limited by `-Xmx`. But the allocator for them seems to be greedy, as if it assumes it should make use of the full capacity of the pool before it starts worrying about reclaiming space from discarded buffers.
The default size of the pool is not well specified. Maybe it assumes it can have its own share of memory just as big as that granted to the jvm heap? Maybe it assumes it can have _all_ the memory, regardless of your desire to have other applications running on this computer?
In any case, adding this to the java start-up options solved the problem where the process size continued to grow, seemingly without bound, despite the on-heap memory being well behaved.
On the internet, everyone knows you're a cat — and that's totally okay.