I just had an idea for keeping track of users who have been reported but we want to just keep an eye on them rather than blocking or silencing:
Have an account specifically for following problem-users, and give all mods access to it. Name it something that lets people know we're keeping an eye on them. If they take action against it, we can reciprocate.
On the internet, everyone knows you're a cat — and that's totally okay.