On 27 December at 05:06 UTC, we received an email alert from Hostinger, indicating an outage. At this time the only information provided was a list of scripts that were running at the time of the alert. We failed to realise we were infected, and that one of the processes was running a crawler inside an infected container. We lifted the restrictions on the VPS, assuming the restriction was related to some testing and setup work being done on that server at the time.

On 27th December at approximately 16:00 UTC, we discovered that all StudioCMS sites were offline. At 21:00 UTC, Adam and I jumped in a call to determine what was happening. We discovered an unexpected running process xmrig, a CPU/GPU miner for Monero. After terminating the process, we stopped the docker service, assuming the entrypoint was likely a docker container, and confirmed nothing else was running.

At 21:16 UTC, Louis joined the call, and began identifying secrets that should be rotated. In parallel, we attempted to track down the container that was compromised. Not long later, we identified several services that used Next.js, and theorized that a container may have been vulnerable to the recently published React2Shell RSC vulnerability.

After some exploration, we located the malware. It consisted of a crawler, designed to identify machines vulnerable to the RSC exploit and deliver a payload. We archived what was found, but quickly moved on to determine vulnerable container and ensure we could get back online without further incident.

At approximately 23:13 UTC, we concluded that Umami was the culprit. And we certainly hadn’t been the only ones affected. There were several types of malware hitting Umami instances. Some had, like us, been infected with the XMRig miner and a crawler.

At 23:40 UTC we reset the VPS, and over the next few hours, tried to lift the resource restriction placed on our server by Hostinger’s systems. Unfortunately we were unsuccessful due to the fact restrictions can only be lifted once every 7 days. However, we were able to begin setting up a proper fallback page in case of future outages, with the help of our network sponsor.

On 3 January, we successfully lifted the restrictions, and redeployed our infrastructure to bring back our sites.

Delayed compromise, delayed discovery

The initial compromise likely happened on or before 18 December according to log files we found in the payload, and Hostinger likely correctly flagging a file as malicious the next day. Because of other work being done at the time, it was assumed that Hostinger had flagged the server due to the testing of a Discord bot or our git mirror auto-clone setup.

React2Shell was publicly disclosed at the end of November, but the analytics scripts on the sites were out of date. These were updated on 17 December to point to the new instance. Making our previously hidden Umami instance, publicly discoverable.

The malware

The payload was designed to run on both Windows and Linux machines, and included a script to avoid detection in task manager. It shipped it’s own version of Node for Windows, along with a series of .bat files for different variants.

XMRig was run in the background, as the crawler visited various seed websites leading to a high quantity of smaller sites. It prioritized sites with uncommon TLDs like .dev, .tech, or .sh. While it did ignore some domains, it didn’t exclude their subdomains, leading it to scan, report, and attempt the exploit on a lot of Medium sites. There were no other vulnerabilities exploited other than React2Shell.

The code was completely plaintext and heavily commented. It mentions that it is a Node port of a Python version of the same malware. Given the code-style, I’m fairly certain this was produced, at least in part, by an LLM.

Limiting future impact

Since the incident, we’ve split third-party services and Discord bots to a second VPS. This means the main VPS now only hosts our containers for important services, like our main websites. This, and other steps we’ve taken, should help prevent (and/or limit the impact of) future incidents.

With this incident handled, and a new year ahead, we set our sights on upcoming and future releases. See you there!



Back to Blog