Security Breach - 7 June 2018 - Overview

Network Network Notices
1

Summary of events

On 7 June I was made aware of attempted logins on PTAWUG router from CTWUG IP. This was in the evening.

Once I heard this I checked our router logs and saw that this IP had broadly attacked routers on CTWUG between 12h05 and 12h25 earlier on the 7th.

I then did the following:

  1. I firstly null routed (blackhole) the whole highsite range from which this originated by broadcasting null routes on OSPF.
  2. Upon further investigation of the IP I null routed only the particular client’s IP range. This was after establishing the routes on the high site for the client.
  3. I obtained logs from our radius server and verified that no radius accounts were breached. These logs are also indicative of the breadth of the attack.
  4. I obtained logs of all routers running WMS where the particular IP was referenced. This was 661 lines. Basically 661 login attempts, actual logins or logouts.
  5. I contacted the high site admin and eventually the wugger (a client of the highsite) to find out what the situation was.
  6. It took some time to get a response from the wugger as I only obtained an email address for them. I obtained logins to the wugger’s routers both on the wug and on his internet.
  7. By the time I was able to login to the routers they had been rebooted (a couple of days later) and no logs from the time were present.
  8. I did however review the configurations on both the WUG router as well as the internet facing router. Both were Mikrotik.

Speed

The first log entry on WMS router was at 12:06:13 and the last entry was at 12:23:44. Just over a 1000 seconds. It generated 661 logins, failed logins or logouts. That’s 37.7 actions per minute. This was an automated attack using a program or script. This may be far higher as the attacker probably also attacked non WMS routers.

Usernames

The hacker attempted to login using admin account multiple times. It also attempted to login using CTWUG user accounts including wug users as well as ctwug user. This is concerning. The presence of wug user names made us concerned that this is a targeted attack, but the speed seems to oppose this. I confirmed that it did breach at least some usernames where passwords had been set.

Ports targeted

All logins were targeted on the web portal of these routers (port 80) and not ftp, winbox or other ports.

Setup of the node

The node from which the attack originated had internet as well as wug access. Both these were run using Mikrotik devices. The Mikrotik device on the internet:

  • Had bruteforce firewalls protections on 2 ports.
  • But no protection on other ports including the web port and winbox ports.
  • Did not have a default drop all on the input chain of the router from the internet.

It was therefore quite insecure. Unfortunately this device was rebooted before I gained access.

Previous attacks

I have seen two similar incidents before where a user’s router was compromised from the internet. This was then used as a launch pad for a further attack on the CTWUG. These attacks however were limited to brute-force user / password attacks and used generic usernames. Not wug specific usernames.

Mikrotik Vulnerabilities

The following seems likely to be relevant:

Both the above would compromise user accounts on a router and make them available to the attacker.

Likely cause

  1. We believe that it’s unlikely that the user completed the attack. They did not appear to have the skills involved as well as the poor quality of the firewall seems to indicate poor security awareness.
  2. Furthermore the user had poor firewalls on their internet facing Mikrotik. It would have been easy for an attacker to exploit the above vulnerabilities or simply bruteforce the password/username on the router.
  3. The attacker could then have used either vulnerabilities on the CTWUG. In particular it seems likely they used the above web port vulnerability since they targeted web port for logins to compromise other routers.
  4. It would then re-use discovered usernames and passwords to attack further routers.
  5. Further poor security on CTWUG worsened the impact:
  • Default logins (admin with no password)
  • Out of date routers running old versions of RouterOS
  • Re-use of user accounts and passwords across routers.
  • No wug wide protection for these rapid attacks.
  1. It is unclear why the attack stopped after such a short time. We were lucky.

Implications

An outside entity probably has the user accounts and passwords for a number of wuggers available to them. These accounts are compromised and may be attacked in future if the same attacker gains access to CTWUG or, if these get published on the dark web or simply on a website somewhere.

Actions Taken

  • The committee has given the wugger involved a warning about security and their node remains null routed until they implement improved firewalls on their internet side. I will assist if needs be.
  • I have implemented wug wide bruteforce protection that will automatically null route mass logins of this nature. This covers many of our routers (but not all) .
  • I have also improved notifications of such attacks so that we are aware sooner when it happens.
  • We have updated minimum Router OS version to avoid these vulnerabilities. This will be enforced on all OSPF rotuers with immediate effect. I also propose that we update this more frequently in future and especially so if RouterOS security vulnerabilities are made public.
  • I will ensure that I ask any wugger which is the source of such an attack try and retain the relevant logs if possible.
  • @dizzle will review routers and find routers with admin accounts without password.

Further possible actions

  • Consider what to do with the compromised accounts and routers.
  • Consider further basic firewall as part of WMS
  • Consider what to do with non OSPF or non WMS routers. It might be good idea to requuire all routers be updated as per OSPF router requirements. It is a bit more difficult to enforce.
  • Re-enable NMS and also consider adding brute-force protections and alerts there.
  • Wuggers should review their logs on all their routers and establish if they were attacked at this point.
  • Wuggers should also keep their RouterOS updated and ensure secure firewalls on internet facing routers.
  • Wuggers should practice good password security by using a tool such as KeePass or similar to store unique passwords per services/router/website.
10 Likes
2

Hi there,

On our side we had this happen once a few days after the vulnerability notice was made public with a user who had WUG and internet on the same router for his home usage. I had noticed rather quickly through an automated telegram bot that a login failed to my routerboards (all my routerboards, including home - so everything in a /24, including the users in the second /24).

After finding the person who had this, I had disabled the routing to that routerboard and left it to that person to upgrade his ROS AND FIRMWARE. He was then allowed to activate it as I disabled it on his side - since our wug users here optionally (and by default steps) have RADIUS enabled, (and the admin password was open :facepalm:).

We further discussed it (I recommend the other guys fill the whole thing there in), but the basic conclusion ended up:

Edit: This action was not taken alone, nor was I the only person who upgraded things, I was part of a bigger “team” of go2guys, admins and highsite owners who all helped, who all helped write the rules.

Further: I can provide the telegram code to anybody if they want, but it’s a simple filter out each recent log, if failed login then /get … keep-result=no with the telegram API.

4 Likes
3

Thanks for your ideas. Building upon them and integrating on what we have planned already:

Notification

We have channel #security on chat which notifies of repeat logins. Will work on the notification messages a bit but it’s already operational and catching people attempting 6 or more logins.

We are enabling our NMS instance again and this will scan all router logs and do also notify (and probably blackhole) repeat logins.

Penetration Testing

In reality our main risk is coming from privately managed internet routers. However we are planning to scan for default logins and such to catch this.

It might also be possible to scan for outdated RouterOS versions that are not centrally managed.

Rules

We need to update our rules in this regard. Mainly by requiring more recent router versions as well as perhaps more leeway to network admins to upgrade routers for example. I’d also suggest automated WMS upgrades be part of WMS rules.

2 Likes