Yesterday's security incident
Posted: 29 Feb 2012 01:45 am
Hi everyone, I'd like to explain the recent problems experienced by the site, resulting in Google labelling us as potentially malicious.
Firstly, the good news. I've manually been through the entire site to ensure that any malicious code has been removed, so the site now has a clean bill of health. I've re-submitted the site for review by Google, so the warnings for Chrome and Firefox users should disappear in the next day or so.
We're very sorry for any inconvenience this has caused, and happy to answer questions regarding the incident or any concerns you may have over your personal computer safety.
For those that are interested in what happened, I'll set out what I understand of the incident:
Earlier tonight, Jazzy and I received notifications via MSN from other forum users that Google was flagging the site as malicious (a big red scary warning screen). A very quick investigation of this confirmed that this was definitely the case, and that the site was visually functional but also trying to load malware onto visiting computers. We took the site down immediately on discovering this, and started the incident response and site restoration process.
Investigation showed that a large number of source code files across the site were 'infected' with short pieces of either php code or javascript code which were attempting to load the additional software from malicious third party websites. Whilst I haven't investigated the malware in depth, my quick, informal and most definitely not professional opinion is that anyone running a fairly modern browser and operating system with patches installed and up to date should not have gained any unwanted additional software from this.
If anyone on the forum is concerned that they may have malicious software on their system following this incident, contact me via private message or email nc@benmitchell.co.uk and I will assist.
More good news; on restoring access to the site using known good software, I checked the main NC database for signs of intrusion. There is no evidence to suggest that the database holding usernames and passwords has been compromised (please note we DO NOT hold passwords in any recoverable form anyway). The database runs on a separate server, with different login credentials to the main site or hosting account.
The malicious code present in the website source files appears to be automatically placed rather than tailored with human intervention, in which case it is unlikely that the database login credentials were taken from the website storage. In any case, access to the database server has to be made from the web server, which has been re-secured. Database credential changing for NC is non-trivial, but will be undertaken in the next few days as a further precaution.
Investigating a local copy of the affected files has not presented a single definite vector for the attack. My opinion, given the content and style of modifications made to the site, and the Google reports of the domains we were pulling malicious content from, is that the FTP credentials were compromised through an indiscriminate brute-force attack (we can't change the server name or user name, which are both linked to publicly identifiable information). If this was the attack vector used, then this could have been partially my fault - during a problematic system upgrade with the web hosts a few weeks ago, I changed the password to grant the host company access, and didn't change it after they were finished. Therefore, any compromise of the web hosts' system, or any copies of the plain-text emails with the password, could have resulted in the compromise of our account credentials.
Obviously, I have performed several actions to lower the likelihood of a similar event happening in future, including implementing a more sensible password policy, site account lockdown, and removal of legacy and maintenance applications. The site restoration has taken a bit longer than anticipated, as the latest complete offline (and thus safe) backup was not up to date, so some degree of manual cleaning and stitching together of files has been necessary to get back up and running.
However, we're back up and running within about 6 hours of first notification. We're very sorry for any inconvenience caused by the downtime or perilous warnings of peril made by Google, and if anyone does have any concerns about the security of their details or their computer following this incident, I am happy to provide more advice.
However, I have to be getting up in ... 4 and a half hours. So I'll tie up any loose ends tomorrow night (UK time), now that the site is safe, secure, up and running, and I've done the paperwork (reporting to you guys, after all, many of you are shareholders...). Goodnight!
Firstly, the good news. I've manually been through the entire site to ensure that any malicious code has been removed, so the site now has a clean bill of health. I've re-submitted the site for review by Google, so the warnings for Chrome and Firefox users should disappear in the next day or so.
We're very sorry for any inconvenience this has caused, and happy to answer questions regarding the incident or any concerns you may have over your personal computer safety.
For those that are interested in what happened, I'll set out what I understand of the incident:
Earlier tonight, Jazzy and I received notifications via MSN from other forum users that Google was flagging the site as malicious (a big red scary warning screen). A very quick investigation of this confirmed that this was definitely the case, and that the site was visually functional but also trying to load malware onto visiting computers. We took the site down immediately on discovering this, and started the incident response and site restoration process.
Investigation showed that a large number of source code files across the site were 'infected' with short pieces of either php code or javascript code which were attempting to load the additional software from malicious third party websites. Whilst I haven't investigated the malware in depth, my quick, informal and most definitely not professional opinion is that anyone running a fairly modern browser and operating system with patches installed and up to date should not have gained any unwanted additional software from this.
If anyone on the forum is concerned that they may have malicious software on their system following this incident, contact me via private message or email nc@benmitchell.co.uk and I will assist.
More good news; on restoring access to the site using known good software, I checked the main NC database for signs of intrusion. There is no evidence to suggest that the database holding usernames and passwords has been compromised (please note we DO NOT hold passwords in any recoverable form anyway). The database runs on a separate server, with different login credentials to the main site or hosting account.
The malicious code present in the website source files appears to be automatically placed rather than tailored with human intervention, in which case it is unlikely that the database login credentials were taken from the website storage. In any case, access to the database server has to be made from the web server, which has been re-secured. Database credential changing for NC is non-trivial, but will be undertaken in the next few days as a further precaution.
Investigating a local copy of the affected files has not presented a single definite vector for the attack. My opinion, given the content and style of modifications made to the site, and the Google reports of the domains we were pulling malicious content from, is that the FTP credentials were compromised through an indiscriminate brute-force attack (we can't change the server name or user name, which are both linked to publicly identifiable information). If this was the attack vector used, then this could have been partially my fault - during a problematic system upgrade with the web hosts a few weeks ago, I changed the password to grant the host company access, and didn't change it after they were finished. Therefore, any compromise of the web hosts' system, or any copies of the plain-text emails with the password, could have resulted in the compromise of our account credentials.
Obviously, I have performed several actions to lower the likelihood of a similar event happening in future, including implementing a more sensible password policy, site account lockdown, and removal of legacy and maintenance applications. The site restoration has taken a bit longer than anticipated, as the latest complete offline (and thus safe) backup was not up to date, so some degree of manual cleaning and stitching together of files has been necessary to get back up and running.
However, we're back up and running within about 6 hours of first notification. We're very sorry for any inconvenience caused by the downtime or perilous warnings of peril made by Google, and if anyone does have any concerns about the security of their details or their computer following this incident, I am happy to provide more advice.
However, I have to be getting up in ... 4 and a half hours. So I'll tie up any loose ends tomorrow night (UK time), now that the site is safe, secure, up and running, and I've done the paperwork (reporting to you guys, after all, many of you are shareholders...). Goodnight!