Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you:
No adverts like this in the forums anymore.
Times and dates in your local timezone.
Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance.
Email notifications for threads you want to watch closely.
Being a part of the oldest wargaming community on the net.
If you are already a member then feel free to login now.
2024/07/04 09:05:12
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
An as-yet-unnamed UK web host has had a disaster (affecting 90+ websites), and it's caught at least 2 hobby stores - Blotz & Hasslefree.
Blotz FB Statement:
Spoiler:
We've just been given the bad news by our IT team that the server which hosts Blotz (and 90+ other websites) has undergone a massive hardware failure and the website (and associated email server) have been wiped out. This means that for the foreseeable future, the Blotz website will be unavailable. We are still open for business, so we'll be trying to get an alternative site up and running as fast as possible. In the meantime we can be reached at the unit by phone on 01285 650646 or on our home email at scbd@btopenworld.com. We apologise for any inconvenience and we'll be back online as fast as we can.
Hasslefree email:
Spoiler:
Yesterday our website went down. It's happened before, it's not usually a big deal. This one is.
I got an email from our web guy that included the word 'catastrophic', not normally a good sign. Our server is toast, and backups or recovery are currently in 'status unknown' territory.
A mysql update broke, let to a number of steps (a rollback, a reboot, a corrupted file system etc.) that got worse and the reboot corrupted the whole thing. (Just for clarity purpose, nobody's data is at risk or anything, this is more akin to a hard drive failure)
However, it gets worse! This happened the literal day that our web guy is leaving the country for a 'cruise', not even a normal holiday, a 'boat in the middle of the ocean, try working online from there' cruise!
If you need to contact us, obviously our domain name is down so please use BWARTEMIS at GMAIL dot COM.
ANNOUNCEMENT SERVER DEAD Extremely bad news from my website host, the hosting server is dead and it is quite possible that my website is going to be unrecoverable, both business email accounts could also be lost. Although an attempt to recover the site will be happening it will not be able to be carried out for at least another 12 days or so. If recovery is impossible then that means a complete rebuild will be required, the site could be out of action for several weeks or more its hard to say at this time. Needless to say this is a disaster of the highest magnitude for me and my business, I'm now looking at possibly spending days and days rebuilding ever product entry which is over 600 plus items of stock. Not to mention the best part of a month of lost sales which I can ill afford. Anyway that's the situation folks Vanguard Miniatures is closed until the website is either recovered (not sounding likely) or rebuilt.
Edit: Also Fighting 15s, Gladiator Miniatures, Early War Miniatures, Black Hat, Too Fat Lardies,
This message was edited 8 times. Last update was at 2024/07/04 11:48:59
2024/07/04 10:20:17
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree
NAVARRO wrote: IT Guys tell us how could this be prevented of happening?
It's hard to tell exactly what happened - but my guess is their backups were on the same server, so were lost when the "corrupted file system" happened. We have a physical main server (which hosts multiple virtual servers), which is copied over to a physically-separate backup server and also backed-up to tapes (which are kept off-site).
2024/07/04 10:32:58
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree
From what can be gathered from the HF mail, it seems like a broken MySQL update, that was installed by the server IT guys without any doings from the affected parties.
If you give webhosting out of house, you do this because you don't want to deal with the hassle of possible hardware and software failures. The hoster should have backups in place, seems like they didn't though, or not enough.
Look for a trustworthy webhosting provider in the first place. Make sure they have backups in place. Test those backups in a simulated environment, not just when something broke. Download a database backup once in a while.
2024/07/04 10:54:52
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
As a web professional this whole situation is bizarre. Firstly, it's pretty rare for hosting to be done on physical servers nowadays. Almost every website is cloud hosted (which are technically still physical things, but the service is distributed rather than tied to a location). Seems like these weren't.
Also, the standard procedure is to take backups regularly, usually every night, and store them offsite (again, cloud hosting makes this simple since "offsite" doesn't need to be a physical location). Worst case scenario is then to do a completely clean reinstall form bare metal if you can't recover after an update.
On which note, step 1 of any update on a server is "check we have recent backups", often followed by "screw it, I'll do a manual backup now just in case". If this is a fairly large hosting company I could see the possibility that they've borked their entire server infrastructure so the reinstall process is taking a long time due to the number of sites involved. But if it's a large hosting company I can't figure out why they're not using cloud-based hosting.
2024/07/04 11:01:14
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Haighus wrote: Sounds like a class action lawsuit in the making.
It's not a good look that the guy still went off on his holidays (assuming the Hasslefree guy is the same as the others - Vanguard's 12 days comment makes me think it is)!
2024/07/04 11:15:16
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Kid_Kyoto wrote: My heart goes out. It's hard enough to be a small manufacturer these days and this loss could be crippling.
Host the serving company has some good insurance
Reading between the lines, this doesn't seem like it's even a hosting company. It almost sounds like one guy who's sort of helping out with a bit of IT knowledge. It's the only explanation I can think of for how something like this can happen. A proper commercial hosting company doesn't stop functioning because an employee goes on holiday. If that is the case, the company's themselves bear some responsibility. Proper web hosting is really not that expensive. Half-assing it leads to consequences like these.
Of course, it may well be that the various companies were under the impression all the proper steps were being taken as far as web resiliency is concerned, in which case they wouldn't really be to blame. Tech people have an infuriating habit of obfuscating and complicating things when there's no need, often to make themselves sound more important. As someone involved in that profession, it annoys me no end to see techies confusing people who they should be trying to help.
2024/07/04 11:29:21
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
90+ websites suggests it is bigger than just a garage operation? That must be a fair amount of traffic altogether. Obviously tiny on the scale of the big players but not nothing.
ChargerIIC wrote: If algae farm paste with a little bit of your grandfather in it isn't Grimdark I don't know what is.
2024/07/04 11:40:13
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
An earth-shattering kaboom at server level at our website provider means that the Fighting 15s and Gladiator Miniatures website is down for an unknown length of time, possibly days.
As this affects the server for several sites in the wargames business, including Early War Miniatures, Black Hat and, IIRC, Too Fat Lardies, you may notice that some of the UK's small wargames businesses are similarly affected.
We and they have not gone out of business. I cannot, however, access orders in the shop while the server is down so there will be a delay in processing existing orders. I may be able to piece together orders from confirmatory emails but it will take longer than usual and I can't issue invoices. Please bear with me.
I may take a week's forced holiday. Anyone desperate
to order and who is registered with Etsy may make a custom order request through the Fighting 15s Etsy shop.
Ian, Fighting 15s
2024/07/04 11:47:34
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Fighting 15s FB post has two interesting comments on it:
Ian Marsh wrote:OK, we're out of action for 7 to 10 days. The site may need to be rebuilt from backups. Applies to everyone affected.
I think Ian is the owner of Fighting 15s. So they might have backups, but because the guy isn't physically there he can't restore them.
Jason Temple wrote:Sorry but the story you're telling about that guy being in contact with some tech team doesn't add up. No issue takes a week or more. Rich at the lardies has admitted the guy hosts the sites from a server he runs in his house and as he's not there so he can't do anything! The staggering naivety of that set up is incredible. Get a proper web host and learn how to manage a website,this isn't 1990 guys in a shed cottage industry any more.
I agree with Jason.
This message was edited 1 time. Last update was at 2024/07/04 11:52:50
2024/07/04 11:52:14
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Righty ho. Well, I am not really sure where to begin, but let's give it a go.
The TooFatLardies website is likely to be down for two weeks. And when it reappears, I'll be in the US for Historicon. Now, there's a double whammy.
To provide a bit of context, we, along with several other wargaming companies, use the services of the man I always refer to as the web elf. It's great to deal with him and he's a wargamer and consequently it makes life really easy when we release products and he understands exactly what we are doing. He's also a bloke who goes the extra mile and often works round the clock to make sure things are running as well as they can.
HOWEVER, he's just had a catastrophic failure of the server he keeps everything on and all the web sites he manages are kaputt, for now. That probably isn't what has actually happened, as there is all sorts of technical reasons this stuff kicks off, and it's all beyond me.
HOWEVER (yes, another HOWEVER!) its his 60th birthday tomorrow and his wife has booked a cruise. They leave in the morning. So, that means that until, he returns, nobody can access our web site and, as it is likely it will have to be rebuilt from scratch, which rather compounds the issue.
Now, there are moments in life which test one's patience and, frankly, this is one. But the reality is that I could jump up and down and scream and shout all I want, but it won't change a thing. What's more, the poor bloke has got to go on holiday and look like he's enjoying it. So, with that in mind, I have told him to have a great holiday and not to worry about it until he returns.
It's obviously a huge blow for us as our cash flow is now completely shut off. It's equally frustrating that the business email doesn't work and if you ordered in the last 36 hours we can't process that. Sadly, I am not here providing any solutions, just informing you of the issue we currently have. I do hope that you can all bear with us until we can get this resolved.
Definitely some clarity there.
And a reminder that business is business and friendship is friendship and it's not always a good idea to mix the two.
2024/07/04 11:56:02
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Haighus wrote:90+ websites suggests it is bigger than just a garage operation? That must be a fair amount of traffic altogether. Obviously tiny on the scale of the big players but not nothing.
Righty ho. Well, I am not really sure where to begin, but let's give it a go.
The TooFatLardies website is likely to be down for two weeks. And when it reappears, I'll be in the US for Historicon. Now, there's a double whammy.
To provide a bit of context, we, along with several other wargaming companies, use the services of the man I always refer to as the web elf. It's great to deal with him and he's a wargamer and consequently it makes life really easy when we release products and he understands exactly what we are doing. He's also a bloke who goes the extra mile and often works round the clock to make sure things are running as well as they can.
HOWEVER, he's just had a catastrophic failure of the server he keeps everything on and all the web sites he manages are kaputt, for now. That probably isn't what has actually happened, as there is all sorts of technical reasons this stuff kicks off, and it's all beyond me.
HOWEVER (yes, another HOWEVER!) its his 60th birthday tomorrow and his wife has booked a cruise. They leave in the morning. So, that means that until, he returns, nobody can access our web site and, as it is likely it will have to be rebuilt from scratch, which rather compounds the issue.
Now, there are moments in life which test one's patience and, frankly, this is one. But the reality is that I could jump up and down and scream and shout all I want, but it won't change a thing. What's more, the poor bloke has got to go on holiday and look like he's enjoying it. So, with that in mind, I have told him to have a great holiday and not to worry about it until he returns.
It's obviously a huge blow for us as our cash flow is now completely shut off. It's equally frustrating that the business email doesn't work and if you ordered in the last 36 hours we can't process that. Sadly, I am not here providing any solutions, just informing you of the issue we currently have. I do hope that you can all bear with us until we can get this resolved.
Definitely some clarity there.
And a reminder that business is business and friendship is friendship and it's not always a good idea to mix the two.
It may be more than a handful of websites, but this is classic "I know a guy" stuff. Sounds very much like a garage operation, just with a slightly bigger garage. There are so many red flags in what's being described here I don't know where to start. It's just an extension of how so many hobby-based businesses are run - i.e. not like a business at all. It's all the more annoying because nowadays this isn't difficult. If you have the knowledge to run a physical server you can easily run a cloud-based service too. Having a set-up that requires you to physically be in the same location as the server to recover from a problem like this is at least 10 years behind the current technology, if not more.
2024/07/04 12:10:10
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Sounds like the typical old workflow that worked fine a decade ago and that should have been made redundant before something nasty happened, has now died.
Its also a good picture of how most of our industry works, not much money to spend anywhere or update anything.
Wargames, Soldiers and Strategy wrote:More than a dozen wargaming companies are affected by a server outage that seems likely to last for the next two weeks. We're keeping a list on the WSS website with alternative details if you want to place an extra order in this undoubtedly challenging time: https://www.karwansaraypublishers.com/en-gb/blogs/wss-blog/server-down-websites-out
Please do let us know if we missed anyone or there are alternative contact details so we can update the list. Obviously, we hope to be able to take it down very soon!
2024/07/05 04:09:20
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
warboss wrote:Was this some wargame specific boutique webhosting service in the UK? Or are non-wargaming sites also affected?
Yeah the Too Fat Lardies post makes it pretty clear it there was a person in the community that a lot of companies turned to. It's a shame really. I just hope all these companies can survive a month or so down.
Righty ho. Well, I am not really sure where to begin, but let's give it a go.
The TooFatLardies website is likely to be down for two weeks. And when it reappears, I'll be in the US for Historicon. Now, there's a double whammy.
To provide a bit of context, we, along with several other wargaming companies, use the services of the man I always refer to as the web elf. It's great to deal with him and he's a wargamer and consequently it makes life really easy when we release products and he understands exactly what we are doing. He's also a bloke who goes the extra mile and often works round the clock to make sure things are running as well as they can.
HOWEVER, he's just had a catastrophic failure of the server he keeps everything on and all the web sites he manages are kaputt, for now. That probably isn't what has actually happened, as there is all sorts of technical reasons this stuff kicks off, and it's all beyond me.
HOWEVER (yes, another HOWEVER!) its his 60th birthday tomorrow and his wife has booked a cruise. They leave in the morning. So, that means that until, he returns, nobody can access our web site and, as it is likely it will have to be rebuilt from scratch, which rather compounds the issue.
Now, there are moments in life which test one's patience and, frankly, this is one. But the reality is that I could jump up and down and scream and shout all I want, but it won't change a thing. What's more, the poor bloke has got to go on holiday and look like he's enjoying it. So, with that in mind, I have told him to have a great holiday and not to worry about it until he returns.
It's obviously a huge blow for us as our cash flow is now completely shut off. It's equally frustrating that the business email doesn't work and if you ordered in the last 36 hours we can't process that. Sadly, I am not here providing any solutions, just informing you of the issue we currently have. I do hope that you can all bear with us until we can get this resolved.
The online list has about 20 companies now.
1st Corps / Facebook / alternative contact: firstcorpsuk@aol.com Army Bits / Facebook / alternative contact: use Facebook Messaging
Blotz Terrain / Facebook / alternative contact: scbd@btopenworld.com / 01285 650646
Crann Tara Miniatures / not on Facebook / Caliverbooks.com
Early War Miniatures / Facebook / alternative contact: Paul.Thompson11@virgin.net or 01983 854214 Mo-Fri 10.00-17.00 UK Fighting 15s / Facebook / alternative contact: news blog, Etsy shop
Fizzer Johnson's Men of Metal / Facebook
GMB Designs / no social media / alternative contact: +44 114 2217872
Hasslefree Miniatures / Facebook / alternative contact: BWARTEMIS at GMAIL dot COM
Ironclad Miniatures / Facebook / alternative contact: miniironclad@yahoo.com / +441446 790451
Lancer Miniatures / Facebook / alternative contact: kievenday@yahoo.com / +44 7914 153531
Newline Designs / Facebook / alternative contact: seanpereira30@gmail.com S&S Models / Facebook / alternative contact: +44 1278 780193
Scarab Miniatures / Facebook / alternative contact: +44 7852 561456
Space Vixens from Mars / Facebook
Shellhole Scenics / Facebook / alternative contact: shellholescenics@hotmail.com TooFatLardies / Facebook
Vanguard miniatures / Facebook
Victorious Miniatures / Facebook / alternative contact: james.dt9@outlook.com
A pretty painful situation for these small businesses that will hopefully be overcome without too much damage, but by Malal, it's quite the intriguing insight into the behind-the-scenes of UK wargaming.
2024/07/05 08:30:50
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
Yes, this is a single one-man in his shed type set up. I won't post his webpage, god knows hes probably got enough to deal with as it is, and doesn't need a bunch of dakka git gak thrown his way, but its pretty much a standard template webpage that a first year web dev student would throw together for CS degree project. £100 a year webhosting cost with no backups, hes been charging mates rates it seems, probably for some hobby cash & some extra work on the side.
Something like 100 wargaming and small business sites all gone in a flash with no backups. The fact the guy is 60 as well with no backup plan for any of these businesses, some of which have posted as recently as this year as having issues with money/sales being low. What did they expect to do when the man retires? (That said, I'm sure most UK people are counting the retirement age being pushed to 85 sometime this year so they had time to plan yet )
Multiple wargaming businesses have lost their income, multiple small local businesses have also lost their incomes, the chap behind the curtain has lost his own business and I suspect some people will step away from using him so will loose some his income stream close to retirement. He's meant to be out enjoying his cruise and not able to relax, ruining his 'holiday of a lifetime' knowing he's got to get back and clean this mess up.
GW artificially creates this fear by underproducing and pulling stuff from production without warning.
But for small companies the fear should be real. If a company makes something you like, something you're thinking of getting, get it now. That company might not be there in a year.
GW will.
2024/07/05 13:18:53
Subject: Re:Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
warboss wrote:Was this some wargame specific boutique webhosting service in the UK? Or are non-wargaming sites also affected?
Yeah the Too Fat Lardies post makes it pretty clear it there was a person in the community that a lot of companies turned to. It's a shame really. I just hope all these companies can survive a month or so down.
Thanks. I missed that with my first read through the thread.
2024/07/05 15:59:33
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
It is kind of cool how gentlemanly everyone has been, at least in public. Too Fat Laries are the only one who've given any details and they're very complimentary and don't cast blame.
I know if it were my business that lost its site, database and email, I'd be screaming bloody murder all over the internets.
2024/07/05 16:08:28
Subject: Several UK Hobby Web Stores Down - Blotz, Hasslefree, Vanguard
It's probably so much retro-tech that most folk that could be brought in to work on it won't have any idea how to use it, especially if he's been adding his own code to keep it going
)
