File storage and short maintenance break

File storage and short maintenance break [completed]

michal frackowiak 1216218142|%e %b %Y, %H:%M %Z|agohover

To adapt to the growing needs of hosted wikis (and huge amount of new content being created at Wikidot.com) we will need to make a few crucial changes to our file storage later today (Wed, 16th Jul), around 10PM UTC. Until we finish the maintenance some of the newly-created wikis may experience problems with uploading files and generating of math equations or image thumbnails. Once the upgrade is complete file uploads will work for them again. So if your wiki is affected — do not worry, we will fix this today!

Also, during the maintenance, some of the uploaded files may become visible with a delay — we will be syncing files progressively and this might affect all wikis.

No outage is expected and the Wikidot.com will be available on-line all the time. Also, we will post updates about the progress.

Thanks,
The Wikidot Team

[UPDATE] we had to spend extra few hours yesterday to find an optimal and long-term solution for the upgrade and hopefully we have found a perfect solution. We will implement it today within a 5-10min break (Wikidot will be off-line) between 10.30AM and 11.30AM UTC.

[UPDATE2] During 12min break we have successfully upgraded internal storage for wiki content and made some internal changes. All file uploads should work fine now.

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalfrackowiak.com

last edited on 1216288998|%e %b %Y, %H:%M %Z|agohover by michal frackowiak + show more

Re: File storage and short maintenance break

michal frackowiak 1216291599|%e %b %Y, %H:%M %Z|agohover

OK, a few words of explanation what was happening:

All wiki-files (file uploads, generated thumbnails etc.) reside in a per-wiki directory on our server. Which means in one of directories you have subdirectories that correspond to wikis, e.g. "www/", "quake/" etc. All together more than 30 000 subdirectories. However there was one thing we did not know when we were creating Wikidot — in a Linux filesystem (precisely Ext3) there is a limit for directories within a single directory and this limit is 2¹⁵ = 32768. This limit does not apply to the number of regular files you can store. But once we hit this limit, no more per-wiki storage was created. Unfortunately this limit is hard-coded into Ext3 and there was no way to change it. Other popular filesystems (XFS, ReiserFS) have this limit much much higher.

The only reasonable solutions were:
1. change filesystem to e.g. XFS — but this would require copying and synchronizing all the files since we would need to move them somewhere first, reinitialize our primary storage and copy the files back. Waste of time and potentially dangerous.
2. change directory structure, so that wikis are stored in hashed-dirs, e.g. if a wiki is called "wiki", its dir would be 'w/wi/wiki'. But this would require changes in the Wikidot application.

So there we were yesterday and we wanted to move to XFS (because it is easier to get XFS working on CENTOS than ReiserFS). But when we made some tests it would appear that the upgrade would take quite a lot of time and possibly could lead to some inconsistency in the stored files (there would be no inconsistency if we could take Wikidot off-line, but we did not want to.)

Finally we decided to go with solution 2. We started patching the code and early in the morning we did testing. Everything worked fine. 11AM UTC we took Wikidot off-line for 12 minutes and after that everything was back to normal. A few issues with private wikis (files were not accessible) were present for a few more minutes.

Sorry for any inconvenience that the storage problem might have caused — we are working hard to keep Wikidot on-line and fully functional all the time. It is a big responsibility to host content owned by our users and we appreciate that so many people trust us with their data! Tkanks!

michal frackowiak & Gabrys

---

Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalfrackowiak.com

Still not working

Zoe Blunt 1216312003|%e %b %Y, %H:%M %Z|agohover

I was able to upload my files again today - but apparently the system does not recognize either Word or PDF documents. So what the readers see is gibberish. Are you still working on this problem? Do I need to convert all my docs to html?

Thanks.

Htm, rtf files not recognized either

Zoe Blunt 1216312723|%e %b %Y, %H:%M %Z|agohover

The good news is that I can upload and open the files. The bad news is the files are displayed as code, not as text. This is an exercise in frustration.

Htm, rtf files not recognized either

Zoe Blunt 1216312736|%e %b %Y, %H:%M %Z|agohover

The good news is that I can upload and open the files. The bad news is the files are displayed as code, not as text. This is an exercise in frustration.

Found the glitch!

Zoe Blunt 1216326835|%e %b %Y, %H:%M %Z|agohover

The "file name" field was the problem - I didn't realize it was asking for a file path with a file extension. I was entering the title, so that's why the site couldn't display the file in the right format. I uploaded the files and left the "file name" field blank.

Re: File storage and short maintenance break

wikario1 1216344736|%e %b %Y, %H:%M %Z|agohover

For me, all works fine… there is no any inconvenience. Thank you for your job.
—
Mike

last edited on 1216344768|%e %b %Y, %H:%M %Z|agohover by wikario1 + show more

Re: File storage and short maintenance break

Helmuti_pdorf 1216363064|%e %b %Y, %H:%M %Z|agohover

The "devil" in an application lies in the fact, that such a file / directory structure limit is never tested with millions of records when a development starts with some thousend data records and files. The growing data volumen brings it "on the light". Than the operating team has always a problem.

Well done with this short decision. This "tricky" solution will work with every file system. The other (XFS) would work for long time as long as the operating environment needs not to be changed. Than you detect you are bound to this file system and have the same problem as now.

Well done in a very short outtime time!

---

Service is my success.
My webtips:www.blender.org, www.zusi.de

last edited on 1216363133|%e %b %Y, %H:%M %Z|agohover by Helmuti_pdorf + show more

Re: File storage and short maintenance break

Phil Chett 1216372749|%e %b %Y, %H:%M %Z|agohover

yes !Well done chaps

---