Monday, February 18, 2008

MySQL as a filesystem

For some time now, I have been pondering about a Storage engine for MySQL that interfaces with flat files. Yes, I see a few needs that can solve for me.
Today, browsing around, I found Ben Martin's article on Using MySQL as a filesystem. The articles uses MySQLfs to get the desired results. Not 100% what I was looking for but still a good read. Ben writes:
With MySQLfs you can store a filesystem inside a MySQL relational database. MySQLfs breaks up the byte content of files that you store in its filesystem into tuples in the database, which allows you to store large files in the filesystem without requiring the database to support extremely large BLOB fields. With MySQLfs you can throw a filesystem into a MySQL database and take advantage of whatever database backup, clustering, and replication setup you have to protect your MySQLfs filesystem.

8 comments:

FallenPegasus said...

I've been thinking of a storage engine for flat files as well, and have started writing one.

More specifically, it's a storage engine that interfaces to the directory system, e.g, each row would be it's own file, each table would be a directory, etc.

I've been calling it "pig" because it eats inodes like corn.

It turns out that it might actually be useful, on systems like Isilon and similar products...

Nilesh Jethwa said...

Hi,
I have been working on an MySQL DBA Dashboard. Would like your opinion to improve and make it useful

Your opinion is highly regarded
Regards
Nilesh
MySQL Dashboards

Michael S. Moody said...

So I'm just curious what the design would look like for your MySQL filesystem?

This could be something I could definitely use, with a master for writes, and all webnodes as slaves.

Ideas about drawing up a design document?


Michael S. Moody
Sr. Systems Engineer
Global Systems Consulting http://www.GlobalSystemsConsulting.com

PaulM said...

Sorry if this sounds flippant but MySQL started life (before MyISAM) as a relational engine on top of the Berkeley DB. The storage engine still exists as BDB.

BDB are flat files. As long as you don't break the license you can still use BDB (or Sleepycat) for free and download the source. This is even though Oracle now owns the codebase.

Have Fun
Paul

Stephan Wehner said...

There is a project hosted on google code, http://code.google.com/p/mysql-filesystem-engine/ -- it seems that
each record is a file.

One could also arrange to have each column-value held in its own file, and setting up a directory for each row, which is a subdirectory of the directory set up for a table. Kind of nice to handle text-columns in this manner, since they just stay text-files.
But text-files usually come naturally with their location in a directory, and that is kind of lost with this setup.

This two different choices suggest to me the idea for such an engine isn't natural enough.

There is already the CSV engine, by the way.

Stephan

Anonymous said...

It is wonderful article from Martin. Very interesting and informative. It real interest for web Developers plus administrators.

Allan "Goldfish" Clark said...

I've taken up the next release of MySQLfs, intending to release 0.4.0 in two weeks. If you've used it, and found a bug we don't know about, hit it on Sourceforge and file a bugreport. Thanks!

Allan "Goldfish" Clark said...

I just saw my comment approved, and re-read the other comments... FallenPegasus, "eats inodes like corn" is hilarious! :)