MySQL Consulting and NoSQL Consulting: MySQL DBA

MySQL for Big Data

2011-06-01T17:45:00.001-04:00

An excerpt from article on mysql for big data published in Dow Jones Venture Wire by Scott Denne.

There is one possible solution to the problem that doesn't include companies having to buy new software tools or even an all-new database: With the right expertise, MySQL can be engineered to handle almost any data-intensive application. The only problem is that there's a shortage of people who have the expertise to make it work.

"There's a big time gap until we, as an industry, think we have data under control," said Frank Mashraqi, chief technology officer at MyLawsuit.com and former database chief at Fotolog Inc., a photo blogging site. "The roadmap to getting that expertise is very difficult and time doesn't allow for it."

Presenting "Real-Life Use Cases From Data Administration Hell" at LAMySQL

2011-02-07T19:12:00.004-05:00

If you're in the Los Angeles area on Feb 15, come hear my talk at LAMySQL inspired by learnings from real-life experiences. In addition to hearing a very unique and interesting talk, you can win an AppleTV thanks to awesome folks at @NoodleYard.

Real-Life Use Cases From Data Administration Hell

Data is the most valuable asset of an organization because it's irreplaceable.

Yet, we hear about f**k ups related to data administration every day by startups and organizations of all sizes. Sometimes it's no one's fault. Sometimes it's the fault of a drunk friend who shouldn't have been [wherever he was] at the first place.

Yet, at other times, the disaster could have been prevented. Sometimes, these f**k ups are caused by bad design. Sometimes, it's a bad query that made it into the production branch. Sometimes, it's a human error that ruins the day.

Ever had a bad query slip through QA? Or a configuration option that you thought would help the situation? Sometimes, the resulting disaster could have been prevented if those operating had simply followed the rules. Sometimes it's the lack of presence of rules that leads to a disaster. Sometimes, the "acts of prevention" worsen the impact of the disaster. Sometimes it's over confidence of those administering data.

Imagine deletion of a wrong record or from a wrong server. Or not treating the only SAN as a SPOF. Sometimes, the f**k up has been happening for years, yet no one realized or fixed it.

Sometimes, the f**k up is created intentionally. By focusing on things other than operational and capacity requirements. Sometimes, a small error threatens the very existence of a company.

At least once of a $100M company.

These f**k ups happen everywhere. At organizations of all sizes.

In this talk, Frank Mashraqi will explore real-life inspired, breath-taking (anonymized) use cases that created data administration hell for an organization. He will also explore how, if at all, these f**k ups could have been avoided.

This session presents an opportunity to learn from the real-life costly data administration mistakes of others and what strategies can help you with not getting caught off guard.

Bio

With more than a decade of scalability, disaster recovery and engineering management experience under his belt, Frank specializes in building and scaling NoSQL and SQL based platforms for graph processing and big data deployments using low concurrencies.

He is an expert in audience acquisition through organic search engine optimization and audience monetization through cutting edge technologies as re-targeting, social targeting and influencer targeting.

His past experience includes co-founding a graph processing company that applies advanced sociological theories to online advertising, scaling Fotolog to help it become the 13th most visited site on the Internet, and advising companies like Betaworks, Bitly, TwitterFeed, Chartbeat and ShermansTravel.

He holds a BBA in Accounting and a BS in Computer Information Systems.

PS: Many thanks to JoeDevon for inviting me to speak

PS: I'll be driving from the Bay Area so if anyone is interested in riding with me from SF to LA and back, let me know :)

SEO / SEM SWOT

2010-12-08T06:04:00.001-05:00

Since most of my friends here have asked me questions on SEO. I thought i'd share this presentation that I posted on SlideShare. This was a pre-engagement presentation. It contains over a 100 slides and covers many dimensions of SEO. Enjoy!

SEO / SEM SWOT Presentation

View more presentations from frankmashraqi.

Disaster @ Tumblr

2010-12-06T16:35:00.002-05:00

Tumblr has been down for more than 12 hours due to an issue with their database cluster. Here is the comment I left on GigaOm.com

This is the freshest lesson for entrepreneurs and startups:
- Learn to value your data
- Implement a high availability plan
- Plan a disaster recovery strategy

“Tumblr likely has the resources to recover…”

I really hope that holds out true but remember, data is the only irreplaceable asset of an organization. Once it’s gone, it’s gone.

When I was handling the disaster at Fotolog (massive database corruption when our SAN crashed), I couldn’t find any company or consulting firm ready to handle the situation and help with data recovery. It was a miracle that I came across the concept of DUDE (Data Unloading by Data Extraction) and started writing InnoDB data recovery programs in sheer desperation. In case of Fotolog, we had all basic infrastructure in place for redundancy and high availability. The component that caused the disaster was the one we relied most upon: “the financial grade strength SAN.”

The point I am trying to make is having access to cash in the bank + large userbase + really smart engineers doesn’t provide any guarantee that your data will be safe in case of a disaster.

Times like these can be of incredible stress on those handling the situation. I feel for folks at Tumblr and hoping for a speedy recovery.

Good luck Tumblr guys! You’re in my thoughts.

Frank

Sequoia backs MongoDB with $6.5M investment

2010-12-03T04:33:00.003-05:00

Some exciting news coming from 10Gen, the company behind MongoDB. It announced today that Seqouia is investing $6.5M in it's high performance, document-oriented (BSON), key-value based NoSQL solution that supports automatic sharding and dynamic queries. Foursquare, Disqus, Etsy, Sourceforge, eVite, EventBrite and New York Times are all users of 10Gen. The features this young NoSQL solution offers is truly impressive. See MongoDB page on my Big Data Low Latency site for quick review of MongoDB.

I had the opportunity to meet with Roelof Botha few months ago as Sequoia was looking to invest in the NoSQL space and was evaluating both hardware and software solutions to solving big data challenge. Since then I was eager to hear which of the many startups in the NoSQL space will receive Sequoia's blessing. Now we know :)

Video: Netflix's migration to AWS cloud

2010-12-01T15:41:00.001-05:00

Found this video regarding Netflix's migration to Amazon's AWS cloud very informative. Enjoy!

Cloud Migration Whitepapers

2010-12-01T13:26:00.001-05:00

Amazon's AWS team has published a series of whitepapers covering various scenarios for migrating into AWS cloud infrastructure. Links to these whitepapers are provided below for your convenience:

- Migrating applications to the AWS cloud
- Migrating web application
- Migrating batch processing applications
- Migrating backend processing pipelines

Big Data: Freedom or Something Else?

2010-12-01T09:35:00.003-05:00

Googling around, I came across Bradford Cross' article, Big Data Is Less About Size, And More About Freedom. Bradford writes, " The scale of data and computations is an important issue, but the data age is less about the raw size of your data, and more about the cool stuff you can do with it."

Even though the article makes some good points, I'm not sure I can agree with Bradford's point of view here. As an architect, when I think in terms of Big Data, the ability to do "cool stuff" is probably the last thing that crosses my mind. Big Data, to me, is about ensuring constant response time as the data grows in size without sacrificing functionality.

What do you think Big Data is about? Is it merely about being able to do 'cool stuff' with your data? Is it about ensuring constant access/response times? Or is it about something else? I'm eager to hear your thoughts.

SQL and NoSQL

2010-11-30T14:23:00.003-05:00

Alaric Snell-Pym discusses why choose between SQL and NoSQL? Why can't you use both in your infrastructure?

"NoSQL engines abandon SQL for the chance to have more flexible data models and softer semantics for update operations - but they also abandon it because it’s a lot of work to implement. And, creating a new database from scratch, they’re keen on solving the interesting hard problems (such as replicated data storage), rather than following the well-trodden path of writing SQL parsers and query planners, with a few decades of catching up with the competition ahead of them."

Hate the dirty recruitment tactics

2010-11-30T14:16:00.003-05:00

I hate it when recruiters reach out to you with a message indicating that they are looking for 'key positions' and when you follow up, the tone changes to "we're just looking for engineers." This happens all the time and the latest company to play this recruitment tactic is LinkedIn. Guys, can't you decide whether you are looking to fill a 'key position' or just an engineering position before reaching out to candidates? I can see that mentioning 'key position' will get a candidate's attention but this is just a low level tactic.

Lean Startups and Scalability

2010-11-30T14:00:00.006-05:00

I wrote this as a reply to Does Lean Startup Methodology Apply to Consumer Startups?" However, due to comment length restrictions on that blog, I am posting my comment here and welcome your thoughts.

"An enterprise will pilot products and iterate with a vendor: Let's run a 6 month consulting engagement/pilot to evaluate if this new database solves the problem."

Only an enterprise where there is a major disconnect between management and engineering will opt for this path. In enterprises where needed data I/O patterns are understood, taking such path may spell disaster.

The primary problem with the 'lean startup' methodology that I see is that it blindly preaches entrepreneurs to close their eyes, cut corners and just get the product to market without fully understanding the future scalability needs. Scalability doesn't has to be sacrificed in order to build a lean startup, except in those cases where there is no architect on board.

Many entrepreneurs think of taking route of frameworks in early days only to find out the haunting effects as growth happens.

Case studies exist where massive infrastructures with low latencies have been built without blowing budgets and need for re-architecting infrastructures. The concept of lean startups is great, but incomplete (especially as it is being preached).

When the right architects and team is on-board, building the right way also becomes the lean way.

<update> Let's not forget that data is the most valuable asset of an organization and every migration a great opportunity for a screwup. Do you really want to migrate it around from database to database as you sit with your fingers crossed hoping that the latest vendor will solve your problem? Or does it make more sense to invest in an experienced architect and then make decisions rather than shooting in the dark? I'll let you decide the rest.

Probably the worst way to deal with a stuck query...

2010-11-22T17:24:00.005-05:00

is to disable a customer's account for more than 24 hours without any warning whatsoever. This happened to one of my accounts and I'm beyond furious at the database and network administrators of HostGator.com. Seriously, guys, I don't know of a more unprofessional way of dealing with a stuck query.

MySQL at Facebook

2010-10-27T17:00:00.002-04:00

Mark your calendars for Nov 2 as Mark Callaghan and Facebook's MySQL team will be talking about how MySQL is used at Facebook.

Hadoop Elastic MapReduce by AWS

2009-04-02T10:04:00.001-04:00

Amazon today launched a beta of it's Elastic MapReduce (hosted hadoop). This is exciting and just in time for my upcoming, Hadoop and MySQL: Friends with benefits, session at the MySQL Conference & Expo.

I can't wait to try it out!

Community One East - What will Sun announce?

2009-03-18T00:53:00.001-04:00

I will be attending Community One tomorrow and on Thursday at Marriott Marquis Hotel, New York, NY. I am especially looking forward to the announcements tomorrow which sound very interesting :)

The first day is a free event featuring:

* Cloud Platforms – Development and deployment in the cloud.
* Social and Collaborative Platforms – Social networks and Web 2.0 trends.
* RIAs and Scripting – Rich Internet Applications, scripting and tools.
* Web Platforms – Dynamic languages, databases, and Web servers.
* Server-side Platforms – SOA, tools, application servers, and databases.
* Mobile Development – Mobile platforms, devices, tools and application development.
* Operating Systems and Infrastructure – Operating systems and virtualization.
* Free and Open – Open-source projects, business models, and trends.

The second day of the event is focused on Deep Dives with two half-day sessions on MySQL and two full-day sessions on Java and Web development. I will be attending the session, "Using Java EE and SOA to Architect and Design Robust Enterprise Applications."

Following the conference, I will be a panelist at a Cloud Computing Seminar at Microsoft office in NY. It's going to be a long but exciting day!

It will be great to catch up with old and new friends at the event.

Cloud Computing - Executive Seminar

2009-03-02T22:26:00.004-05:00

Tomorrow, I'll be attending the Executive Seminar on Cloud Computing at NASDAQ MarketSite (NY). Speakers include Dr. Werner Vogels and Mårten Mickos (ex-CEO of MySQL). Big thanks to Amazon and RightScale who were able to accommmodate my RSVP even when the registration had formally closed.

I hope to be able to catch up with Mårten Mickos during the event. In case I do succeed in catching up, is there any question you want me to ask him? You can email me or post a comment.

It's funny that the event site still shows Mårten's title as "SVP of Sun Microsystems’ Database Group."

FriendFeed uses MySQL to store "Schema-less" data

2009-03-01T02:56:00.004-05:00

Came across an interesting post by Bret (co-founder of FriendFeed) about how FriendFeed uses MySQL to store "schema-less" data. According to the post, they weren't having issues with scaling existing features but rather they were experiencing pain when trying to add features.

Now the way they are using MySQL is interesting and bizarre at the same time. At a very high level, it seems their approach is to use a RDBMS as if it is a column-oriented database. Of course, it makes me wonder why not just use a column-oriented database? I need to read the post again in the morning (too tired right now so just gave it a quick glance).

I am very interested in hearing thoughts from my peers at Planet MySQL regarding this approach. They seem to have gone great lengths to go this route. What issues and benefits you see of this approach and whether you ever see yourself taking this route? I, for one, am not entirely convinced of this approach and whether it can really scale down the road. Also, if it was someone other than Friend feed going down that route, I might have actually lost my tempered and yelled :)

Side note: Friendfeed is growing fast, and it would have been cool if Bret was speaking at one of the three upcoming MySQL events in April.

OpenSQL Camp Starts Tomorrow!

2008-11-13T23:40:00.003-05:00

So the good news is that the inaugural OpenSQL Camp is going to be an awesome event with mouth watering sessions by noted experts. The bad news (for me) is that I won't be attending it, which makes me sad. I cannot leave my town because my wife can go into labor anytime now.

The session list looks great! Congratulations and thanks to Baron, Sheeri, Ronald, all the sponsors and contributors for organizing the first OpenSQL Camp.

I will be watching PlanetMySQL closely for juicy blog posts. Hopefully, the one and only Sheeri is taking her camcorder!

Stack Overflow: Q&A Site

2008-11-10T10:34:00.002-05:00

Today I discovered Stack Overflow, a collaborative site that focuses on technical Questions. You can ask questions related to any language, apparently without having to register. The site is currently in beta. There are also a few MySQL questions that are currently unanswered.

Scalability As A Functional Or Non Functional Requirement

2008-11-09T00:34:00.003-05:00

I am currently tasked with writing Software Requirements Specification (SRS) document for a project. Effective sharding (based on specific criterion) and Scalability are key requirements of the project.

Scalability is traditionally classified as a non-functional requirement. My question to the community is that if scalability is crucial to a project, would it still be classified as a non-functional requirement? Are their cases when scalability requirements would be best classified as functional requirements?

Open Source Pony Tail

2008-10-31T13:51:00.002-04:00

Sorry for not updating this blog regularly. My wife's due date is soon so I've busy.

Anyway, I wanted to share this very funny interview with Jonathan Schwartz (puppet):

Startup Scalability Strategies @ Startonomics

2008-10-02T03:08:00.002-04:00

Tomorrow morning I am presenting a session, Startup Scalability Strategies, at Startonomics, a conference being organized by Dave McClure and Deal Maker Media. The sessions will be streamed live using UStream. Check the Startonomics website at http://startonomics.com for more details.

Also check out my guest blog post titled How Important Is Scalability written for Startonomics blog.

Notes from Structure 08, Velocity and Graphing Social Patterns East

2008-08-24T13:37:00.006-04:00

I attended several events in June of this year including Graphing Social Patterns East, Velocity and Structure 08. At each of these events, I tried to take some notes and posted them to my personal blog. I received a few pings from readers of this blog to point them to a list of these posts. It took some time but here is the list of my notes. In some cases, I have linked directly to the presentation files.

High-performance Ajax Applications: Julien Lecomte (Yahoo!) talked about how to effectively use AJAX in your applications without compromising performance.
Slideshare: High performance Ajax Applications

Stress, Load and Performance Testing in Quality Assurance: Excellent tips on stress and performance testing by Goranka Bjedov of Google.

Actionable Logging for Smoother Operation and Faster Recovery: Mandi Walls from AOL talked about logging in general including actionable logging, why it's important, logging goals, log file management, things to avoid in logs and more. (Presentation slides)

Clouds are No Substitute for Competence: Presented by Javier Soltero of Hyperic

Energy Efficient Operations: Some Challenges and Opportunities: Luiz Barroso from Google presented this very interesting and informative session about making operations energy efficient.

Innovation That Drives Opportunity for the Web Infrastructure: John Folwer (Sun Microsystems) was the speaker at this talk about Web 2.0 architectures. (Presentation slides)

Importance of Operations and Performance: Artur Bergman of Wikia talked about lessons learned while running 7000 wikis.

Jiffy: Real World Performance Measurement: In this session Scott Ruthfield talks about Jiffy, an open source tool for performance measurement and instrumentation. (Presentation slides)

KITE: Keynote Internet Testing Environment Launch: KITE was one of the interesting products launched at Velocity. KITE allows you to test from desktop to the Internet cloud. At the time of launch KITE was free. Don't know the current pricing model. (Presentation slides)

Harnessing Explosive Growth: Infrastructure Strategies and Tactics: Panelists including Sandy Jen, Akash Garg, Jeremiah Robinson, Jonathan Heiliger and James Barrese discussed strategies and tactics for handling explosive growth.

The Race to the Next Database: Overclocking and Analytics Augment Your Data Layer: At Structure 08, panelists on this session included Mayank Bawa (Aster Data Systems), Doug Judd (Zvents), Luke Lonergan (Greenplum), Damian Black (SQLstream), Dave Schrader (Teradata) and Scott Wiener (Cloud9Analytics). Each panelist provided insight into the ground breaking work their company is doing in solving data processing and handling BI challenges faced by consumers today.

Working the Clouds: NextGen Infrastructure for New Entrepreneurs: This panel on cloud computing featured panelists including Geva Perry (GigaSpaces), Jason Hoffman (Joyent), Tony Lucas (XCalibre), Lew Moorman (Rackspace), Christophe Bisciglia (Google / AppEngine) and Joe Weinman (AT&T). Christophe got grilled heavily by other panelists but he handled it pretty darn well.

Werner Vogels: Keynote at Structure 08: Dr. Vogels keynote was one of the highlights of Structure 08. He presented case study of Animoto and talked about the 70/30 switch among other things.

The Platform Revolution: A Look into disruptive technologies: Jonathan Yarmis AMR Research (VP of Disruptive Technologies) talked about technology trends, social networking, mobility, mobile , cloud computing, stream computing, business models, user 2.0 and the new enterprise reality.

Green Data Centers: Bill Coleman (Cassatt Corporation) presented this session. Bill is known for being responsible for the B in BEA. (Presentation slides)

Creating Bebo Applications: Bebo is now part of OpenSocial and this presentation presented at Graphing Social Patterns talks about how to create applications for Bebo.

Open Social and Google App Engine: Patrick Chanezon (API Evangelist) and Paul McDonald (Product Manager for Google App Engine) presented a technical overview of OpenSocial and Google App Engine at Graphing Social Patterns East.

OpenSocial: Open for Business: In this session, panelists were Patrick Chanezon (Google), Paul Lindner (hi5), Max Newbould (MySpace) and Sachin Rekhi (imeem). (Also see)

Viral Marketing and Advertising Strategies for social networks: One of the best sessions at the Graphing Social Patterns conference presented by Kevin Barenblat and Jeff Ragovin. (Also see)

Mobile Social Networks: A Comparison: Benjamin Joffe's excellent eye opening session for anyone interested in using mobile platform for creating social networking solutions.

Top 5 Things that fail and win on social networks: Dave McClure, chair of Graphing Social Patterns, presented this concise but every effective presentation on what fails and what wins on social networks.

Geek Metrics: Using App Analytics to Drive Distribution, Engagement, & Monetization: Dave McClure (500 Hats) moderated this panel which included Hiten Shah (CrazyEgg / KISSmetrics), Ian Swanson (Sometrics, Inc.), Albert Lai (Kontagent) and Roy Pereira (Refresh)

Social + Mobile = Sociable (Social Networks for SMS, IM & Mobile Devices: Panelists in this session included Benjamin Joffe, Ben Keighran, Gregory Cypes, Craig Dalton and Chris Butler.

Widget Strategies & Social Platforms: Hooman Radfar, CEO of Clearspring Technologies discussed the new role of widgets and how to go about creating them.

Facebook Business and Marketing Solutions: Kent Schoen talked about how to use Facebook for business and marketing.

Developing and Promoting Social Network Applications: Rules of thumb: What does FACEBOOK means when it comes to creating and promoting applications for social networks?

Social Networks for Business and Marketing Managers: Ro Choy of Rock You! gave an overview of social networks for business managers:

Scaling MySQL - powered Web Sites by Sharding and Replication: Slides from Peter Zaitsev's session at Velocity. (Presentation slides)

Capacity Management: John Allspaw's signature presentation on capacity management. John also has a book coming out on this topic. (Presentation slides)

Structure 08 on demand: Watch the Structure 08 conference on demand at Mogulus.

LinkedIn Communication Architecture: Slides about LinkedIn's platform built in Java. (Presentation slides)

SOX Compliance: A presentation by Skye Rogers. I missed this presentation but then caught up with Skye at dinner. I wish Skye would have received more time to discuss SOX Compliance. (Presentation slides)

There were several sessions I didn't get to go to which is a sad thing. You may want to check the conference websites directly (linked at top of this post) to see if there are presentation slides available. Also, if you took notes at these sessions, please feel free to drop the links as comments to this post.

S3 suffers major outage

2008-07-20T23:15:00.004-04:00

“Funny how Amazon doesn't use S3 to store any assets for amazon.com”tweet by @gruber

Amazon's S3 suffered a major outage today knocking many websites offline. S3 outage started at approximately 12:00 PM EST and the last time I checked at 11:11PM EST, Smugmug, a popular photo hosting site that extensively uses S3, was still down.

- S3 down for more than 7 hours
- S3 outage, 7 hours and counting
- S3 down again
- Amazon failure downs Web 2.0 sites
- Amazon's S3 experiencing outage

Web Developer / Graphic Designer Job Openings

2008-07-20T21:44:00.004-04:00

Currently, there are several great opportunities with exciting companies available in the New York area. If you're a rock star Java/PHP/Ruby developer or a pixel-obsessed designer, contact me at your earliest convenience.

Web Developer:

Give Real is a well-funded startup in the midst of an exciting period of growth and success. Our technology uses a patent pending platform that combines the ubiquity of credit card transactions and the power of social networks to create a new gifting experience.

Our primary platform is Rails, but there are programming challenges that range from SOAP APIs to Facebook application development. We are searching for full-time developers with expertise and broad experience in:

* Ruby on Rails (we also use rSpec, Starling, Memcache)
* MySQL
* xHTML & CSS, and comfort with Javascript
* Team development with tools like Git & Trac

In addition, we are also interested in candidates who have:

* Expert Javascript skills
* Java & SOAP experience
* Experience scaling with Rails, or any other web platform
* Comprehensive Linux knowledge
* UI and graphic design backgrounds

We are willing to pay top-notch developers very competitively (plus the possibility of options) to join our team and help write code that will be used by hundreds of thousands of users within a few months. We are ideally located in downtown Manhattan less than a minute walk from the BDFV and NRQW lines

Also, if you know someone who may be a good fit for us (developer or graphic designer), we are offering a $1000 referral reward for anyone we hire.

Please contact us at jobs@givereal.com

Graphic Design:

Give Real is a well-funded startup in the midst of an exciting period of growth and success. Our technology uses a patent pending platform that combines the ubiquity of credit card transactions and the power of social networks to create a new gifting experience.

We're searching for full-time designers with experience in:

* Design for advertisements
* Design for consumer focused websites & applications
* xHTML & CSS coding
* HTML & design for emails
* Working on top of an MVC or template system (we use Rails)

In addition, we are also interested in candidates who have:

* Team development with tools like Git & Trac
* Comfort with Javascript programming
* Rails programming experience

We are willing to pay top-notch developers very competitively (plus the possibility of options) to join our team and help design the look and feel of a service that will be used by hundreds of thousands of users with a few months.

Also, if you know someone who may be a good fit for us (RoR developer or graphic designer), we are offering a $1000 referral reward for anyone we hire.

Please contact us at jobs@givereal.com

Please Help Save Ivan (Needs a Bone Marrow Transplant)

2008-07-13T23:18:00.007-04:00

Please help save Ivan, son of Andrii Nikitin (MySQL Support Engineer), who needs a bone marrow transplant. Andrii's message is below:

"My family got bad news - doctors said allogenic bone marrow transplantation is the only chance for my son Ivan.

"8 months of heavy and expensive immune suppression brought some positive results so we hoped that recovering is just question of time.

"Ivan is very brave boy - not every human meets so much suffering during whole life, like Ivan already met in his 2,5 years. But long road is still in front of us to get full recover - we are ready to come it through.

"Ukrainian clinics have no technical possibility to do such complex operation, so we need 150-250K EUR for Israel or European or US clinic. The final decision will be made considering amount we able to find. Perhaps my family is able to get ~60% of that by selling the flat where parents leave and some other goods, but we still require external help."

-- Andrii Nikitin, MySQL Engineer

Please remember, every little bit will help the family pay for Ivan's operation! Be as generous as you can.

For donation: Donation can be made through PayPal (via MySQL/Sun website)

Andrii and Ivan, our prayers are with you.

Memcached for MySQL: Advanced Use Cases (Recoding and Slides)

2008-07-09T11:08:00.004-04:00

My second webinar (hey it's an officially recognized word now!) on memcached, Memcached for MySQL: Advanced Use Cases, is now available on-demand from MySQL website.

Also see:

Slides for Memcached for MySQL: Advanced Use Cases

Memcached for MySQL Webinar: Advanced Use Cases

2008-07-03T10:39:00.003-04:00

Today at 1PM EST I am presenting the second part of memcached for MySQL webinar. I was told that the registration numbers look as good as the previous one. This one will be a bit more technical than the previous webinar. Sorry for the late notice but hope you can join!

Chad Hurley at Startup2Startup Dinner

2008-06-26T18:05:00.004-04:00

Tonight, I am attending Startup2Startup Dinner on Dave McClure's invitation (Thanks, Dave!). Chad Hurley, CEO and co-founder of YouTube will be speaking at this invitation only event. I will post more updates on my personal blog or you can follow me on Twitter.

Graphing Social Patterns - East

2008-06-06T17:28:00.003-04:00

In just a few minutes, I will be leaving for Graphing Social Patterns East, a conference by Oreilly. Dave McClure of 500 Hats is the conference chair. I plan to meet old friends and make new ones. It should be a lot of fun. More about Graphing Social Patterns.

Goosh: Google Shell for Geeks

2008-06-03T00:07:00.004-04:00

Ever wish you could have a browser based shell for Google? One that was clutter and advertising free? Say hello to Goosh, one of the coolest service to hit the web.

It even recognizes 'clear' :) For now, I am addicted to it.

Disaster is Inevitable - Must shutdown generators

2008-06-01T15:50:00.004-04:00

Disaster is really inevitable. Even with all the redundant power investments, ThePlanet (formerly EV1 and RackShack), had to shut down their backup generators at their H1 data center on the instructions of the fire crew. This happened after a wire-short in fault transformer led to an explosion that knocked off one of their walls, ultimately bringing 9,000 servers down. Luckily no one was injured.

This just goes on to show that just because a data center has redundant power and backup generators, it does not mean that a disaster cannot happen. IIRC, ThePlanet's last disaster was blamed on backup generators not kicking off properly.

While there was no damage to servers, I wonder how many MyISAM repairs need to be triggered once the servers do come back online?

- The Planet Status Update

Michael Arrington Asks Twitter a Few Tough Questions

2008-05-31T21:20:00.003-04:00

Michael Arrington of TechCrunch asks Twitter a few questions. I have only included a sample list below but you should read his blog post for all the questions:

Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?

A 'yes' answer to any of these questions by Twitter would be disturbing to say the least. However, it won't be surprising as companies expect databases to just somehow magically work without creating and supporting a proper architecture. High availability doesn't comes cheap and reputation for companies is everything.

I find it amusing that Twitter isn't even looking for a DBA. May be that's considered a job for the SA over there :)

Memcached Webinar - 560+ registrants

2008-05-29T22:29:00.003-04:00

A big thank you to all those who attended the memcached webinar today on which I was a panelist. I was told that there were more than 560 registrants.

The feedback I received directly and indirectly shows that there is a lot of interest about memcached. In the future, I hope to work again with MySQL/Sun on more memcached related webinars.

If you attended the webinar and have some suggestions, comments or questions, please contact me at fmashraqi at yahoo dot com or post a comment on this blog.

Special thanks to Jimmy Guerrero, Monty Taylor, Rich Taylor, Edwin DeSouza and Alex Roedling for their hard work in arranging the webinar. Also thanks to Brian Aker, Matt Ingenthron and Trond Norbye for their assistance at various phases.

In case you missed the webinar:

webinar recording: Designing and Implementing Scalable Applications with Memcached and MySQL ( you'll need Webex player to view the recording )
memcached webinar slides

Memcached Webinar - 420 Registrants and Counting!

2008-05-28T03:26:00.000-04:00

Regarding my earlier post on memcached webinar, I was informed today that more than 420 registrants have signed up. Space is limited and filling up fast so if you are interested in memcached and haven't registered yet, click on the following link to register now!

Designing and Implementing Scalable Applications with Memcached and MySQL (June 29)

Presenting a Webinar on Memcached Use Cases

2008-05-26T23:47:00.000-04:00

Quick link: register for Designing and Implementing Scalable Applications with Memcached and MySQL webinar (June 29)

Ever since its introduction, memcached has been changing the way cost-efficient caching is perceived. Some passionately love it, others cynically hate it.

Today, many large scale web 2.0 properties (including my employer) save millions of dollars by depending on memcached to bring their application response time under control and to offload pressure from databases.

There are several success stories about using memcached to speed up database driven websites. Facebook, for instance, runs the largest memcached installation and the numbers only keep increasing. In May 2007, Facebook was reportedly running 200 dedicated servers with 3TB of memory in their memcached cluster. At the "Scaling MySQL Up or Out" Keynote, Facebook revealed they are now using 805 dedicated memcached servers. That's more than a 400% increase in less than a year!

Twitter, digg, Wikipedia, SourceForge, and even Slashdot depend on memcached to keep their users happy.

For my employer, memcached has been a crucial component of the infrastructure that has been instrumental in handling explosive growth in a cost-efficient manner. In addition, memcached has helped us offload billions of queries from our database.

To highlight several real-life use cases of memcached (see below), I will be presenting a memcached webinar on Thursday, June 29 at 1 PM EST (10 AM PST). Monty Taylor (Senior Consultant, Sun Microsystems) and Jimmy Guerrero (Sr Product Marketing Manager, Sun Microsystems - Database Group) will also be speaking at the event. Space is limited and filling up fast (200+ registrants already) so I highly recommend registering now.

In this webinar, I will be covering several use cases for memcached including (but not limited to):

deterministic cache
non-deterministic cache
proactive cache
"state" cache
filesystem cache replacement

Hope to "see" you at the webinar.

Note: This memcached webinar is not to be confused with the memcached webinar being presented by Ivan Zoratti on June 28.

Interview by Sun TV at MySQL Conference

2008-05-21T21:36:00.001-04:00

At the MySQL Conference and Expo, right after my participation in scaling up or scaling out keynote panel, I talked to Sun's Multimedia team about Sun and MySQL in our environment.

Recently, I found the interview on Sun's Multimedia page. The video of my discussion is embedded below:

Interesting Internet Usage and Social Networking Statistics

2008-05-07T00:23:00.004-04:00

Over the weekend I took some notes from a presentation and did some research from various sources. The result was a blog post about Internet trends that I posted on my personal blog. There are some very interesting statistics about Internet usage and social networking. Also, Facebook fans will find some interesting facts as well.

Solaris 10 User Group Part X

2008-05-06T23:28:00.004-04:00

Tomorrow I will be attending the Solaris 10 User Group Part X at the offices of Sun Microsystems, 101 Park Ave., New York, NY. This is an all day event and there is even a MySQL talk by Philip Antoniades. Other presenters include Ambreesh Khanna, Isaac Rozenfeld, Neal Weiss, Sunay Tripathi, Amjad Khan, Damien Farnham and Dave Teszler.

Unfortunately, the event registration is now closed, but if you're attending I look forward to meeting you.

Sun's exciting technologies

2008-05-06T19:56:00.000-04:00

It's exciting to see how many technologies Sun is working on.

On May 1, I took a few members of our operations and database team to meet with Vasu Prakash who is an Engagement Architect with Global Systems Engineering division of Sun Microsystems. Vasu generously let us pick his brain regarding a wide range of exciting technologies Sun is working on and to see how they may potentially address our needs and challenges.

The following notes are my personal notes expanded with some articles from my bookmark collection.

Thumper
- Thumper (X4500) offers 48TB (SATA HDD) in a 4U at around $1.30/GB, runs Solaris OS and ZFS and supports RAID 0, 1, 0+1, 5, 6 enabled by RAID-Z and Raid Z2. X4500 supports 16GB RAM and needs 200-220 V AC for power. For non-Solaris users, other operating systems are supported as well.
- We initially evaluated Thumper as our backup storage solution but then ended up going with Sun Storage Tek. I am, however, interested in evaluating it further.
- Robert Milkowski wrote a post benchmarking Thumper and found that he was able to get more than 2GB/s aggregate write throughput using raid-5 volumes! He concludes with "Woooha! It can write more data to disks than most (all?) Intel servers can read or write to memory"
- Jason Hoffman also seems pretty pleased with Thumper
- Jonathan Schwartz's blog post announcing Thumper

ZFS
- ZFS, for those who need an introduction, is a 128-bit transactional file system offering self-healing capabilities and useful if you are running into limitations of 64-bit file systems. It is 18 billion billion times larger than 64-bit file systems.
- ZFS pooled storage can grow and shrink automaticaly.
- One of the questions I am most often asked by people is that if ZFS is really what it is then why hasn't it replaced UFS as default file system for Solaris. I would love to see a blog post by a Sun insider addressing this question.
- ZFS Best Practices Guide
- ZFS Learning Center

Solaris Containers
- For a really interesting project, I may need to create a couple hundred zones on a server (no this is not for a production system as we are a Redshift application). I was surprised to learn that more than 8000 zones (8191 non-global zones to be precise) can be created within a single operating system instance. Of course, if you do create a very high number of zones, don't benchmark boot time as it will take a very long time to boot up:)

SAM-FS
SAM-FS is short for Sun StorageTek Storage Archive Manager and it is a very exciting policy based file system by Sun. According to Sun website (it is marketing lingo but saves me the hassle):

"SAM software provides data classification, centralized meta-data management, policy based data placement, protection, migration, long-term retention, and recovery to help organizations effectively manage and utilize data according to business requirements. SAM enables users to reduce the cost of storing vast data repositories by providing a powerful, easily managed, cost-effective way to access, retain, and protect business data over its entire lifecycle. This self-protecting file system offers continuous backup and fast recovery features to help enhance productivity and improve resource utilization."

In a nutshell, if I understand correctly, SAM allows you to specify policies and then based on those policies it can move your data around from a fast-but-expensive storage to inexpensive-but-slower storage to give you the most bang for the buck. All data migration and transfer is transparent to the application. MLB is a major user of SAM. There is also an interesting case study on how MLB uses SAM.

QFS:
If NFS is your headache then QFS may provide a solution. QFS provides "nearly raw device access to information and data consolidation for read/write file sharing," according to Sun. My understanding is that using QFS requires a fibre channel to connect application servers to storage (if that's not true, can someone please correct me).

A maximum of 128 systems running QFS can share access to the same data without compromising file integrity. QFS volumes can scale up to 4PB. More QFS features are available on Sun site.

The main limitation to note: Mixed architecture (SPARC with x64) metadata servers are not supported for failover purposes. Neither are mixed architecture multi-reader configurations supported.

More Sun technologies I want to write about: Sun Cluster implementations in local (node to node), metro (run a fibre :) ) and global (global load balancer) modes. Sun cluster requires common storage that should be either direct attached or attached through a SAN switch. In addition, failure fencing, memory mirroring and vertical threading in M4000, Sun's Victoria falls processors (T5140 and T5240), PNFS and last but not least, Greenplum (claiming to be world's best database for BI and built upon PostgreSQL). Hopefully, I will talk about them in my future posts.

New Responsibilities

2008-05-03T02:28:00.005-04:00

During my university days when I was working towards a dual degree in Accounting and CIS, I co-founded a small managed hosting company which I ran for four years along with two other co-founders. Then I started a consulting company and eventually moved into online publishing. Things changed and after nearly nine years of being self employed, I took over the very challenging responsibility of single handedly managing and scaling databases of a top 50 site (in 2006). It was definitely not an easy journey and I feel ecstatic to have helped my employer handle 6x growth and rise to being a top 13 site (using same Alexa algorithm).

While I enjoy working with MySQL, Solaris and technology a lot, I really missed being part of business side. Those of you who know me outside my database role, know how much I crave problem solving related to day to day business operations especially strategic decisions, financial, product architecture, monetization, marketing, advertising and SEO etc. For me databases and scalability are very important part of running a successful business in today's environment and I am so happy to have been a key player for my employer in that area.

In short, I wanted to be more involved in both business side and technology side. So I recently accepted a new role with my current employer as Director of Business Operations and Technical Strategy. In addition, I will still be leading and training our database team.

This new role will allow me to get involved with much more than just databases at my job, something I am really looking forward to. Big thanks to my management team for recognizing my skills and giving me a chance to help my company reach new levels.

Sun loses 23% market capital

2008-05-03T01:56:00.008-04:00

Sun missed its earnings and sales estimates and as a result it lost approximately 23% of its market capital. Even more disturbing news is the announcement that Sun will be cutting 1500 to 2500 jobs. Eric Day raised his concerns as to whether this job cut will affect MySQL hiring to which Marten replied and pointed to several open positions within MySQL.

Sun has an array of very interesting and useful technologies under its hood. The amount of care Sun takes for its customers is truly impressive and I hope MySQL will follow in Sun's foot steps. Yesterday, I met with a Sun engagement architect and the amount of interest he showed in the technical challenges my team faces was unmatched. I am already working on a blog post to highlight some of the technologies my team discussed with Sun's representative.

With Sun's stock now down, I think it is an excellent time to buy some JAVA stocks which closed at 12.64. I may actually put a small order myself.

Yahoo! Mail Bug? Emails disappearing upon reaching 65,535 emails in one of the folders

2008-05-03T00:14:00.005-04:00

I am very confused.

I subscribe to several email lists including MySQL and Ruby on Rails lists. Generally, I keep my mailbox clean except for a folder in which I was archiving messages Ruby on Rails.

A few days ago I noticed that my Ruby on Rails folder reached 65535 messages. Today, I was looking to reply to an email from Keith Murphy (to which I had previously replied as well) and was surprised to find that the particular message didn't show up in my search. This particular message was sent on April 30 so I started scanning all my emails received on that day.

Surprisingly, I didn't find it even after a careful visual scan. Not only that, I noticed several of emails I received in the last 2 weeks missing. My initial reply to Keith was still sitting in my Sent mail folder. My trash folder also had several emails that I had deleted but not the ones that were missing.

For the life of me I cannot figure out where these emails went. Then suddenly I noticed that the Ruby on Rails folder still has 65535. Which is very weird as this is an active list.

I decided to send an email with criteria that would make it land in Ruby on Rails folder. After 6 hours, the email is still isn't in my inbox.

With 65,535 being a magical number representing a limitation of 65,536 objects, I believe this is a limit of a Yahoo! folder. Not only that, it seems that once you hit that limit, all sort of weird things start happening. Like, in my case, random missing emails.

This is pretty upsetting as I am not sure how many of my emails are missing. As soon as I deleted a few emails to bring the count down to 65,535, new emails from Ruby on Rails list started arriving (although not the one I had sent myself earlier today).

Now, unfortunately, I feel paranoid, not knowing how many important emails I have lost.

So, I have decided to open a new email account fmashraqi at yahoo and will be updating my contacts to start sending me email on that address.

My reason for posting it on this blog is to ask the community members if they have noticed anything like this? I know 65,535 emails is an insane number of emails but at one point I was interested in archiving the list. With Yahoo! offering unlimited storage, I wonder why isn't this limit documented?

MySQL / Linux swap problem doesn't exist on Solaris 10

2008-05-02T19:07:00.004-04:00

Right now there is a discussion on Planet MySQL regarding MySQL / Linux swap problem. Peter Zaitsev originally brought the problem of MySQL swapping to light. Recently, Dathan Pattishall also wrote about it in his post Linux 64-bit, MySQL, Swap and Memory. Don McAskill followed up with his post, MySQL and the Linux Swap problem, and an interesting way to get around the issue: "make swap partitions out of RAM disks." Don also points to another article by Kevin regarding using O_DIRECT to fix the swap issue.

To get to the point, some time ago, I experienced a similar issue on few of my old servers running Solaris V210, UFS with plenty of memory available. My initial thinking was that I am experiencing similar issue so during my presentation at MySQL Conference, Optimizing MySQL and InnoDB on Solaris 10, I pointed that this *may* exist in Solaris 10. Luckily a Sun representative (I believe it was Matt Ingenthron) corrected me towards the end of my session and pointed that UFS and Solaris 10 kernel have features built to avoid just that. That confirmation from a Sun representative was authoritative. We have already decommissioned the affected servers from production so it may be some time before I can find the precise reason why we experienced the swapping issue. Note that I haven't seen this issue on any of our other V210, V440 and T5120s in production.

Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community - Video

2008-04-30T23:25:00.003-04:00

The video of one of my three sessions, "Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community", presented at MySQL Conference & Expo 2008 has been uploaded by Sheeri. I am very thankful to her for doing all the hard work and making it available.

There are a few slides that were edited out of video because of reasons beyond my control. However, you should still be able to enjoy most of the video.

There is one point related to this video that I would like to make: Based on my particular experience I was leading to believe that Solaris 10 Kernel had the same issue as Linux Kernel related to swappiness and swapping where the kernel will start putting more importance on maintaining file system cache than the mysqld process. However, towards the end of the session, it was pointed out by a Sun engineer (thanks!) that there must be something else going on as UFS on Solaris 10 shouldn't depict this behavior and a process shouldn't swap in favor of maintaining file system cache. I am having this issue on 3 of my servers and I am currently working with Sun engineers to get to the bottom of the issue.

Velocity Conference -- Web Performance and Operations Conference

2008-04-30T22:22:00.007-04:00

I just made my reservations to attend Velocity Conference in Burlingame, CA. Velocity is a new two day conference being organized by O'Reilly. I was happy to learn at Lunch today that one of my good friends from CafeMom will also be attending. Over at Facebook I see Don McAskill has RSVP'd for the event as well.

Jesse Robbins, chair for Velocity conference graciously provided a 20% discount coupon as a comment on my blog post.

The early registration is about to end, but I find it really interesting that many slots still mention TBC (to be confirmed). I would have expected the schedule to be fully determined by now, however, I still believe this should be a great conference to attend.

Earlier I wrote about my proposed session being rejected at Velocity Conference which was a big disappointment especially since my presentation was about a top 13 website in the world. Wasn't that the point of this conference to begin with? There are several sessions at this conference that have been presented several times at other conferences including MySQL and a little Google search turns up the slides. So some company's 'secret sauce' is worth repeating and others not? Oh well, no hard feelings. As I said, I still think there would be some interesting sessions.

Let me know if you are planning to attend the conference. I will be flying to SFO on Sunday evening and flying back on Wednesday afternoon.

Don McAskill - People I met at MySQL Conference

2008-04-27T15:53:00.003-04:00

"The two metrics that are most important to me are first customer satisfaction and second growth." - Don McAskill

Today, I noticed Don is featured on Sun's customer success stories page:

Don McAskill is the CEO and Chief Geek of Smugmug, a photo and now hi-def video (using H.264) sharing site with a successful business model behind it.

I initially met Don last year at the MySQL Conference when my then boss told me that he is interested in meeting him. That was my introduction to Smugmug. I was impressed by SmugMug's presentation of photos and the care they took to make your photos and galleries look awesome.

This year, as a member of Smugmug, me and my wife got to interact with Don on a personal level.

We had several suggestions related to how our Smugmug experience could be improved and Don listened very carefully. One of the things I was most interested in seeing implemented was blocking Smugmug subdomains from being indexed if a customer is hosting them on their own subdomain.

I was truly impressed by how much Don thinks and cares about his members. It isn't a surprise that he runs a very successful site. From my conversations with Don, It seems there are many interesting projects Don and his team are working on and I can't wait to see them implemented. Almost all of the projects we heard about were focused towards customers. No wonder Smugmug has a high customer retention rate.

Technology wise, I am a fan of decisions Don has made to run Smugmug. He uses MySQL, S3, EC2 for processing and video conversion, Solaris 10 and Sun hardware.

Despite being the CEO, Don is the MySQL guy at Smugmug. His latest blog post, Death of MySQL read replication high exaggerated, was a good natured jab at discussion Brian Aker started with Arjen Lentz and me jumping in.

In the following video, Don Grantham interviews Don McAskill (yup, two Dons together) about Smugmug's relationship with Sun and the challenges of running a successful Web 2.0 business with more than 350,000 paying customers and more than 300,000,000 photos. As you can see in the video, customer satisfaction is more important than growth to Smugmug.

Since I joined Smugmug, several of my friends including Ronald Bradford have also joined. You can view my galleries by clicking on the image below and Ronald's photos from the MySQL conference by clicking on the image underneath:

If you use Smugmug as well, drop your Smugmug URL as a comment (of course, only if you want to share).

To stay up to date with exciting stuff happening at Smugmug checkout Don's blog.

People I met at the conference

2008-04-27T14:48:00.004-04:00

Every year I meet a lot of new and old friends at the MySQL conference. To highlight their involvement in the MySQL community and at the conference I have decided to start a new series: "People I met at the MySQL conference."

I probably won't be able to cover everyone I met (sorry about that) but I intend to cover as many as possible. There will be no order in which I cover people. Also, there is no secret agenda and of course whatever I say is just my personal opinion. Just whenever I have a few thoughts ready about someone, they will pop out :)

Disaster is Inevitable -- SQL Injection: Poorly Written Code and No Backups!

2008-04-26T00:03:00.006-04:00

Let me start out by saying: the best response to a disaster is backup you can count on.

Found a scary story today about hundreds of thousands of websites using Microsoft IIS and SQL Servers being affected by Internet-wide SQL injection attacks. The story originally reported by F-Secure is now on Slashdot as well.

On the IIS forum, panic is visible. Those who had backups are breathing a sigh of relief like one administrator who commented, "We have been hit by this as well. Lucky backup ran last night just prior to the attack."

Others without backups are just screwed.

F-secure reports in an update to the story, "Do note that this attack doesn't use any vulnerabilities in any of those two applications. What makes this attack possible is poorly written ASP and ASPX (.net) code."

Although this attack is targeted towards IIS and SQL Server, there are lessons to be learned for sites using other servers and databases. There are several guides available on the Internet that will show you how to secure your application against SQL Injection attacks, like http://www.blogger.com/img/gl.link.gifthis one that is focused on securing PHP and MySQL applications.

In this year's "Disaster is Inevitable--Are you Ready" presentation at the MySQL Conference (Yes, I have read Baron's post), I covered a few types of disasters. However, I missed an important kind of disaster: ones that are caused by SQL Injection. My next presentation on this topic will certainly cover this. BTW, if you missed my presentation, you can thank Artem Russakovskii, who took meticulous notes that you can read.

What saddens me is comments like, "but we have all patches applied to the version we are using." There is of course, a disconnect here as far as understanding the problem is concerned.

Patches don't secure you against SQL injection attacks; Properly written code does. Sanity check is very important!

Replication as a backup method won't help against SQL Injection
Based on my survey, a disturbingly high number of sites use replication as their backup strategy. If replication is your sole method of backup, then beware, SQL injection based disasters aren't going to help. Unless, of course, you have time delayed slaves and are able to stop replication before the slaves are affected.

Every year there are a number of backup related presentations at MySQL Conference. All, except one of the following, were presented this year!:

- What do you mean there's no backup? -- A timeless presentation by Mike Kruckenberg and Jay Pipes originally presented in 2006.
- Backup and Recovery Basics by Kai Voigt
- MySQL Backups go near continous by David Wartell
- MySQL Online Backup: An In-depth presentation by Chuck Bell
- Online Backup, Open Replication and a world of contribution by Lars Thalmann and Chuck Bell
- Performing MySQL Backups using LVM Snapshots by Lenz Grimmer
- Top 5 Considerations While Setting Up Your MySQL Backups

Scaling Up Or Out - Keynote at MySQL Conference 2008

2008-04-25T19:43:00.000-04:00

At this year's MySQL Conference I was invited to be a keynote panelist at Scaling MySQL Up Or Out keynote. Other keynote panelists included Jeff Rothschild (VP of technology at Facebook and a consulting partner with Accel Partners), Paul Tuckfield (DBA at YouTube), John Allspaw (manager of operations engineering at Flickr) and Domas Mituzas (DBA at Wikipedia). There were also representatives from MySQL (Monty Taylor) and Sun (Matt Ingenthron).

I really enjoyed being a keynote panelist with my peers. We were seated according to our Alexa ranking with the highest ranking YouTube on the right side. Even though I was representing the thirteenth largest site, our traffic compared to Facebook and YouTube was humbling.

All of the keynote panelists met early in the morning to get equipped with microphones and to go over the format.

See the video (below) to hear some funny "can't say" answers by Paul Tuckfield. I wish Google won't keep him so secretive about numbers such as how many database servers etc. Does that really give out YouTube's secrets?

Following are some photos, videos and links to notes from the keynote.

From left to right:
Monty Taylor (MySQL),Matt Ingenthron (Sun), John Allspaw (Flickr), Farhan "Frank" Mashraqi (Fotolog), Domas Mituzas (MySQL/Wikipedia), Jeff Rothschild (Facebook) and Paul Tuckfield (YouTube)

Jam packed ballroom during the keynote.

Above Photos copyright: James Duncan Davidson.

Kaj Arnö leads the scaling MySQL keynote panel discussion.

Me getting animated.

Domas Mituzas, Jeff Rothschild and Paul Tuckfield at the keynote.

Matt answers a question as everyone listens

More photos from the keynote session are available at http://photos.mashraqi.com.

Video of keynote session:
-

- Sheeri/Technocation: Download, Play

- A short video by Zack Urlocker

Notes from scaling up or out keynote:
- Biographies of keynote panelists
- Keith Murphy: Scaling MySQL - - Up or Out? Panel @ UC
- Ronald Bradford: Scaling Wisdom
- Venu Anuganti: Notes from Scaling MySQL Up or Out

MySQL on Solaris 10 -- Buffer Overflow and Security Bypass Vulnerabilities

2008-04-24T20:15:00.004-04:00

So found some recently discovered buffer overflow and security bypass vulnerabilities when running MySQL on Solaris 10. According to FrSIRT, these vulnerabilities "could be exploited by attackers or malicious users to bypass security restrictions, gain knowledge of sensitive information, cause a denial of service, or execute arbitrary code." A final resolution for these vulnerabilities is pending completion according to their website.

Unfortunately, I do not have a FrSIRT account currently (need to get one ASAP) so I couldn't dig this vulnerability further. However, I am dying to learn more about this.

Java getting fully Open Source

2008-04-23T23:52:00.007-04:00

The big news coming from Java One is that Sun is removing the last licensing hurdles in Java. What this means is Java is becoming fully Open Source.

Java users can especially thank Sun now. Also this supports Sun's vision of Open Source.

"We've been engaging with the open-source community for Java to finish off the OpenJDK project, and the specific thing that we've been working on with them is clearing the last bits that we didn't have the rights," to distribute, Sands said.

"Over the past year, we have pretty much removed most of those encumbrances," Sands said. Work still needs to be done to offer the Java sound engine and SNMP code via open source; that effort is expected to be completed this year. Developers, though, may be able to proceed without a component like the sound engine, Sands said.

Source: Yahoo News

I think Monty found the right environment to work in.

Update: Original post mentioned "Java now fully Open Source" however as the article points, Java is expected to become fully open source later this year. I wonder how much role MySQL conference played in this announcement coming earlier.

Mashable Party at Webster Hall

2008-04-22T01:34:00.003-04:00

I will be at the Mashable Party at Webster Hall on May 16, 2008. The party starts at 8PM and goes till 4 AM although I won't be staying till 4.

There are less than 100 tickets left. If you are attending and use MySQL, Solaris 10 or Sun hardware in your environment, I would love to chat with you.

And, there are no presentations :)

------ EVENT DETAILS ----

What: MashBash NYC : Mashable’s NYC Spring Party!
Who: 2,500 Sold Out Crowd, 400 Mashable VIP Tickets on Balcony, Grandmaster Flash starts the night off
When: Friday, May 16th, 2008
Drinks: Open Bar, 8 - 10 pm sponsored by Kluster
Where: Webster Hall, 125 East 11th Street, New York, NY

Schedule for the Evening: 8 - 10 pm: Mashable is hosting an exclusive 400 person VIP event on the 2nd Floor Balcony of Webster Hall’s Grand Ballroom. There will be an open bar sponsored by Kluster.com.
10:00 pm: Doors open to the public, a 2500 person sold out crowd
10:15 pm: Opening for Mashable’s VIP guests is none other than the legendary Grandmaster Flash.
Midnight till 4 am+: Mashable’s VIP guests are welcome to stay in the VIP area all night for music from acts including MSTRKRFT, L.A. Riots and more.

Back from the MySQL conference

2008-04-21T20:55:00.006-04:00

This morning I landed back at my home airport, EWR, after spending a fun-filled week and a half at the MySQL Conference 2008.

This year's conference was the best ever for me. I have a lot of people to thank and a lot to blog about. The number of pings I have received about lack of my blogging during conference is truly humbling. However, I did have a good reason for not being able to blog.

First, I was presenting three sessions, with two on the final day of the conference. Since I have the habit of continuously revising my presentations, that put a little bit of pressure on me. A big thanks to all those who came to my sessions.

Second, I was given a great opportunity to be a keynote panelist at the "Scaling up or out" session at the MySQL Conference. If you missed the keynote, you can watch the full video of the keynote posted by Sheeri.

Third, me, my wife and a few friends were invited to a trip of the lifetime by hardcore community evangelists at Proven Scaling (Jeremy Cole, Eric Bergen and Mike Griffiths). We had a great time visiting Yosemite National Park (more on this later). This was my first time without checking email or being on the Internet in nine years.

Now that I am back, I intend to put all my thoughts regarding the conference and the trip as blog posts in the coming days so stay tuned.

Heading to MySQL conference in Santa Clara

2008-04-13T15:43:00.005-04:00

I am leaving in a few hours from Monterey for Santa Clara, the home of MySQL conference. I should be in the Hyatt Regency Lobby at 5:45 PM. I still have one more space in my car so if you haven't found a ride yet to go to the pre-conference dinner, you can reserve the spot by calling me or sending me a text message at 5/5/1/6/5/5/5/5/9/0.

Facebook Scary Message

2008-04-09T12:13:00.004-04:00

A friend emailed me a message he had received when attempting to login to Facebook:

The message reads,

Warning: Facebook detected a potential scam to steam your account!
To prevent future problems, please reset your password.

Also, I was hearing in news today that a significant percentage of scams are now targeted towards social networking sites.

Of course, it goes without saying that one should not use their "important" passwords with social networking sites.

On my way to MySQL conference

2008-04-09T02:38:00.007-04:00

Later today around 5PM EST both me and my wife will be flying to San Jose to attend MySQL Conference happening next week. We will be staying the first two nights in Burlingame to meet family and friends.

Then on Friday evening we will be going to visit more family in Monterey. We will arrive at Hyatt Regency, Santa Clara, on Sunday afternoon.

Once at Hyatt, I will be happy to give a ride to anyone going to the Pre-Conference dinner.

After the conference, my plan is to spend time with a few friends. I will be flying red-eye, Sunday night, back to home.

Like previous conferences, I can't wait to see all my old and new friends.

My passions include InnoDB, memcached, BLOBs, Latent Semantic Analysis, Ruby on Rails (why won't it scale), SEO, monetization, Solaris 10, Sun hardware, Hadoop, Lucene, replication and Blue Moon :), I would love to meet/talk with other users passionate about similar stuff.

Can backup really kill performance?

2008-04-07T23:50:00.003-04:00

Yes, if you are running backing on a large database that is also handling production traffic (not a very smart idea to begin with). This is especially important for backups created using snapshots based on copy-on-write algorithm.

Brian makes an important point in a comment to my post regarding backup. He points out "Backups are always onerous on IO" and that a better way to backup is to use slaves or a standby master (if using multi master replication).

If you *must* run backups on a production server, then ibbackup becomes very important as it doesn't affect performance as much as the evil snapshots created by snapshot tools like fssnap and LVM. I have found that in our case purchasing ibbackup licenses were worth every penny.

In our environment, running backups using copy-on-write snapshots was killing performance. Writes would start stalling several hours into the backup process. It didn't help that backups would take 27 hours to finish. I moved most systems to using ibbackup and for those systems running backups hasn't been an issue at all.

Of course, if you must backup production servers, take snapshots to backup everything except the databases. That way the snapshots will be held for a much smaller period and you can continue backing up databases using ibbackup.

What about mysqldump? I don't consider mysqldump an appropriate tool for periodic backups. I can see it working for small databases but running it on enterprise level databases for daily backups is just not going to be feasible.

I would love to discuss backups more at the conference. I also would like to evaluate some of the backup vendors exhibiting at the conference.

Google App Engine Announced - Limited to 10,000 Accounts

2008-04-07T23:19:00.014-04:00

Google's announcement tonight is much bigger than I thought. Google is releasing Google AppEngine (site goes live at midnight EST) tonight, a fully-hosted, "automatically scalable" web application platform that consists of Python App servers, BigTable and GFS.

By making App Engine available only for Python, Google is giving the language a big boost.

Amazon's EC2 (Elastic Compute Cloud) allows developers to choose their own stack. Furthermore, Amazon's S3 allows third party applications to connect directly. With Google AppEngine it seems one must interact with BigTable using Python application.

Here's what Google's AppEngine promises developers:
- Write code once and deploy
- Absorb spikes in traffic
- Easily integrate with other Google services

Google App Engine is limited to first 10,000 developers
The website for Google App Engine (http://appengine.google.com/) goes live at 12:00 AM EST tonight. Only the first 10,000 developers will be given beta accounts. So hurry now before you are left out.

What is offered
The current limits imposed by Google include:
- 500 MB storage
- 200 million megacycles/day CPU time
- 10 GB bandwidth per day

Google App Engine Pricing
During the beta period, the service is completely free. Google has not announced the pricing after beta period finishes.

UPDATE:
I tried gaining an account right at 12:01 AM but thanks to Google "profiling" (which they have complete right to :) ), I got the following message:

Unfortunately, space is limited during Google App Engine's preview release. As we expand, we'll invite more developers, but for now you'll have to wait.

Would you like to be notified by email when space becomes available?

It seems like an "invite only" service. If you have invites or figure out how to get an account, please let me know. I'd love to get one.

UPDATE 2:
Many thanks to Nick Johnson of Google and others for sending me invites. Also, thanks to those who posted a comment. I was able to get an account and couldn't be happier.

Sources:
- Google Jumps Head First Into Web Services With Google AppEngine.
- Google App Engine readies for brawl with Amazon
- Google Launching App Engine for Python Developers
- Google Cloud Now on Tap for Python Developers

"The apps all appear on the appspot.com domain. Each developer currently gets three application ids. When apps are uploaded they will appear at http://application-id.appspot.com. Developers can, of course, bring their own domains. You can see the current set of apps in the application gallery. I love the Appspot domain name; it's an homage of sorts to Blogspot and fits in nicely with Jotspot."
- App Engine: Host your Python Apps with Google
- Google App Engine Blog- Introducing Google App Engine

Google's BigTable as a Web Service Announcement Expected Today

2008-04-07T10:38:00.002-04:00

According to TechCrunch, Google is expected to announce BigTable as a web service tonight.

For those unfamiliar with BigTable:

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

Pre-conference Community Dinner - MySQL Forge Registration

2008-04-07T09:21:00.003-04:00

I like Arjen's suggestion of having a pre-conference community dinner and wanted to put my name. I tried to register with softwareengineer99 and got this:

This kinds of usernames usually indicate spammers. If you feel like this was in err, contact the wiki administrator.

Return to MySQLConf2008CommunityDinner.

Ok, whatever. I then tried again with a "non spammer" username and multiple email addresses but kept getting this:

A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:

(SQL query hidden)

from within function "User::addToDatabase". MySQL returned error "1062: Duplicate entry '' for key 3 (localhost)".

So there seems to be an issue with MySQL Forge registration.

Is Backup Really Irrelevant?

2008-04-06T23:28:00.004-04:00

Brian Aker writes in his "PostgreSQL to scale to 1 billion users" post:

Backup is irrelevant for those of you who care about this discussion. LVM/ZFS snapshots are the rule of the land.

While I agree with most of Brian's statements in the article, I respectfully disagree with the statement above, especially the bolded part. Copy-on-write snapshots are EVIL for very large databases operating in a high I/O environment and backup, by no means, is entirely irrelevant. Please correct me if I am wrong but it is my understanding that both LVM and ZFS implement copy-on-write snapshots. Backup may be irrelevant for most sites but not for us.

If, however, by "irrelevant" Brian meant that not important in choosing one database over another, I can agree with that. Why? Because no one benchmarks backup methodologies until backup process starts becoming a major PITA.

Backup methods can be a performance killer when dealing with very large databases. If you're interested in finding out why, and more importantly how, ask me at the conference, come to my scaling MySQL and InnoDB on Solaris session, or check on this blog after the conference.

Is Read Replication Really Dying in Favor of Memcached?

2008-04-06T22:43:00.006-04:00

I spent my Sunday working on my three presentations that I will be presenting at the upcoming MySQL Conference. About two hours ago, as I was reviewing my stuff, I told my lovely wife that I may talk in my sessions how replication for read scalability no longer makes sense in high traffic environments. I told her, I am probably going to vote in favor of investing in memcached vs read slaves for scaling reads.

Believe it, or not, she hammered me with all sorts of questions. I spent some time answering her questions. I scanned my brain to gather more evidence to support myself including that at work we are moving and staying away from replication as much as possible.

Then, I got busy writing the post about Facebook using MySQL replication to update Memcached. After publishing the post, I checked Planet MySQL and found Arjen discussing (and agreeing with) Brian Aker's post about "The Death of Read Replication."

At that point, I simply turned my MacBook screen towards my wife and smiled :)

I consider Brian's post a brave one from MySQL point of view as I can imagine not everyone at Sun/MySQL will be happy about this. I appreciate his can

However, what Brian says about replication, caching and memcached is very true. memcached is an incredibly important part of our infrastructure. It doesn't has painful latency of MySQL replication associated with it. It requires much less hassle to setup, reset and scale. Like Facebook and all other major Web 2.0 sites, we have a considerably large memcached farm that allows us to serve our ever increasing demand.

P.S. Just to be clear, I highly favor using master-master replication for high availability and a small number of slaves. I just don't favor investing money in slaves alone for scaling reads.

P.P.S. I will leave you with a quote from Arjen's post:
"What needs to be fixed is distributed writes. And economically!"

Facebook using MySQL to replicate Memcached

2008-04-06T21:56:00.006-04:00

Faced with the challenge "to figure out a way for memcached servers to replicate data concurrently with the MySQL databases," across the country, Facebook came up with a clever solution of "embedding extra information in to the MySQL replication stream that allows [Facebook] to properly update memcached [servers] in Virginia."

This is very smart! I am curious about how they implemented this. I wonder if by "replication stream" they are just referring to binary logs. The article didn't mention whether they hacked MySQL to do synchronous replication as well, like Google. That would be really neat: synchronous replication that updates memcached.

Synchronous or not, the idea is still uber cool and I would love to see more discussion from Planet MySQL community regarding this.

Making replication possible for Brian Aker's memcached storage engine for MySQL can be another way in the future to making MySQL replicate to memcached. Brian's blog post shows:

ENGINE=MEMCACHE DEFAULT CHARSET=latin1 CONNECTION='localhost,piggy,bitters'

The multiple host specification looks very interesting. I will definitely love to talk about this with the brains at the conference.

Also, something like this would make a nice candidate for programs like Google summer of code.

Thanks to my colleague and friend A. Lee for brining this to my attention.

T5120 goes into production

2008-04-04T21:22:00.008-04:00

On Friday, after weeks of benchmarking T5120 and T5220 and studying the Sparc T2 (Niagara 2 chip) architecture, I finally put one in production and the results have been very promising. Though Friday evening wasn't a peak period, we experienced 62% more throughput than the previously deployed V210. I expect T5120 to be able to handle our peaks without breaking a sweat.

We'll have to take a hit in certain database operations to benefit from this 62% gain. (Update: however, luckily, those operations do not occur everyday.) I will be presenting results of my benchmarks and information at the MySQL conference. If you are evaluating Sun servers for MySQL, you will find my session very interesting.

Now, I can't wait to receive bunch of T5120s to replace all our db servers.

Kickfire looking to push MySQL limits

2008-04-04T14:32:00.005-04:00

For the past few months, like Baron, Jeremy and Keith, I have been consulting KickFire (formerly known as C2App). There is another startup currently in stealth mode with some very impressive solutions for MySQL. Unlike Kickfire, this other startup isn't SSD based. I was hoping they will be ready for announcement at the conference as well, but it seems they will need some more time. I cannot go into much detail on this startup at this point.

I have been wanting to write on KickFire but I certainly won't be able to beat Baron. He does a wonderful job in capturing what is KickFire and presenting a detailed insight for PlanetMySQL readers.

Like Baron, I only provided consulting and didn't get a chance to actually play with the solution. If KickFire is able to deliver what they have been promising then I can see them becoming a major solution provider to MySQL community.

I can't wait for Kickfire's keynote. Should be very interesting for those interested in giving MySQL scalability a whole new meaning.

Challenges and Payoffs of running a Tech Business in NY

2008-04-04T11:17:00.003-04:00

Another great event happening in New York on April 14th is the monthly meeting of the New York Software Industry Association. This month's topic is "Running a Tech Business in NY: Challenges and Payoffs." There is no cost to attend but you must pre-register.

Golf with Scott McNealy?

2008-04-03T22:12:00.004-04:00

Today, I missed on an awesome opportunity: to play golf later this month with Scott McNealy. Scott held the title of 'best golfer among top executives' for eight straight years.

I was made the offer to play golf today at our weekly manager's meeting. Why will I miss it? Because I will be in California, speaking at the MySQL Conference.

There are several Sun related interesting events happening in New York during the time I will be in California for the MySQL conference. This would have been a great chance for me to mingle with the top executives and talent at Sun.

I feel sad for missing this opportunity but very excited as the conference time comes closer and closer. Can't wait to see old friends and make new ones.

Lunch with Sun

2008-04-03T21:54:00.012-04:00

Yesterday, I had a very yummy lunch (Seared Halibut at Gotham Bar and Grill) with a team from Sun including Al Ballerini, Anthony Mazzei, Steve Spitz and Vasu Prakash.

The discussions were very interesting and informational. Some of the topics (that I am allowed to discuss publicly) were PNFS, QFS, LDAP for large scale authentication, Sun's new servers developed with Fujitsu and Sun's storage solutions.

Architecture wise, I was able to gain some more insight into UltraSparc T1/T2, Sparc M series, and M1 vs M2 architecture. Yes, there was clarification needed every time someone said T2 and T1 to differentiate T1000s and T2000s from UltraSparc T1 (Niagara 1) and UltraSparc T2 (Niagra 2). Someone please tell Sun they can use other letters of the alphabet to describe their servers and series.

The food, though very small in portions, was just out of this world. I can't wait to take my wife there.

Ronald is an evil genius. But we'll get you!

2008-04-02T20:38:00.004-04:00

Never in my life I have fallen victim (as severely) to an April Fools joke than the one Ronald played through his blog.

My morning started with checking servers, then heading to PlanetMySQL where I found the "sad" news. Both me and my wife spent the next hour discussing nothing else but Ronald and every topic we could think of related to his 'situation'. In the back of my mind, I was thinking that this could be a joke, but then I thought I knew Ronald well enough that he won't play a joke like this. Of course, I was wrong.

When I got Ronald's message saying "April Fools!" my response was "I HATE YOU!!!!"

In the evening, when I talked to a very good mutual friend, Marc, I found he was equally "mad" at Ronald. Today, I see that we were not alone and poor Jay was very worried.

I would love to form a coalition of all those affected by this so we can take revenge :)

Velocity Conference

2008-04-02T19:53:00.007-04:00

O'Reilly's Velocity Conference is happening this year from June 23-24 at Burlingame, CA. Velocity site describes this new conference as:

"Web companies, big and small, face many of the same challenges: sites must be faster, infrastructure needs to scale, and everything must be available to customers at all times, no matter what. Velocity is the place to obtain the crucial skills and knowledge to build successful web sites that are fast, scalable, resilient, and highly available."

When the call for papers was open for Velocity, I submitted a talk proposal regarding cutting MySQL IO for cost effective scaling and performance optimization.

Fotolog is one of the largest sites on the Internet. We are ranked 13th most visited site by Alexa and 3rd most active social network by ComScore. In the past two years, we have experienced and continue to experience incredible growth. By focusing on efficient data modeling and cutting I/O, we have literally pushed the limits of optimization and scalability when it comes to MySQL.

Learning today that my session was not accepted obviously came as a major disappointment to me. While I truly respect the conference chair's decision, I believe my session would have been useful for those who are experiencing strong growth but cannot afford to re-architect their database backend for one reason or another.

There is some good news as well: While Velocity rejected my proposal, I am presenting a somewhat similar session at this year's MySQL Conference. The session is called "Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community". If you're attending the conference and interested in knowing how you can push the limits of your MySQL database servers on Solaris, don't forget to attend my session. It will be a lot of fun, I promise!

I am also presenting two more talks at the MySQL Conference, Disaster is Inevitable—Are You Prepared? and The Power of Lucene.

'Decade Zero of Open Source'

2008-02-26T00:07:00.006-05:00

Bruce Perens, the man who created the Open Source definition on February 9, 1998, writes about the past and present of Open Source. In his State of Open Source Message, he labels the past and the future of Open Source. In his own words:

"Friday, February 8 is the last day of Decade Zero of Open Source. Saturday, February 9 is the anniversary of Open Source and the start of Decade One. It's a computer scientist thing. We always start counting from zero :-)"

The article talks about the rise of Open Source from Red Hat to most recently Sun's acquisition of MySQL. He also re-iterates the need for non traditional profit centers for Open Source companies, like in the case of MySQL.

"The largest part of the payment for Open Source development today comes from cost-center budgets of IT users, be they companies, institutions, or individuals, rather than profit-centers based on Open Source like that of MySQL. By participating in Open Source development, users distribute the cost and risk of the development of enabling technology and infrastructure for their businesses. Their profit centers are not tied to software sales, but to some other business. To find them, look to the communities rather than the companies. "

To Perens, 'Microsoft remains a problem.' whose current strategy, according to him, 'seems to be to poison us with money, most recently by making patent agreements with a number of Linux distributions.'

And regarding the potential impact of Microsoft's acquisition of Yahoo!:

"Some see the potential purchase of Yahoo by Microsoft as a threat. Certainly it might curtail or corrupt some of Yahoo's involvements in Open Source communities, and in half-Open-Source products like Zimbra. But a buy-the-loser strategy could potentially suck up a large part of Microsoft's unpleasantly (to us) ample cash while leaving them with the loser. An increase of Microsoft's influence in the content business could mean the entrance of DRM into conventional web pages. Goodbye "view source", printing without a fee, and Firefox, if Microsoft is ever successful with that. It wouldn't surprise me if Microsoft were to make more plays in the content market, perhaps investing in music and film companies. "

He also expresses his annoyance with SCO approriately calling it 'a toast'.

Overall, a very interesting read.

MySQL as a filesystem

2008-02-18T14:21:00.004-05:00

For some time now, I have been pondering about a Storage engine for MySQL that interfaces with flat files. Yes, I see a few needs that can solve for me.
Today, browsing around, I found Ben Martin's article on Using MySQL as a filesystem. The articles uses MySQLfs to get the desired results. Not 100% what I was looking for but still a good read. Ben writes:

With MySQLfs you can store a filesystem inside a MySQL relational database. MySQLfs breaks up the byte content of files that you store in its filesystem into tuples in the database, which allows you to store large files in the filesystem without requiring the database to support extremely large BLOB fields. With MySQLfs you can throw a filesystem into a MySQL database and take advantage of whatever database backup, clustering, and replication setup you have to protect your MySQLfs filesystem.

Someone please change mysqlconf.com redirection

2008-02-10T14:28:00.001-05:00

MySQLConf.com uses a non-optimal temporary 302 redirect to the MySQL conference website.

This is very bad for mysqlconf.com domain name and equally bad for people who link to http://mysqlconf.com (instead of linking to en.oreilly.com/mysql2008/) as their links then DON'T BENEFIT the conference site and from Google's point of view they are linking to a page that engages in temporary redirect. The result is unless you link directly to an oreilly.com page for your conference links, your votes/links don't get passed on to the conference site.

A 302 redirect is considered bad from search engine's point of view due to its temporary nature.

So please folks, change the redirection to 301 or I will have to go back and change my links to be "rel='nofollow'" links.

Currently, the site gives:

wget mysqlconf.com
--14:27:44--  http://mysqlconf.com/
           => `index.html.2'
Resolving mysqlconf.com... 209.204.146.28
Connecting to mysqlconf.com|209.204.146.28|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://en.oreilly.com/mysql2008/ [following]
--14:27:44--  http://en.oreilly.com/mysql2008/
           => `index.html.2'
Resolving en.oreilly.com... 208.201.239.26
Connecting to en.oreilly.com|208.201.239.26|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://en.oreilly.com/mysql2008/public/content/home [following]
--14:27:44--  http://en.oreilly.com/mysql2008/public/content/home
           => `home'
Reusing existing connection to en.oreilly.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 16,852 [text/html]

100%[==========================================>] 16,852        45.68K/s             

14:27:45 (45.63 KB/s) - `home' saved [16852/16852]

What it should give:

wget fotolog.net  
--14:33:02--  http://fotolog.net/
           => `index.html.2'
Resolving fotolog.net... 65.118.195.131
Connecting to fotolog.net|65.118.195.131|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.fotolog.com/ [following]
--14:33:03--  http://www.fotolog.com/
           => `index.html.2'
Resolving www.fotolog.com... 64.111.215.105, 64.111.215.120
Connecting to www.fotolog.com|64.111.215.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24,585 [text/html]

100%[==========================================>] 24,585        --.--K/s             

14:33:03 (191.15 KB/s) - `index.html.2' saved [24585/24585]

It doesn't just ends here. MySQL is also destroying its mysqluc.com domain in a bad manner. Look at the scary number of 302 redirects here:

wget mysqluc.com
--14:35:29--  http://mysqluc.com/
           => `index.html.3'
Resolving mysqluc.com... 209.204.146.28
Connecting to mysqluc.com|209.204.146.28|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.mysqlconf.com/ [following]
--14:35:30--  http://www.mysqlconf.com/
           => `index.html.3'
Resolving www.mysqlconf.com... 209.204.146.28
Connecting to www.mysqlconf.com|209.204.146.28|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://en.oreilly.com/mysql2008/ [following]
--14:35:30--  http://en.oreilly.com/mysql2008/
           => `index.html.3'
Resolving en.oreilly.com... 208.201.239.26
Connecting to en.oreilly.com|208.201.239.26|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://en.oreilly.com/mysql2008/public/content/home [following]
--14:35:31--  http://en.oreilly.com/mysql2008/public/content/home
           => `home.1'
Reusing existing connection to en.oreilly.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 16,852 [text/html]

100%[==========================================>] 16,852        41.99K/s             

14:35:31 (41.95 KB/s) - `home.1' saved [16852/16852]

Now I can just hope that someone actually takes action. It's small changes like this that make you get the most out of your domain or waste all the efforts you previously did in making your sites rank high.

For instance, just think how many links people created to mysqluc.com when MySQL conference was known as MySQL Users Conference. Just because MySQL used an insensible 302 redirect, all the efforts of community in linking to that domain went down the drain. The reason being that Google neither transfers the rank with 302 redirect, nor it consolidates the incoming votes/links from old domains to new domains.

Update: Why I keep talking about this: With Google's Bourbon update, I lost a very well performing site that I had worked on for many years. My investigations all pointed to using 302 redirect instead of 301 redirect. Basically, my site was wrongfully classified by Google as engaging in sneaking redirects. The site never rebounded. It was listed in Google news and was doing millions of page views a month. It had proper licensing from all content and news providers. 'Not knowing' didn't set me free in Google's court. By all means, it was a legitimate, high quality content provider site. Although this may never happen to you, Google still considers 302 a very bad form of redirecting and whenever possible it should be avoided.

InnoDB Sessions at MySQL Conference

2008-02-10T14:10:00.000-05:00

This year MySQL Conference features some of the best talks on InnoDB and I couldn't be more excited. We'll be hearing from Heikki Tuuri, Ken Jacobs, Mark Callaghan, Vadim Tkachenko, Peter Zaitsev and me :)

Kudos to conference organizers have really done a great job in balancing the sessions this year.

MySQL conference is a great venue to get up to date with what's happening with your favorite database/storage engine. Early registrations end soon so save yourself some money and register now.

If you have known me for sometime or if you are a regular blog reader, please send me a note and I will send you a coupon code to save even more when you register at the conference. You can email me at sofwareengineer99 at yahoo.

Without further ado, here are mouth watering InnoDB sessions scheduled for this year.

Mark Callaghan: Helping InnoDB Scale on Servers with Many CPU Cores and Disks

Ken Jacobs: InnoDB: Status, Architecture and New Features

Heikki Tuuri / Ken Jacobs: InnoDB: Fast, Reliable, Proven Transactional Storage for MySQL

Vadim Tkachenko and Peter Zaitsev: Investigating InnoDB Scaling Limits

Heikki Tuuri: InnoDB: Status, Architecture and New Features

Farhan "Frank" Mashraqi (Me): Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community

Sun's MySQL Acquisition Cleared by Antitrust Regulators

2008-02-10T14:02:00.000-05:00

Sun's acquisition of MySQL received an "early termination" of antitrust review by Federal antitrust regulators. As a result, Sun/MySQL acquisition has been given a green light.

Fotolog seeks MySQL DBA

2008-02-09T16:44:00.000-05:00

Fotolog is seeking a MySQL DBA. You'll be working with me in a fast-paced, small-team-running-a-very-large-scale environment.

About Fotolog.com:
• 14th most trafficked site (Alexa)
• 3rd most active social network (ComScore)
• Parent company: Hi-Media (France)

Duties
• Work with Director of Database Infrastructure to maintain/improve and support a high traffic, fine-tuned, scalable and reliable database environment on a day-to-day basis.
• Pro-active and reactive performance analysis, monitoring, troubleshooting and resolution of issues.
• Optimize and tune contentions within the database environment
• Create logging environment(s) to log usage statistics about the environment
• Regularly monitor and periodically conduct random tests of restoration from backups generated within the environment
• Participate in large storage engine migrations
• Compile/patch/install MySQL
• Work closely with operations, development and product teams to ensure smooth deployment of new iterations and availability of database services.
• Create scripts for monitoring key server utilization indicators using DTrace/Perl etc.
• Migrate production systems from MySQL 4.x to 5.x

Ideal Candidates:
Note: (You’re encouraged to apply even if you are missing a few of these guidelines)
• BS in Computer Science or equivalent
• Minimum 3 years experience with Linux, Unix or Solaris
• Minimum 2 years of MySQL experience in production environment
• Experience with Partitioned architecture(s) and data shards
• Production experience with tuning and administering moderate-large scale mission critical MySQL/InnoDB environments.
• Experience with optimizing InnoDB for both OLTP and DW queries
• Experience with estimating database capacity planning
• Passion for hunting bottlenecks and optimizing IO operations
• Solid experience with both SQL and MySQL internals
• Experience with performance analysis tools for MySQL
• In-depth experience with storage engines
• Experience with backup methodologies for MySQL
• Document best practices and routine procedures
• In-depth knowledge of MySQL tuning parameters and performance strategies
• Experience with MySQL replication
• Excellent communication and problem solving skills with attention to detail without losing the big picture
• Must be a team player as well as able to tackle projects on your own upon assignment.
• Able to handle high stress situations without losing calm and focus.
• Comfortable with carrying a pager and participating in “on call” assignments with other members of the operations team and willing to provide 24/7 escalated on-call support.
• Experience with creating and deploying scripts to automate tasks wherever possible.
• Experience with benchmarking methodologies / tools / best practices
• Experience in supporting MySQL for production / development and QA environment(s).

Bonus
• Experience with storage arrays / 3Par
• Experience with master/master replication
• Experience with Hibernate
• Alumni of MySQL Conference
• Active MySQL community member or Planet MySQL blogger
• MySQL Proxy experience
• Experience with disaster recovery

Note
• You’ll be working at our office on 5th Ave., in New York City. We are just a block away from Union Square. Please note that telecommuting is not an option for this job.

Apply
• Only the candidates themselves should send their resume to fmashraqi at fotolog dot com.

Yahoo! rejects Microsoft's hostile bid

2008-02-09T15:37:00.000-05:00

Yup, Yahoo! has finally decided to show balls and reject Microsoft's bid.

Thinking just from search point of view, a Microsoft-Yahoo merger is less evil for the search economy (and by extension online economy) than a Yahoo-Google deal. Of course, this is based on my biased view.

Update: I wonder how long before YHOO drop back to their pre-Microsoft-bid levels.

Update 2: Yahoo's 'Demented Board' rejects Microsoft.

Update 3: fixed typo.

Update 4: I should specify that the ideal outcome for Yahoo!, is to survive on its own without selling to Microsoft and without a deal with Google.

Will next billion dollar open source acquisition come in 12-18 months?

2008-02-09T01:10:00.000-05:00

Found this quote from Michael Tiemann on Matt Asay's blog:

"I would not be surprised to see another $1B deal of some sort in the next 12-18 months. The reason is simple economics...."

Will this come true? Very unlikely, especially within that time frame. Statements like this make MySQL's billion dollar acquisition look like a walk in the park. The reality is that MySQL is a leader in creating an innovative model that brings them pretty decent revenue. It has taken a lot of work from the leadership at MySQL to get it where it is.

So why is it 'simple economics'?

open source beats proprietary software as a development platform and as a value-delivery platform, and given how many millions of dollars companies are seeing wasted on proprietary software, it's only a matter of time before the majority of software technology deals are denominated in open source.

Ok, to me 'open source beats proprietary software' and opportunity cost of using proprietary vs open source software are NOT enough reasons in their own for an open source company to become a billion dollar company. Proprietary software has a high value licensing model. Open source software lacks that particular model.

There will need to be a very solid monetization plan behind the open source software for it to become that much valuable.

In order to become a billion dollar open source company, first you need enterprise strength customers. Then, you need stable and solid business model. Then you need great leadership to get your momentum going. Finally, you need a company with much more than a few billion dollars to make a leap of faith and buy you out. Or, you will need a history changing IPO and getting there will be a very difficult journey.

So, I personally do not believe at all in the argument of 'simple economics' that is presented above.

What company, if you believe in the quote, you see being acquired for a billion dollar in the next 1.5 years? RedHat, may be, but lets stick to cases where the first acquisition/IPO hasn't happened yet.

Scaling the third most active Social Network with MySQL/InnoDB/Solaris

2008-02-07T22:05:00.000-05:00

Sun Microsystems has published a case study, Delivering quality service to eleven million users with MySQL, InnoDB,
and the Solaris 10 Operating System, on Fotolog, the third most active social network according to ComScore and 14th most trafficked site according to Alexa.

Challenges:
• Scale to support eleven* million members and more than 100** million page views a day
• Increase performance without increase in database hardware (significant cost savings)

Solution
• MySQL database software
• InnoDB transactional storage engine for MySQL
• Solaris™ 10 Operating System
• Sun V440 and V210

Results
• Support for four times as many users with no additional hardware
• Higher percentage of working dataset in the memory with efficient schemas
• Four times the number of concurrent threads without adding servers
• Anticipated ability to double current number of threads

I will be presenting an updated and much more juicier version of how we achieved such scalability level from database point of view at the MySQL Conference 2008. The talk is titled, Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community. So if you needed a sign to attend the conference, well you got one now :)

Notes:
* Now, Fotolog is reaching 15 million members with same number of database servers.
** Now reaching 150 million plus page views a day

re: mysql.com and mysql-press.com in Google

2008-02-06T23:15:00.000-05:00

So I don't know who's in charge for this at MySQL so I thought I would just gain their attention by posting here. In addition, I hope this can help others in similar need/situation of serving content on multiple domains.

Basically MySQL serves content on both mysql.com and mysql-press.com which is a NO NO from the point of view of major search engines, most notably Google. Both domains go to the same IP (at least when I checked). This can potentially trigger duplicate content penalty and may even be hurting MySQL's ranking in SERPs (Search engine result pages).

Google seems to be already aware of the other domain.

Right way to serve content over multiple domains?
The ideal way to deal with serving content over multiple domains is to use a 301 Permanent Redirect from one domain to another. So when users go to mysql-press.com a 301 redirect should take them to mysql.com.

Don't dilute your PageRank
One more thing, by serving same content on multiple domains, you are diluting your PageRank. Using 301 redirect consolidates both your PageRank and potentially valuable incoming links.

BTW, This advice is not just for mysql.com. If you are serving content over multiple domains, then it can trigger Google's famous duplicate content penalty. The penalty is severe specially if you serve Google's contextual advertising on all those domains as that can potentially get your site wrongfully classified as a MFA or "Made for Adsense" site.

One quick note: there is an exception to this where if your last 2 subnets of IP address for both domains are different then there seems to be a lesser chance of penalty associated with duplicated content. At least, this was true until not so long ago.

Microsoft's $44.6 billion hostile bid for Yahoo!

2008-02-01T09:24:00.000-05:00

Microsoft has finally decided to bite the bullet and made a hostile $44.6 billion bid to acquire dying Yahoo!

I personally think that Microsoft will eventually just kill Yahoo! Although who knows. I can't wait for Microsoft advertising to advertise to hire BSD and MySQL gurus to keep Yahoo! running. That'll be funny. For some reason, I can't see them switching Yahoo! to SQL Server.

MyISAM to InnoDB Conversion: Converting almost a billion records

2008-01-29T20:35:00.000-05:00

So my challenge last week was to convert almost a billion records in 37 databases and 74 tables. Oh, and I wanted to do it only during the night (between 12:30AM and 6AM) and in the least number of nights.

This was not my first massive MyISAM to InnoDB migration, nonetheless, I still manged to learn a few things. We're at a stage where MyISAM just isn't the solution any more. Our plans are to go almost 100% InnoDB and this migration project completes yet another massive step towards that direction.

The first night was depressing. I applied every trick up my sleeve but only ended up getting 9.48% complete. Second night was only half depressing allowing me to complete ~ 33%. Third night was a charm and I got ~42.5% done. I would have finished the project that night but there was some storage issue and so I completed the last ~15% on the fourth day. Of course, optimizations and the method of conversion I selected really saved the day in the end.

Now that this has been accomplished, the next step will be to cover another 700-900 million records in a different cluster. Oh well, I will start doing that this weekend.

As far as performance is concerned, I couldn't be more happier. Load average on all servers in this cluster is well under 0.30 and each server is happily doing upto 1800 queries per second.

Note: Indexes were not dropped. Primary key did not consist of an integer. There was a composite primary key on all tables. The servers continued to handle production traffic on all databases on which migration was not being performed.

Update: What I did in a nutshell, for very large tables, was:

1. set innodb_flush_log_at_trx_commit=0 in my.cnf and restart
2. create new table tbl_innodb
3. set sort_buffer_size and read_rnd_buffer_size

SET SESSION sort_buffer_size=512*1024*1024; 
SET SESSION read_rnd_buffer_size=512*1024*1024;

4. then insert into tbl_innodb selecting from tbl_myisam and sorting by primary key
5. rename tbl_innodb to tbl

Of course, you need to have sufficient memory or adjust the buffers accordingly. But doing this ended up working great for me.

Update2: Regarding read_rnd_buffer_size, MySQL Manual says,

"If you are performing GROUP BY or ORDER BY operations on tables that are much larger than your available memory, you should increase the value of read_rnd_buffer_size to speed up the reading of rows following sorting operations."

Peter Zaitsev has also written about read_rnd_buffer_size.

Update3: My system (Solaris 10) had 10G of memory and read/writes were done to mounts residing on a SAN.

Update4: Best case scenario was 9600 records / second on average. Worst case was 500 records / second on average.

Did IBM's ERP implementation drive American LaFrance bankrupt?

2008-01-29T17:32:00.000-05:00

American LaFrance has filed for Chapter 11 after not being able to properly migrate from their ex-parent company's ERP system to a new one that was being implemented by IBM. They said that the new system caused production disruptions and left the company with a $100m debt. The company is now taking action "based upon services provided by IBM in connection with the problem-riddled transition to the ERP system."

Hmmm... since the transition was done last year, I wonder whether IBM really is to blame? Did they not have a QA environment?

MySQL dumped - Oracle loaded - Whose fault is it?

2008-01-29T12:30:00.000-05:00

The purpose of this post isn't by any way to bad mouth MySQL but rather to show the impact of lack of qualified DBAs to those who use MySQL.

I was shocked last year, when a CTO of a large company confided in me and said basically that they are so tired of searching for qualified MySQL DBAs that they may switch to Oracle. At some point, I guess the frustration alone can justify an otherwise unneeded cap-ex of such magnitude.

Today, I hear that ValueCentric, a pharmaceutical technology consulting firm based in New York, has decided to let go of MySQL in their environment and instead switched to Oracle on Oracle Linux.

"as the firm expanded and began taking on bigger clients like AstraZeneca, Roche and P&G Pharmaceuticals, company officials became concerned over MySQL's ability to keep up with rapidly growing data stores and complex quality-of-service requirements."

Now, MySQL may not be an ideal fit for many

"We were faced with some pretty big issues around performance, both in processing and on the back end, as well as reporting, where we had problems with queries returning on time and being able to handle the processing of data at night -- or whenever it came in -- in an expedient manner," Janca explained. "We needed to step back and see where we were headed strategically, do an assessment and try to remove some risk that we thought we had in continuing down the path with MySQL."

This gives the sense that MySQL cannot hold up as a company grows. However, to me, it seems like they had someone with little or no MySQL experience working on it. Or worst, their data architect did a horrible job of architecting their data and reporting environment. Or even worse, they hired services of a consultant who showed them the only way out of their issues was MySQL. There isn't a shortage of such consultants, you know.

Folks, MySQL is a tool and you need to know how to use that tool to get the job done. Just because you have been able to successfully open a tool doesn't mean that you know how to use it.

No one argues the need to have a competent DBA when one is using Oracle. However, the notion of Open Source makes people somehow believe that MySQL should run great out of the box even if the database architecture of the company sucks.

One of the biggest mistakes companies and IT managers make, IMHO, is to hire a data architect with minimal MySQL experience. They think their data can be architected by someone who doesn't necessarily has the expertise with underlying database. This can be a disastrous call as a competent data architect must possess an in-depth knowledge of the underlying database.

ValueCentric had also become disenchanted with MySQL's disaster-recovery capabilities and support services, added John LoFaso, ValueCentric's director of technical operations.

Those of you who attended my talk at MySQL Camp II, know the hell I went through on 7.5.07, when our SAN crashed leaving more than 1TB of InnoDB data severely corrupt (won't stay started even with innodb_force_recovery set to 6). For sure, I wasn't able to tap any readily-available resource to recover from this disaster. However, I was able to write programs myself to extract data from InnoDB tablespaces and the recovery rate was truly impressive. However, I realize not everyone will be wanting to write programs themselves especially in the times of extremely high stress (i.e. when disaster has occurred). This is one area that is definitely still ignored by MySQL. Peter Zaitsev recently blogged about releasing a DR tool for InnoDB however, I don't think that has happened yet.

"The support was very sketchy, and we just couldn't rely on it," he said.

Ok, since this is a somewhat touchy subject, for me and many others so I will mainly let it pass. I do however feel that you need to have the highest support plan to really get MySQL to help you. We, for one, have paid for support but hardly use it (MySQL software is so great).

I find it pretty amusing that while people use MYSQL, their budget for MySQL support is just pennies compared to when their support budget for Oracle. I believe had they spent same amount of resources on MySQL support, they would be a happy MySQL user.

One thing I will say is that we are preferring to bring in independent consultants for our next major project, though, that may/may not qualify as support to some.

So in case you are wondering why ValueCentric didn't go with Microsoft?

"because of "impersonal" support operations" and because "The goal was that we needed a partner, not an 800-number"

I believe in one thing: MySQL has a remarkable community, remarkable product, remarkable leadership and a remarkable vision so why not make the support remarkable as well? Right now, it may be great support, but it certainly isn't remarkable. And to know the difference between great and remarkable, well, you gotta read Purple Cow. (hint: they are both opposite of each other).

Going back to the shortage of competent MySQL DBAs, this is something that, I think, MySQL can artificially overcome by making their support remarkable.

However, to be fair to MySQL, the support is all there, however as with most things in life, it has a price. Ok, I seem to be in a catch 22 situation so I guess, enough for today.

BTW, thanks Ronald, for giving me Purple Cow as a gift in 2006. The book has really changed my view of thinking.

Fotolog crosses 400 million photos!!!

2008-01-28T20:03:00.000-05:00

Just a few hours ago, Fotolog crossed the 400,000,000 photos milestone. Considering that free members can only upload one photo a day, that's a lot of dedication by 14.5 million members (and growing).

Vendor Neutral Database Certification?

2008-01-28T16:47:00.000-05:00

Dave wrote about new vendor neutral database certification, or as I like to call it "Jack-of-all-trades-but-master-of-none certification "

spotting the wolf in sheep's clothing

2008-01-28T15:05:00.000-05:00

This is all my personal rant based on my little investigation as I have been very upset.

If you're the one who I am targeting this post to: please don't take it personally. You have no one but yourself to blame for this backlash.

==start rant==
So, Looking at the sites of one-who-shall-not-be-named-or-linked-to-any-more or aka wolf in sheep's clothing, I am coming to an opinion. To me, it seems like he is basically using his sites to benefit his current and/or future clients. He also seems to be a DMOZ editor or very closely associate with it, something that really hurts his credibility in my eyes. DMoz should be shredded by now as it has absolutely no value other than to lame SEOs who still either beg editors to get their site listed or enjoy being begged to. Even the parent company of DMoz has no interest in it. I mean come on. If you are 1% serious in your "database research" business you wouldn't spend time working on getting your client links into Dmoz, knowing the massive corruption that exists within Dmoz. I have dealt with Dmoz bullshit and know anyone who really values time would stay away from it.

To me, it is only a heaven for Internet marketers who try to show-off to their prospective clients about their control over certain categories and who knows, may be it even helps them get more money to write more biased posts.

Ever since Google launched Bourbon update, Dmoz and Dmoz based directories lost its value. Whoever told you that Dmoz is still powerful is just entering the SEO world. It is a base for every scraper site out there. Once you land on those scraper sites, you are asking for trouble.

To help with an analogy for the lack of better words: If you are associated with red-light district then people are going to point fingers when you go to church looking all innocent.
P.S: I don't go to church but it gets the point across

Do yourself a favor and leave DMOZ and instead start playing with MySQL or start researching your MySQL articles before posting them to an aggregated blogs community. May be now that MySQL is owned by Sun, they can give you a free Sun box and MySQL subscription to curb your ignorance about MySQL.

The guy's last three posts were the biggest waste of my time since this. I mean how do we even let people like this on Planet MySQL? To me, he is definitely interested in keyword and company plugs rather than an honest, objective review of a database that is my bread and butter. And MySQL being my bread and butter is the precise reason I am so mad about his posts. Talk about entering a community and losing respect within a week. We don't need your un-researched and biased commentary. No matter what you say, it will not make me or hundred of thousands of websites to stop using MySQL and start following you. It only makes you look bad. I mean he won't even dig up past presentations or research articles and instead ask people if they know of any one using MySQL in a mission critical environment. Ughhhh! Do your homework yourself dude.

May be there should be a moderation system where the community can mark such biased posts as 'flame bait' and after wasting enough time of the community with their stupid flame baits, they should be kicked off of Planet MySQL. Also for those who pimp their blogs, even once, there should be a badge saying 'pimped' similar to the 'employee' badge.

Last, but not least: If you don't know how to work with MySQL, don't say MySQL doesn't work since both have different meanings.

==end rant==

I may sound upset but honestly, I am tired of seeing people just loosing their conscious and saying whatever comes in their mind. Grow up!

Update: Ending thoughts: Not everyone in MySQL community loves everything about MySQL. Jeremy, Peter Zaitsev, Baron, Ronald, Roland are fierce community critics. However, their critique is always well received because it is based on some solid observation, experience or research. It is that kind of well-researched, fact-based critique that calls for discussion and dialog. Poorly researched and biased critique just wastes the time of everyone and no one gets anything in the end.

Update3 (Update2 is on top): Oh, how can I forget the numerous Planet'eers who drive this constructive-criticism-demanding community. Thanks also goes to Mark, Giuseppe, Beat, Sheeri, Keith, Kevin, Colin, Brian (Kelley and Aker), Mark and many others for providing factual based criticism. You should read all their blogs for an educated decision of whether MySQL is right for you. These are people who actually use MySQL in their jobs.

Update4: added links to constructive blogs

Hammer and Nail

2008-01-28T10:25:00.000-05:00

To Curt Monash, in response to his latest attacks on MySQL, all I have to say is:

"Those who just know how to use a hammer, approach every problem as if it were a nail."

Maria released

2008-01-27T19:06:00.001-05:00

Monty has announced on his shiny new blog that Maria is now available.

He says, "No clustered keys on roadmap."

Ah! that's the one thing I would have liked a lot. The benefits we have achieved with clustered keys are so awesome, I can't think of a good reason to reverse them. So as far as I see, Maria won't be a possible future InnoDB replacement until this is added. But other than that, it sounds that Maria is coming along pretty good.

Welcome Monty!

2008-01-27T18:03:00.001-05:00

Finally, the wait is over. "The" Monty has started blogging. Welcome, Monty!

I WILL NOT BLOB

2008-01-27T13:48:00.000-05:00

A comment was left for me by Roland and that made me want to write my thoughts.

My thoughts have changed on this topic over time. Primarily, because I now work on a large scale infrastructure where I can see how things would have been if BLOBs were stored in database. Most of the following discussion keeps the size and scale of my work load in mind and is targeted towards those who are interested in evaluating BLOBs for a work load, access patterns, budget and performance requirements similar to ours. The following are my thoughts and opinions.

When I started my current job, we were near 4 million members and just over 100 million photos. There were a lot of performance issues at that time. Today we have 14 million members and reaching 400 million photos and topping 152 million page views a day with at times reaching more than 900,000 photos a day and 8.9 million comments a day. During this time, on the database end, we only recommissioned one new test database server as production server in our guestbook cluster. Other than that one "addition", all our growth is being currently served from the same number of purchased servers/instances in our photo, friend/favorite and guestbook clusters. In that sense, one can say we have been scaling up with MySQL and Solaris 10 on Sun hardware.

There are more than available features, cost and quantity discount factors to take into account when making a decision about the DBMS that should be used for a particular environment and architecture. Just because a data type (that you weren't going to use) is available in a DBMS, shouldn't be the reason to pick it.

When you're in charge of data, you have to wear many hats to make decisions. As a data architect, your job is to take a principle based approach that is not influenced primarily by what extra features a DBMS offers but rather takes into account the "organization's strategic goals," and its budget. Experience in architectures is an important requirement to being an effective data architect as an approach considered optimal in one environment may not be equally ideal in another environment especially given unique constraints of each organization. Same is true from a database administrator point of view except that now maintainability, performance and scalability requirements are of utmost concern to you.

I was not the decision maker when my company decided to store images on file system, however, I couldn't have been more happier. If given the choice tomorrow, I would go with the decision of keeping photos outside the databases.

For an extreme high performance environment like ours where there is constant explosive growth, BLOBs could not have been the SOLUTION. In fact, had we stored nearly 400,000,000 photos (we are about to reach that milestone probably tomorrow) in *a* DBMS, we would be not only be crippling our performance and forcing our existing database infrastructure to become mostly unusable. For us, it's not just about storing 400 million photos. It's also about handling load that is generated by serving more than 4 billion page views a month. I would be very interested in seeing if any company, that is similar in specs like us and has intense performance requirements per server, like ours, uses BLOBs to store their images.

The concept of databases is generally suited to storing a very large number of objects that are small in size. Here, both "very large" and "small" are relative to each industry and requirements of the system. When you look at overall picture, file systems are more suited to handling large objects, especially if the large object consists of an image.

The decision of whether or not to store images as BLOBs MUST NOT BE DONE based on what your DBMS offers (even if its Oracle, SQL Server or MySQL). If you primarily base that decision based on your DBMS offering alone, you have done a great disservice to your organization in my opinion. The decision should be based on your performance requirements, scalability needs, ease of maintenance, cost of future features to be added to the application etc., and not on what you are more comfortable using.

Just for a moment, we have to think outside the box to be able to understand the argument of why storing BLOBs in file system is a wiser decision especially in a data-intensive web based environment.

When a client browser requests a page, it parses it out for presence of any media files and then makes a separate request to retrieve and display each media file. Typically, you can query the database for the data elements and then for each media request made later, simply use the file system to serve the request. If, however, you store media files as BLOBs, then for each media request you will need to communicate with database to get the contents of the requested media file. This can put unnecessary load on your database and increase I/O contention on your database server. Then, suddenly, you have your database server doing a lot of IO activity and your plans to keep disk I/O to minimum and serving your working data set mostly from memory goes hay wire. Some would argue that the cost of opening files is greater than accessing it from database but I wonder how long that would hold true for large media files.

For my company the size of the images is in double-digit (TT) terabytes. The size of databases is in multiple, but single digit terabytes (T). It makes no sense to me now that a database administrator would agree to bloat their databases to add TT more terabytes when they could have kept them at T terabytes and kept the other TT terabytes away from the database.

Unless the BLOB is kept in a separate table, say photo_blob, from the main table, say photo, you are going to make schema changes a nightmare for yourself, since you will be performing migrations on an additional TT terabytes. Suddenly, upgrading your database to a new version would seem like a slow walk in hell.

Same remains true for backups of database servers. By keeping the media files away from your database, you relieve your database from having to process TT terabytes of data for backup purposes which can be performed separately using file system backup tools.

If you use snapshot software to take database backups, there is something else to worry about. Snapshot utilities such as fssnap can create snapshots very quickly but as data is written to the original volume, the backup store files are also updated. This means in large environments as write load on database increases, so does your performance issues while the snapshot is held. With BLOBs you are now keeping snapshots much longer on database server so the additional TT terabytes can be backed up. While this happens, your performance will keep crippling. Instead of releasing snapshots within 5 hours, you are now waiting more than 24 hours. Yes, you will still have to backup the TT terabytes of media files, but at least you won't be making it the headache of your poor database server who already has so many things to worry about. By keeping media and its associated metadata separately, you can apply optimal backup techniques to each environment.

Oh, and what happens when you have to reset replication and move extra TT terabytes to slaves?

And let's not forget about disaster recovery issues where you need to recover data from crashed databases. More than half of your data would avoid having to be recovered in case of a database crash, if stored on a separate file system.

As we move towards higher and higher resolution devices, the size of the media files will continue to grow. 20-30MB image files are not rare anymore. Do you really want to store high resolution media in your database? If not, will you store low resolution media in database and high resolution on file system? Why?

Then there is the whole issue of fragmentation of database storage compared to file system storage. Not to mention making a mess by mixing sequential I/O and random I/O instead of separating them and benefiting from the separation.

This is not just an issue with MySQL. A 2006 study conducted by Microsoft researchers states:

"One surprising (to us at least) conclusion of our work is that storage fragmentation is the main determinant of the break-even point in the tradeoff... In essence, filesystems seem to have better fragmentation handling than databases and this drives the break-even point down from about 1MB to about 256KB."

What about Portability? Image formats such as PNG and JPEG are platform independent, whereas your database files may not be.

Of course, there are issues with garbage collection when you store your media outside of database. However, these issues should be trivial to solve for any experienced developer.

The decision to store BLOBs on file system makes even more sense from financial point of view. If an organization of our size was to move to Oracle or SQL Server, then they can significantly reduce their licensing costs by storing TT terabytes of media files on file system. I wonder how much our Oracle licensing costs would be to store and serve TT terabytes. Even with MySQL, our license fee for support would be significantly more for additional servers employed to serve TT terabytes of media.

I'll end with this thought: If you're storing your BLOBs, especially large media files in database, aren't you choosing the most expensive file system available?

Further reading:
To BLOB or Not to BLOB: Large Object Storage in a Database or a Filesystem.
To BLOB Or Not To BLOB by Deep Select.
When to put images in MySQL by Sheeri

Spider-Man or Telepathy?

2008-01-27T13:11:00.000-05:00

(Source) 3. ... Besides, an Oracle or SQL Server DBA has access to some pretty good remote tools, which let her administer many database servers at once.

Ah, this has been my biggest wish. I wish all MySQL tools will also let me remotely access databases so I don't have to put on my Spider-Man costume and jump from building to building so I can get to my database servers. With all the New York sky scrapers in my way, I can get really hurt, you know.

BTW,Does any one know if MySQL has any plans to allow me to communicate with my database servers via telepathy?

MySQL and threads: my observation and experience

2008-01-25T15:00:00.000-05:00

First: Here's to give an idea of how much traffic our site handles:

I just read the post by Monty Taylor about MySQL and threads which was in response to Curt Monash's post on why not to use MySQL.

As Director of Database Infrastructure for a top 14 Internet destination (according to Alexa), I just wanted to pitch-in with a quick comment.

Monty says, "Try running an top-10 web property with 100+ Oracle databases with a team of 1 DBA and see how long it remains running. I’d be happy to take the challenge of running the same thing on MySQL."

I couldn't agree more. I am a single person at my work working on MySQL. Our MySQL instances handle close to 30,000 queries per second at peak hours. We run MySQL on Solaris 10 with 3Par as our SAN and couldn't be more happier. We are running 64 concurrent InnoDB threads with average of 1000 max connections per server and the results are truly pleasing. For sure, I feel desperate to get some assistance but for now, it is just me managing all our installations. I have been doing that since June 2006 and without MySQL, it would not be possible. So far, I haven't yet come across a similar situation where 1 guy was managing a top 15 web property (traffic-wise) using Oracle.

And what about Curt's remaining points about why not to use MySQL? I will let Roland, a good friend of mine, address that.

When no disaster recovery plan helps

2008-01-25T14:31:00.000-05:00

Regardless of how "prepared" and "ready" one feel for a disaster, it will, in one form or another, inevitably happen. The best thing you could do is continuously revise and test your disaster recovery plan, strengthening it each time against any kind of disaster you can think of. Things generally go wrong when you least expect them to go wrong.

I was getting chills reading about Charter Communications, a St. Louis based ISP, accidentally deleted 14,000 active email accounts along with any attachments that they carried. All the deleted data of active customers is irretrievable. As someone who is responsible for data of one of the top 15 heaviest trafficked site in the world, according to Alexa, I know, I'd HATE to be in shoes of the person responsible for this.

As I was reading the news story, I was constantly thinking about the title of Jay and Mike's 2006 presentation: "What Do You Mean There's No Backup?"

Once a disaster happens, you can immediately think of the possible ways it could have been avoided. The real challenge is implementing ways of avoiding all types of disasters before they happen.

For instance, to protect against such a disaster, or at the very least, be able to recover from its effects, Charter communications could have:

1. fully tested the script on a QA/test box to ensure no test records of active users are deleted.
2. created a backup of the data by creating a file system snapshot just before running the script. That way deleted data can be recovered. Depending on your operating system/storage system, there are a lot of tools available that let you take file system snapshots such as fssnap (Solaris), LVM (Linux).
3. had a recoverable backup. There are a lot of cases out there where either no backup exists or the one that does exist, turns out to be corrupt. With a periodic backup, Charter could have, for instance, just announced to their customers that they lost their new emails since last week, instead of dropping the ball and saying that *all* their email is lost. Even having an off-site backup in this case would help if selective restore from that backup was possible.

BTW, Just a few days ago, I was testing a random sample of backups and found backups of a database to be corrupt. That triggered a system wide check of backups. The best way I have found is to have a list of backups from all databases sent to me by email. My report contains information about backups running at the time the script was generated and the backups that were created the previous night.

4. If the data deleted was on a database such as MySQL, recovery from this disaster would be possible by keeping a slave intentionally behind.

What are some of the other ways you can think of to avoid a disaster or to execute a recovery plan?

There are many ways a disaster like this can be triggered. A few, seemingly bizarre but very real, that come to my mind:

- What if you accidentally re-run a previously executed DELETE command, stored in your mysql client history, in a hurry, on the wrong server? Or you re-ran a disastrous command in your shell history in the wrong directory?

- What if you used a list of IDs generated from your QA/test machine to delete users from production machines/databases? Oh and the IDs were generated from an auto-increment column?

Can you think of more?

Sure, there are ways to prevent against each kind of disaster. The question then is: Are you prepared against 'all' of them?

The disaster recovery plan of your company may help your steer out of such a disaster, but in the case of Charter, their DR plan didn't cover this. They do, however, have plans to reimburse their customers $50. Don't know if that'd be sufficient to keep customers from switching.

If you are someone responsible for administering and executing disaster recovery plan(s) for your company, you may find my "Disaster is Inevitable--Are you ready" session at the MySQL Conference 2008 interesting. Plus, we can have great conversation afterward. :)

See also: disaster recovery, disaster recovery journal, mysql conference

Speaking at MySQL Conference 2008

2008-01-22T21:26:00.000-05:00

I have been meaning to blog about this for quite some time but time seems to be the most scarce resource in my life.

This year, I will be presenting three sessions at MySQL Conference:

Disaster Is Inevitable—Are You Prepared?

What’s the worst database disaster you expect to happen? Are you prepared? Does your architecture support quicker recovery or do you have recovery bottlenecks built throughout your architecture? What will happen if InnoDB crashed beyond repair or if you have a massive irrepairable data corruption? What can you do to better prepare for the disaster, when it does happen? Do you have data restoration tools and procedures in place in case you need to resort to extreme measures? Join us in this eye opening, heart-racing, real-life inspired presentation by Fotolog’s Director of Database Infrastructure, Farhan “Frank” Mashraqi” to find out answers to these questions and more.

Optimizing MySQL and InnoDB on Solaris 10 for World's Largest Photo Blogging Community

Fotolog is a top 19 Internet destination with more than 12 million members, 315 million photos and more than 3 billion page views a month. In just a few years Fotolog has become a social phenomenon in Europe and South America. Through modifications to its data architecture, Fotolog was able to serve four times the number of users using the same number of database servers. A non-conventional, hybrid presentation that conveys the importance of scalability, performance tuning and schema optimizations in a practical way.

The Power of Lucene

Lucene is a powerful, high-performance, full-featured text search engine library that is written entirely in Java and provides a technology suitable for all size applications requiring full-text search in heterogeneous environments.

In this presentation, learn how you can use Lucene with MySQL to offer powerful searching capabilities within your application. The presentation will cover installation, usage, optimization of Lucene, and how to interface a Ruby on Rails application with Lucene using a custom Java server. This session is highly recommended for those looking to add full-text cross-platform, database independent search capability to their application.

Registration is now open. See you all soon!

Yay! Sun Acquires MySQL

2008-01-16T15:22:00.000-05:00

In 2006, when I joined Fotolog, I made a decision: my specialty is going to be Solaris 10 and MySQL. Very quickly, I realized that MySQL on Solaris is not very commonly deployed architecture. Atleast not as common and popular as MySQL on Linux. Since then, I have promoted MySQL/InnoDB and Solaris combo at various conferences, meetups and through podcasts.

So you can imagine, I couldn't be more happier with Sun's announcement to buy MySQL. It has been one of my top wishes for MySQL and Sun to work together. When I checked my email today, I had the following comment left on my wish list post.

"Looks like your No. 3 has come true:"

And indeed, it seems it has come true.

Fotolog, the company I work for, is one of the biggest users of MySQL on Solaris. Honestly speaking, I felt a little left out being a MySQL on Solaris guy. Now, hopefully all that will change and SAMP/AMPS will hopefully takeover LAMP. At least, that's what I want to believe.

So heartiest felicitations to Marten, Monty, Brian, David, Kaj and everyone at MySQL for working so hard on MySQL and getting it to this point.

DTrace and Predictive Self-Healing Courses

2007-10-30T22:44:00.000-04:00

I plan to take a few advance SUN courses to refresh and advance my knowledge. I am particularly interested in checking out the DTrace and Solaris 10 Predictive Self-Healing courses. Has anyone taken them before? Are they any good?

BTW, a very good resource on Solaris 10 is Solaris 10 System Administration Collection.

Scaling MySQL with Solaris 10: Webinar

2007-10-24T00:57:00.002-04:00

On Wednesday, October 24, at 2 PM EST I will be presenting an Information Week webinar about scaling MySQL on Solaris 10. The webinar is free to attend and sponsored by Sun. The webinar will be followed by a Live QA session.
Register for the webinar

_DESCRIPTION
Fotolog is the world's largest photo blogging social network, boasting more than 700,000 new photos per day and more than 3 billion page views a month. More than 11 million fotologgers communicate and connect through their photos on Fotolog. With the help of Solaris 10 and MySQL Enterprise, Fotolog has scaled to become a top 20 destination on the Internet according to Alexa.

In this TechWebcast, Sun Microsystem's partner Fotolog shows creative ways used to scale their Web 2.0 architecture and MySQL Enterprise with Solaris 10 advanced features. Unlike typical scalability presentations which focus on the entire stack, Fotolog will emphasize on the role that understanding one's application, operating system and storage engine can play in addressing scalability challenges.

Web 2.0 applications world-wide rely on Sun to deliver the right combination of product innovation and capability, coupled with market leading partner solutions like Fotolog and MySQL.

Representatives from both Sun and MySQL will be present to take your questions. Hope to see you there!

Update: The recording (with audio) of this webinar is now available on Information Week website. Please note that registration is still required. After registration you should be able to view the recording.

Hi-Media to acquire Fotolog for approx. $90 million

2007-08-27T12:28:00.000-04:00

Today, Hi-Media (HIM.PA), a French publicly traded company, has announced that it is acquiring Fotolog (my employer) for approximately $90 million in a 23% cash and 77% stock transaction.

Hi-Media was founded in 1996 and currently the Hi-Media Network is the French leader in interactive marketing and the third largest in Europe.

Last year, in April, we had 3 million members and 100 million photos. Today, the number is more than 10.5 million. We are also very close to reaching 300 million photos.

"Hi-Media was attracted by Fotolog, which has only just begun converting its strong audience growth into revenues. We think that social networks are one of the pillars of what the Internet is and will be important in the years ahead, especially when underpinned by simple mechanisms. We are also convinced that Hi-Media can benefit substantially from Fotolog’s efforts to generate maximum revenues from its audience thanks to the advertising and micro-payment services that Hi-Media has developed over the past decade. In addition, we believe that the expertise and experience of the Fotolog teams who are joining us will allow us to reinforce and accelerate the development of our publishing division." -- Cyril Zimmermann, CEO of Hi-Media

Like everyone else here, I am excited. I think Fotolog will be a good match for Hi-Media.

More resources:
Official press release
Hi Media acquires Fotolog - Paid Content
Fotolog Acquired by French Ad Network - Om Malik

---
For press enquiries, please contact:
New PR Group
Andrei Bogolubov, 917-849-9300
Andrei.Bogolubov at newprgroup.com