Wednesday, June 27, 2007

My day

I would like to remember this day. It was just crazy. Most of my day went into troubleshooting InnoDB locking issues. At this point, the situation is all clear and I am very pleased with the results. Here are two graphs from MySQL Network Monitoring tool:

Keep manageable threads/connections and your server will be very happy.

New Planet MySQL Layout!

Just noticed that Planet MySQL has a new layout. Overall, I like the change. The text seems much easier to read. Ads are now gone. For each active contributor, a score is shown. Don't know if the score is number of posts or what? Does anyone care to shed some light?

Wednesday, June 20, 2007

MySpace for 25% Yahoo!

Note: The following views are just that: mine and not my employer's.

News Corp is considering swapping MySpace for 25% share in Yahoo.

Made me think: Is it "the" sign that Yahoo! has given up or is MySpace really worth that much?

Yahoo! holds some key IP assets and it can certainly make a come back, or so I believe. It is just a matter of leadership that can find the right balance between monetization and user experience.

Monetization today is all about contextual. Sure there are many companies today that are pitching behavioral targeting as bigger, but honestly speaking, we are not there yet. Behavioral targeting just cannot monetize as well, at least not now.

Contextual, on the other hand, is a gold mine with relevancy being the key. I cannot stress how important the relevancy is. It is relevancy that is not spam, satisfies a query and makes a user feel ads are not intrusive. Relevancy is so important that I believe it was the primary reason that DoubleClick didn't become Google. Lack of attention to relevancy is where MSN fails as well.

Sure, Google can and will continue to diversify into many areas, but I doubt they will find a cash cow that is not based on, or directed related to relevancy.

Valuing MySpace for 25% of Yahoo! is not a good deal from Yahoo's point of view, at least not from the financial perspective. Giving the changes that are in the air, I would value Yahoo! for what it will be worth five years from now. Making such a big decision today is going to cause a lot of regrets later.

I believe, that's a lot of Yahoo! for a lot of teenagers who may not get Yahoo! out of the situation it currently is in: lack of balance between monetization and relevancy. If you see things differently, I'd love to know.

Tuesday, June 19, 2007

My Top X Wishlist for MySQL

DISCLAIMER (and yes, it's a long one):
Over the years, I have come to love MySQL. I have many friends, colleagues and peers that depend on MySQL in one way or the other to make their living. Friends whose lives and passions revolve around MySQL. Friends who want to change the world (with MySQL). Friends who want to take MySQL to the next level.

I am so fascinated about MySQL's business that when I first received a call for comments that MySQL is re-launching MySQL Enterprise by dividing the development tree, I felt very happy. I had been a part of various conversations by then debating whether a company should invest in MySQL Enterprise. My view was: if you're a tad bit serious about your business or product, you need MySQL Enterprise. For businesses of all sort, that's an investment, which actually has the potential of lowering TCO (yes, I believe so). I remember saying that, in many ways, branching off the development tree is a great idea as it will help define the Enterprise "product" and make the decision making process easier for CIOs, CTOs, DBAs and other decision makers involved in the process.

My good friend, Jay who organized the very excellent MySQL Conference this year, wrote about his top 5 wishes for MySQL. Jay also managed to get the Top 5 Wishlist of our favorite CEO. Several others on Planet are writing theirs, so here is mine.

My personal wishes aren't just about MySQL software since my involvement is more than the general role of a DBA. I do NOT want this to sound like anything other than what it is: my personal wish list based on my experience of becoming a MySQL customer.

1. Integrated Backup Solution: Enough said!

2. Open Marketing for an Open-Source Company: I wish MySQL would tell customers about products/services that are just about to launch.

What? or more importantly, Why?

I am always surprised by entrepreneurs who would use MySQL, but not pay for support. So naturally, I made a case for MySQL to my boss. We signed up though there seemed to be a sense of urgency. Then within weeks, MySQL Enterprise Unlimited was launched. Sad part is that no offer was made to newly acquired customers on how to take advantage of this. Actions like these have the potential to turn even a company's most loyal customers with a bad taste in their mouth.

When I was running my own very small business before my current position, I learned very quickly that a surprised customer is a half-lost customer. Surprises are just never good and I wish every company would understand that from a customer's point of view.

My company paid virtually the same amount as MySQL Unlimited customers, but they ended up getting unlimited servers and a plan higher than ours.

3. Better support for MySQL on Solaris

Solaris 10 is a great operating system and MySQL a great database. Together, they can do miracles and I have become a believer. I seriously wish MySQL and Sun would join forces and work together even more. While there is currently work going on between the two companies, it is not enough. There is a huge demand for MySQL on Solaris and by not addressing this demand, both companies are losing out.

4. Upsell and Service: Two polarities to manage
I understand the need to upsell, however, that's certainly a polarity to be managed along with service. Seeing polarity as a problem, or just seeing one end of the polarity means not seeing the whole picture. I wish the very good folks at MySQL would not try to label every other question as falling into categories that aren't covered by a customer's current support plan.

One way is to put some more meaty content in the knowledge base that is available to customers. I wish the knowledge base was filled with real-life content. MySQL's Professional Services has so many smart brains working for them and their generified experiences will add a lot of meaning to the knowledge base.

5a. Tablespace per database that is portable for InnoDB.

It will be nice if reliance on a global tablespace is decreased and all information related to tables within a database resides in a tablespace for that database.

5b. InnoDB Parameters on the fly: Ability to change InnoDB parameters on the fly without having to restart MySQL.

6. Better supervising support: If a support ticket receives 40 (or more) emails, someone should check what's going on and why an issue is taking so much time to resolve. For sure, folks can ignore that, but stuff like that is sure to leave a customer annoyed.

7. Open Source or Launch MySQL NMAAS as a stand-alone product: I know that is almost impossible, but it would be a great product. I think price of reporting software shouldn't be based on the support contract. Customers should be able to license NMAAS alone. Why stop at that? I'd say open source NMAAS as well.

Once again, these were my personal thoughts and wishes. I tried hard to limit them to the most important ones (as I feel). Thank you for reading them.

Monday, June 18, 2007

MySQL's The 12 Days of Scale Out

Note: My views are just that: mine and not employer's
MySQL's new series The 12 Days of Scale Out received some very positive criticism from Jeremy Cole and Kevin Burton. MySQL was very quick to act on the opportunity and launch a section primarily targeted towards CIOs, a move that I applaud.

I wish a certain division to be made in the training courses as well where I felt that quite a lot of time is spent on giving marketing messages rather than provide meaty training. Don't take me wrong, there is stuff to be learned however the content has a lot of room for improvement.

The other thing that kind of surprised me is that Fotolog was not included in the 12 Days of Scale Out. Kind of weird since we are a MySQL customer (more on this soon) and deliver more page views than most companies mentioned.

I included just for fun

Anyway, other exciting news is I just found out that Fotolog is now 20th site on Alexa Global 500 list just below eBay.

Ok, just one more thing. I just completed my first year at Fotolog :)

Terry Semel (Yahoo! CEO) Quits

Weeks after Yahoo! CTO's resignation (or was he forced out :)), Yahoo! CEO quits. Unlike Google, Yahoo! turned to old media to lead the company. The results of course were way below par. Terry Semel was a controversial choice but one that at times, brought hope. At no time in Semel's tenure that hope seemed sustainable. Destruction of Overture, failed acquisitions like Web Jay and lack of effective strategy (yes Yahoo! Photos, yes Flickr!, no Yahoo! Photos).

Jerry Yang is himself stepping up to take charge. According to "The Google Story," this is exactly what was so hard to give up at Google where all kinds of pressure was applied to let go of the CEO position.

But does it really matter whether Jerry Yang holds the CEO title or not? Yahoo!'s focus on centralized decision making is famous enough to be part of Strategic Management courses. The question that comes to my mind is whether Terry was allowed to do what he wanted or his position was just used to get the signatures.

Is Jerry the right person who can lead Yahoo! to recovery? We'll see. But the change itself doesn't seem exciting or promising to me. As a Yahoo! user, I would have liked to see someone other than the "obvious choice".

On a side note, I am highly disappointed by the extreme focus on commercialization of Yahoo!'s home page. Too many ads and too many commercial links.

InnoDB table locks

When I first learned that having an AUTO-INC column causes a table lock in InnoDB, I was surprised. Since that day, I have been warning just about everyone regarding this.

In addition to the AUTO-INC table lock, there seem to be some other conditions where table lock is requested. I am saying this because on some of my InnoDB tables, I am seeing "tables in use 1, locked 1."

Do you know of any other conditions that can cause InnoDB to request a table lock? I have heard that using REPLACE or UNIQUE indexes on MySQL 4.1.X can cause this behavior.

If you know of any other situation where InnoDB will request a table lock, would you please leave a comment or email me (softwareengineer99 at Yahoo).

As I find information regarding this, I plan to document it here.

MySQL Internals: Concurrent inserts causing table locks in InnoDB that go away when innodb_table_locks=0 is added to my.ini on Windows. The question raised, but not answered: "Do some queries implicitly add LOCK_TABLE?"

MySQL Internals: What is the intention lock check in InnoDB needed?
Heikki: "the intention lock is required: always when InnoDB sets an S-lock on a
record, there must be an IS-lock on the table."
- "InnoDB need read lock because binlog is enabled"

MySQL Bug 16229: InnoDB uses full explicit table locks in trigger processing: Fixed in 5.0.19.

Jeremy Cole encouraged Heikki and MySQL to change the way table and row locking is handled by MySQL and InnoDB, something that I totally support. I have heard Jeremy pitch this at nearly every ocassion but sadly I haven't seen any action. Jeremy says,
It seems like it would make more sense to reverse the locking procedures 
in MySQL and their handling by the storage engines. As I understand it,
currently MySQL asks for a table lock in all cases and the storage
engine (InnoDB, in this case) may downgrade that to a row lock.

Wouldn't it make far more sense for MySQL to ask for a row lock, and the
storage engine (MyISAM, in this case) to upgrade that to a page lock,
table lock, etc. as support is available? Basically MySQL should ask
the storage engine for the most granular lock possible, and the storage
engine should upgrade that lock if it is unable to lock with that

MySQL Internals: Heikki pushed a change into 4.1.9 (just a comment). The changes were primarily made in InnoDB table lock is now acquired in LOCK TABLES if
AUTOCOMMIT=1. This helps to avoid deadlocks when porting old MyISAM applications to

+ /* Starting from 4.1.9, no InnoDB table lock is taken in LOCK
+ TABLES if AUTOCOMMIT=1. It does not make much sense to acquire
+ an InnoDB table lock if it is released immediately at the end
+ of LOCK TABLES, and InnoDB's table locks in that case cause
+ VERY easily deadlocks. */

Wednesday, June 13, 2007

New York MySQL Meetup

Yesterday night, I presented "Scaling InnoDB for Fotolog" at the New York MySQL Meetup. Judging from the number of questions, and how many people stayed after the presentation was over, I can say that it was a success.

I was very excited when I saw Partha Dutta (of Right Media) enter the room. Had really good chat with folks from New York Times.

Later, Kerry Ancheta, Partha, Michelle and I went to Les Halles. I had "cotes d'Agneau grillees au Romarin", which was really yummy.

A big thanks to Marc Simony of Logic Works for inviting me to speak and for sponsoring the event.

Sunday, June 10, 2007

No, I didn't release MySQL Archiver, Baron did

On my earlier post, I got a comment that PlanetMySQL outage was related to a data center issue. However, something I am seeing tells me a different story.

Baron (Xaprb) released MySQL Archiver. However, today's PlanetMySQL outage seems to have caused an interesting effect: it seems to have gotten confused about the relationship between posts.

So here's what is happening:

The post was written by Baron. My name is linking to my blog ( The post's title "MySQL Archiver 0.9.2 released" is linking to a post I made today (

Of course, the post should be linking to ( and my name shouldn't be there.

This doesn't looks good at all.

UPDATE: It seems like the relations on Planet MySQL are off by one.

Saturday, June 09, 2007

PlanetMySQL down?

I am getting
2013: Lost connection to MySQL server during query

I think they need the services of MySQL Professional Services :)

Tuesday, June 05, 2007

New York Tech Meetup

Tonight, I went for the first time to the NY Tech Meetup. (Meetup's founder is also Fotolog's co-founder).

Ever wanted to create your own television station And be able to professionally produce live shows just like your favorite cable tv channel? What about being able to instantly add any video from YouTube, Revver or other video sharing sites? And what if all this was free? Close your mouth, I can see your mouth watering already.

Check out what I honestly think has the potential to be ground breaking: Mogulus. I had the chance to meet and see Max Haot, CEO of Mogulus, at the NY Tech Meetup. The entire interface is flash based and therefore no download is necessary.

I think Mogulus seems to be using S3 and EC2 as Max answered a question about CDN by making a reference to Amazon's bandwidth. If so, seems like a perfect match for EC2 and S3.

Mogulus excited me, but the highlight of the NY Tech Meetup tonight was David Weinberger, an excellent speaker and author of "Everything is Miscellaneous." David's presentation was one of the fastest (in terms of words per minute) presentation that I have ever attended. Yet, it made perfect sense.

As someone who is very passionate about data mining, information retrieval and latent semantic analysis, I found David's definition to be eye opening. It's amazing how you live and breathe one thing and then suddenly a few words make you think that's all you have been wanting to here. I will provide the definition here but it has a context to which it belongs which must be understood to really appreciate this definition. The classical way to define metadata is that it is "data about data." David takes this definition to one step further and says that metadata is data itself and that "metadata is what users know" and "data is what they are searching for."

Some other ideas presented were: GoLoco, ExpoTv and LiveLook.

BTW, at the next New York MySQL Meetup, I am presenting a session on scaling InnoDB. If you are in New York, stop by for a few drinks (and to attend the presentation :) )

Monday, June 04, 2007

Amp'd Mobile bankrupt?

If I recall correctly, I saw a post about Amp'd Mobile going bankrupt, but now that I wanted to read it, it's gone. So how can this be? I thought they were doing really good especially since they were one of the winners at this year's MySQL Application of the year. I wonder what went wrong.

The official statement from Ampd Mobile:

"As a result of our rapid growth, our back-end infrastructure was unable to keep up with customer demand. We are taking this step as a necessary and responsible action to sustain and strengthen our momentum in the market place".

Ok, Now I am really curious. Which part of backend? Network? Application? Storage? ... Database?

And whatever these backend scalability issues were, why weren't they actively resolved? Could making a company scalable drive it bankrupt? If not, then what happened here?

From what I can think, the problem here probably lies with the implementation of the backend, not the individual components of the infrastructure.

It will probably be some time before we will know what really happened to Amp'd. But I am curious to know what happened.