Michael Arrington Asks Twitter a Few Tough Questions

Michael Arrington of TechCrunch asks Twitter a few questions. I have only included a sample list below but you should read his blog post for all the questions:

• Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
• Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
• Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
  

A 'yes' answer to any of these questions by Twitter would be disturbing to say the least. However, it won't be surprising as companies expect databases to just somehow magically work without creating and supporting a proper architecture. High availability doesn't comes cheap and reputation for companies is everything.

I find it amusing that Twitter isn't even looking for a DBA. May be that's considered a job for the SA over there :)

Memcached Webinar - 560+ registrants

A big thank you to all those who attended the memcached webinar today on which I was a panelist. I was told that there were more than 560 registrants.

The feedback I received directly and indirectly shows that there is a lot of interest about memcached. In the future, I hope to work again with MySQL/Sun on more memcached related webinars.

If you attended the webinar and have some suggestions, comments or questions, please contact me at fmashraqi at yahoo dot com or post a comment on this blog.

Special thanks to Jimmy Guerrero, Monty Taylor, Rich Taylor, Edwin DeSouza and Alex Roedling for their hard work in arranging the webinar. Also thanks to Brian Aker, Matt Ingenthron and Trond Norbye for their assistance at various phases.

In case you missed the webinar:

• webinar recording: Designing and Implementing Scalable Applications with Memcached and MySQL ( you'll need Webex player to view the recording )
• memcached webinar slides

Memcached Webinar - 420 Registrants and Counting!

Regarding my earlier post on memcached webinar, I was informed today that more than 420 registrants have signed up. Space is limited and filling up fast so if you are interested in memcached and haven't registered yet, click on the following link to register now!

Designing and Implementing Scalable Applications with Memcached and MySQL (June 29)

Presenting a Webinar on Memcached Use Cases

---

Quick link: register for Designing and Implementing Scalable Applications with Memcached and MySQL webinar (June 29)

---

Ever since its introduction, memcached has been changing the way cost-efficient caching is perceived. Some passionately love it, others cynically hate it.

Today, many large scale web 2.0 properties (including my employer) save millions of dollars by depending on memcached to bring their application response time under control and to offload pressure from databases.

There are several success stories about using memcached to speed up database driven websites. Facebook, for instance, runs the largest memcached installation and the numbers only keep increasing. In May 2007, Facebook was reportedly running 200 dedicated servers with 3TB of memory in their memcached cluster. At the "Scaling MySQL Up or Out" Keynote, Facebook revealed they are now using 805 dedicated memcached servers. That's more than a 400% increase in less than a year!

Twitter, digg, Wikipedia, SourceForge, and even Slashdot depend on memcached to keep their users happy.

For my employer, memcached has been a crucial component of the infrastructure that has been instrumental in handling explosive growth in a cost-efficient manner. In addition, memcached has helped us offload billions of queries from our database.

To highlight several real-life use cases of memcached (see below), I will be presenting a memcached webinar on Thursday, June 29 at 1 PM EST (10 AM PST). Monty Taylor (Senior Consultant, Sun Microsystems) and Jimmy Guerrero (Sr Product Marketing Manager, Sun Microsystems - Database Group) will also be speaking at the event. Space is limited and filling up fast (200+ registrants already) so I highly recommend registering now.

In this webinar, I will be covering several use cases for memcached including (but not limited to):

• deterministic cache
• non-deterministic cache
• proactive cache
• "state" cache
• filesystem cache replacement

Hope to "see" you at the webinar.

Note: This memcached webinar is not to be confused with the memcached webinar being presented by Ivan Zoratti on June 28.

Interview by Sun TV at MySQL Conference

At the MySQL Conference and Expo, right after my participation in scaling up or scaling out keynote panel, I talked to Sun's Multimedia team about Sun and MySQL in our environment.

Recently, I found the interview on Sun's Multimedia page. The video of my discussion is embedded below:

Interesting Internet Usage and Social Networking Statistics

Over the weekend I took some notes from a presentation and did some research from various sources. The result was a blog post about Internet trends that I posted on my personal blog. There are some very interesting statistics about Internet usage and social networking. Also, Facebook fans will find some interesting facts as well.

Solaris 10 User Group Part X

Tomorrow I will be attending the Solaris 10 User Group Part X at the offices of Sun Microsystems, 101 Park Ave., New York, NY. This is an all day event and there is even a MySQL talk by Philip Antoniades. Other presenters include Ambreesh Khanna, Isaac Rozenfeld, Neal Weiss, Sunay Tripathi, Amjad Khan, Damien Farnham and Dave Teszler.

Unfortunately, the event registration is now closed, but if you're attending I look forward to meeting you.

Sun's exciting technologies

It's exciting to see how many technologies Sun is working on.

On May 1, I took a few members of our operations and database team to meet with Vasu Prakash who is an Engagement Architect with Global Systems Engineering division of Sun Microsystems. Vasu generously let us pick his brain regarding a wide range of exciting technologies Sun is working on and to see how they may potentially address our needs and challenges.

The following notes are my personal notes expanded with some articles from my bookmark collection.

Thumper
- Thumper (X4500) offers 48TB (SATA HDD) in a 4U at around $1.30/GB, runs Solaris OS and ZFS and supports RAID 0, 1, 0+1, 5, 6 enabled by RAID-Z and Raid Z2. X4500 supports 16GB RAM and needs 200-220 V AC for power. For non-Solaris users, other operating systems are supported as well.
- We initially evaluated Thumper as our backup storage solution but then ended up going with Sun Storage Tek. I am, however, interested in evaluating it further.
- Robert Milkowski wrote a post benchmarking Thumper and found that he was able to get more than 2GB/s aggregate write throughput using raid-5 volumes! He concludes with "Woooha! It can write more data to disks than most (all?) Intel servers can read or write to memory"
- Jason Hoffman also seems pretty pleased with Thumper
- Jonathan Schwartz's blog post announcing Thumper

ZFS
- ZFS, for those who need an introduction, is a 128-bit transactional file system offering self-healing capabilities and useful if you are running into limitations of 64-bit file systems. It is 18 billion billion times larger than 64-bit file systems.
- ZFS pooled storage can grow and shrink automaticaly.
- One of the questions I am most often asked by people is that if ZFS is really what it is then why hasn't it replaced UFS as default file system for Solaris. I would love to see a blog post by a Sun insider addressing this question.
- ZFS Best Practices Guide
- ZFS Learning Center

Solaris Containers
- For a really interesting project, I may need to create a couple hundred zones on a server (no this is not for a production system as we are a Redshift application). I was surprised to learn that more than 8000 zones (8191 non-global zones to be precise) can be created within a single operating system instance. Of course, if you do create a very high number of zones, don't benchmark boot time as it will take a very long time to boot up:)

SAM-FS
SAM-FS is short for Sun StorageTek Storage Archive Manager and it is a very exciting policy based file system by Sun. According to Sun website (it is marketing lingo but saves me the hassle):

"SAM software provides data classification, centralized meta-data management, policy based data placement, protection, migration, long-term retention, and recovery to help organizations effectively manage and utilize data according to business requirements. SAM enables users to reduce the cost of storing vast data repositories by providing a powerful, easily managed, cost-effective way to access, retain, and protect business data over its entire lifecycle. This self-protecting file system offers continuous backup and fast recovery features to help enhance productivity and improve resource utilization."

In a nutshell, if I understand correctly, SAM allows you to specify policies and then based on those policies it can move your data around from a fast-but-expensive storage to inexpensive-but-slower storage to give you the most bang for the buck. All data migration and transfer is transparent to the application. MLB is a major user of SAM. There is also an interesting case study on how MLB uses SAM.

QFS:
If NFS is your headache then QFS may provide a solution. QFS provides "nearly raw device access to information and data consolidation for read/write file sharing," according to Sun. My understanding is that using QFS requires a fibre channel to connect application servers to storage (if that's not true, can someone please correct me).

A maximum of 128 systems running QFS can share access to the same data without compromising file integrity. QFS volumes can scale up to 4PB. More QFS features are available on Sun site.

The main limitation to note: Mixed architecture (SPARC with x64) metadata servers are not supported for failover purposes. Neither are mixed architecture multi-reader configurations supported.

More Sun technologies I want to write about: Sun Cluster implementations in local (node to node), metro (run a fibre :) ) and global (global load balancer) modes. Sun cluster requires common storage that should be either direct attached or attached through a SAN switch. In addition, failure fencing, memory mirroring and vertical threading in M4000, Sun's Victoria falls processors (T5140 and T5240), PNFS and last but not least, Greenplum (claiming to be world's best database for BI and built upon PostgreSQL). Hopefully, I will talk about them in my future posts.

New Responsibilities

During my university days when I was working towards a dual degree in Accounting and CIS, I co-founded a small managed hosting company which I ran for four years along with two other co-founders. Then I started a consulting company and eventually moved into online publishing. Things changed and after nearly nine years of being self employed, I took over the very challenging responsibility of single handedly managing and scaling databases of a top 50 site (in 2006). It was definitely not an easy journey and I feel ecstatic to have helped my employer handle 6x growth and rise to being a top 13 site (using same Alexa algorithm).

While I enjoy working with MySQL, Solaris and technology a lot, I really missed being part of business side. Those of you who know me outside my database role, know how much I crave problem solving related to day to day business operations especially strategic decisions, financial, product architecture, monetization, marketing, advertising and SEO etc. For me databases and scalability are very important part of running a successful business in today's environment and I am so happy to have been a key player for my employer in that area.

In short, I wanted to be more involved in both business side and technology side. So I recently accepted a new role with my current employer as Director of Business Operations and Technical Strategy. In addition, I will still be leading and training our database team.

This new role will allow me to get involved with much more than just databases at my job, something I am really looking forward to. Big thanks to my management team for recognizing my skills and giving me a chance to help my company reach new levels.

Sun loses 23% market capital

Sun missed its earnings and sales estimates and as a result it lost approximately 23% of its market capital. Even more disturbing news is the announcement that Sun will be cutting 1500 to 2500 jobs. Eric Day raised his concerns as to whether this job cut will affect MySQL hiring to which Marten replied and pointed to several open positions within MySQL.

Sun has an array of very interesting and useful technologies under its hood. The amount of care Sun takes for its customers is truly impressive and I hope MySQL will follow in Sun's foot steps. Yesterday, I met with a Sun engagement architect and the amount of interest he showed in the technical challenges my team faces was unmatched. I am already working on a blog post to highlight some of the technologies my team discussed with Sun's representative.

With Sun's stock now down, I think it is an excellent time to buy some JAVA stocks which closed at 12.64. I may actually put a small order myself.

Yahoo! Mail Bug? Emails disappearing upon reaching 65,535 emails in one of the folders

I am very confused.

I subscribe to several email lists including MySQL and Ruby on Rails lists. Generally, I keep my mailbox clean except for a folder in which I was archiving messages Ruby on Rails.

A few days ago I noticed that my Ruby on Rails folder reached 65535 messages. Today, I was looking to reply to an email from Keith Murphy (to which I had previously replied as well) and was surprised to find that the particular message didn't show up in my search. This particular message was sent on April 30 so I started scanning all my emails received on that day.

Surprisingly, I didn't find it even after a careful visual scan. Not only that, I noticed several of emails I received in the last 2 weeks missing. My initial reply to Keith was still sitting in my Sent mail folder. My trash folder also had several emails that I had deleted but not the ones that were missing.

For the life of me I cannot figure out where these emails went. Then suddenly I noticed that the Ruby on Rails folder still has 65535. Which is very weird as this is an active list.

I decided to send an email with criteria that would make it land in Ruby on Rails folder. After 6 hours, the email is still isn't in my inbox.

With 65,535 being a magical number representing a limitation of 65,536 objects, I believe this is a limit of a Yahoo! folder. Not only that, it seems that once you hit that limit, all sort of weird things start happening. Like, in my case, random missing emails.

This is pretty upsetting as I am not sure how many of my emails are missing. As soon as I deleted a few emails to bring the count down to 65,535, new emails from Ruby on Rails list started arriving (although not the one I had sent myself earlier today).

Now, unfortunately, I feel paranoid, not knowing how many important emails I have lost.

So, I have decided to open a new email account fmashraqi at yahoo and will be updating my contacts to start sending me email on that address.

My reason for posting it on this blog is to ask the community members if they have noticed anything like this? I know 65,535 emails is an insane number of emails but at one point I was interested in archiving the list. With Yahoo! offering unlimited storage, I wonder why isn't this limit documented?

MySQL / Linux swap problem doesn't exist on Solaris 10

Right now there is a discussion on Planet MySQL regarding MySQL / Linux swap problem. Peter Zaitsev originally brought the problem of MySQL swapping to light. Recently, Dathan Pattishall also wrote about it in his post Linux 64-bit, MySQL, Swap and Memory. Don McAskill followed up with his post, MySQL and the Linux Swap problem, and an interesting way to get around the issue: "make swap partitions out of RAM disks." Don also points to another article by Kevin regarding using O_DIRECT to fix the swap issue.

To get to the point, some time ago, I experienced a similar issue on few of my old servers running Solaris V210, UFS with plenty of memory available. My initial thinking was that I am experiencing similar issue so during my presentation at MySQL Conference, Optimizing MySQL and InnoDB on Solaris 10, I pointed that this *may* exist in Solaris 10. Luckily a Sun representative (I believe it was Matt Ingenthron) corrected me towards the end of my session and pointed that UFS and Solaris 10 kernel have features built to avoid just that. That confirmation from a Sun representative was authoritative. We have already decommissioned the affected servers from production so it may be some time before I can find the precise reason why we experienced the swapping issue. Note that I haven't seen this issue on any of our other V210, V440 and T5120s in production.