Morgan Tocker (mtocker) wrote,
Morgan Tocker
mtocker

On Synergy: Culture conflicts between Sun and MySQL

Working at Sun was my first acquisition experience. I guess it was what I expected; managers hyping it up about being a "perfect match", and how much the two companies had in common. It was kind of interesting to see this even turned up a notch after they received additional "Sun management training". Anyway, I digress....

I'll state upfront I consider my experience a bad one (but I'll save the personal stories for another day). Here was an issue I saw while training Sun staff on how to user MySQL:

Sun's has a conflict of interest in selling hardware.



MySQL (InnoDB) doesn't actually *work* on big computers. It only scales up to about 4-8 CPU cores, and then it hits all sorts of internal bottlenecks. Most architectures work around this by using many small machines rather than one big one (aka "scale out").

But for Sun the profits are larger on selling *bigger* hardware. Most of Sun's bigger hardware (SPARC) has many more CPU cores, but each of these cores are infact slower than most Intel/AMD cores. So it doesn't work.

I'm not sure that the "old guard" of Sun Sales people will take to selling smaller, lower margin systems. I can predict them still trying to continue to either sell bigger machines (and suggest deploying Oracle), or sell bigger machines that are actually unsuited to MySQL[1]. I remember hearing a Clayton Christensen talk on when Intel launched Celeron - and they sold them out of a completely different office. That sounded smart.

I think the idea of using commodity hardware installing DRBD+Heartbeat was the hardest to explain to Sun employees in HA classes. They didn't see why someone wouldn't buy a $5,000-$10,000 SAN and be done with it (Note: I should point out DRBD has other advantages besides being a low-cost SAN replacement).

[1] This review is just one example. The analysis is even more interesting.
Tags: mysql
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 19 comments
InnoDB is owned by Oracle, so the scalability of it is up to Oracle Corp, not Sun. MySQL is working on a new storage engine (Falcon) which will hopefully scale better with big hardware.
Right - but as the options stand today - there is nothing.

MySQL has been rather quiet on Falcon lately. How well it manages to address this issue is still unknown:

http://www.mysqlperformanceblog.com/2008/06/22/will-falcon-fly/
MySQL was going to have learn how to scale on +8x core machines anyway, CPUs are trending that direction anyway. Sun provides unique expertise in scaling multi-core systems. I don't know if you have been following Mikeal Ronstrom's blog (http://mikaelronstrom.blogspot.com/) or not but we're making good progress on this front, especially with regard to NDB Cluster. The fit may not be perfect yet, but it will be getting a whole lot better.

Matt M.
Yeah, I don't dispute work is being done (but thanks for the link anyway). This was just an observation of things I've seen existing as they are today.

Sun will probably have a lot of hurdles for Falcon to be able to 'Fly' and MySQL to really exploit having multi-core, multi-disk big machines:

* The storage engine interface is row based - I know of batched key optimizations in 6.0, but not much else.
* Query execution plans offer no parallelism (retrieve from table A and Table C at the same time, then join A->B etc).
* The query cache needs fixing - http://forge.mysql.com/worklog/task.php?id=1468
* MySQL needs a real data dictionary (related to Mikael's post).

And I don't dispute their capabilities in fixing every one of these ;)
I think the query cache is essentially obsolete. It can be ok for some small sites and thus it can be left in, but disabled by default.
We know that for big sites it can be a hindrance, plus there's Memcached.
Essentially, the query cache is from the pre-memcached era.

I've seen a lot of (unoptimized) sites that benefit from the query cache; they examine millions of rows, then just return a few. I have a love/hate relationship with it though, since it comes at a big expense to do that.

Maybe pushing it to the client like Oracle does in 10G would be a better solution?

But yeah, there's a lot of engineering work to be done. It needs to have some sort of the things that memcached has (like a slab allocator to stop fragmentation which is really common).
Arjen, so long as the query cache free block list is not over about 10k free blocks it's generally fine. FLUSH QUERY CACHE works well enough to keep it there by defragmenting the free list. The problems exist but they aren't usually troublesome. Even huge places tend to have it turned on and benefit from it, though there are always exceptions and I'd personally turn it off if the hit rate fell below 50% or so, unless that 50% was slow queries.

It's definitely not obsolete on common workloads, just showing its age and design for much lower RAM capacities than are possible today sometimes.

It's easy for either of us to pick exceptional workloads that make it look bad, of course. That worklog is built of a nice collection of those unusual workloads... :) It'll still be nice to get an improved architecture for it in place so those issues go away and we get lower overhead for the routine cases as well.

James Day
As a Sun/MySQL employee, this is one of the first things that popped into my mind when we announced the acquisition last January:

MySQL has been the 'Scale-out' company but Sun is really all about "Scale-up".

Personally, I don't see this as a problem -- but more of a challenge/opportunity. Sun has lots of really smart people who have been testing/tuning/working with Oracle for many years. How can they now use that expertise to help MySQL perform better in _all_ environments, big & small hardware?

And of course, MySQL has lots of expertise getting the most out of commodity hardware and open source software. What can we teach Sun's classic engineering, sales and marketing teams about serving new markets?

I'm not sure if this really fits under the headline of "culture", but culturally I've been impressed at how open most Sun folks have been about trying new things (not always easy for anyone) and the MySQL Dolphins have to be equally welcome to new ideas as well. Interesting times!
Perhaps I didn't explain it as well as I could have, but I think the way I met Sun people, this surfaced as culture:

- Sun comes from the top end of the market, fighting it's way down.
- MySQL comes from the bottom end, fighting it's way up.

My post was not really as much on upcoming developments - as it was to what's happening now:

MySQL doesn't scale. Sun folks will try and make it scale, but even until the moment it does, they will continue to treat it like all the other software/tools they are used to using... many of which do scale. They're not ready for scale out, they're scale up. They're not ready to learn otherwise, and in terms of sales commissions it's counter productive to learn anyway (bigger margins on bigger machines).

Then the second question is (not mentioned in my post):

If Sun does manage to make MySQL scale, will it be easy to use, and will MySQL's traditional customer base still associate with it - and will it scale for all Operating Systems? OpenSolaris might be "advanced", but its userland is painful.

I can't really think of any Sun products that are easy to use - except for maybe OpenOffice, and even then I don't use it because Go-oo.org based OpenOffice is better.
> My post was not really as much on upcoming developments - as it was
> to what's happening now:

Ha -- I can't help but notice you followed this up with a bunch of discussion of what you predicted would happen in the future! ;-)

You may be right, but only time will tell. I think both sides are ready for a change -- perhaps not evident when you departed.

Cheers!
Companies can't operate in a market below their cost-level. It's a physical impossibility.

Given industry trends, MySQL obviously has to correct its performance issues on 8+ core systems. So perhaps the goals are more in line than you think.

Regardless, you are correct that its foolish to push high end servers for use with MySQL. Instead Sun should push their excellent X4140/X4150 servers, as they are ideally suited for the demands/benefits of MySQL.
Which apparently we do...

Sun Servers Recommended for MySQL Performance & Scaling:
Sun Fire X2250 Server
Sun Fire X4150 Server
Sun Fire X4250 Server
Sun Blade X6250 Server Module

http://www.sun.com/systems/solutions/mysql/performance.jsp#offerings

Matt M.
I have no problem with the models listed on the 'Solutions for MySQL' page. I'm skeptical as to what actual improvements Sun makes for these to be 'optimized for MySQL', but they are all sensible choices. I think the biggest system has 8 cores.

What my point is:
The sales people trying to sell these 'Solutions for MySQL' systems, will be competing with other sales people *in the same company* who are trying to price together Oracle quotes (with bigger hardware, and much larger margins on it).

They'll also be in a position where they could profit *a lot more* if they suggest running MySQL on a bigger machine not on that list (scale up), rather than selecting 2 machines on the list.

As evident by 'Solutions for MySQL', Sun obviously intends to use MySQL to sell hardware. The conflict is that the way MySQL scales is at-odds with the profitability of hardware sales (which I'm pretty sure today stands as a much larger chunk of Sun's business).
Hi Morgan

Since you never were in sales, it's scary how close to the truth your vision is :-) This is exactly how it works. We then tap dance around it one way or another to make the customer end up with hardware that works well for them. It's not really a problem for the customer, but it is a silly internal thing each time it happens.

As I see it, the point of good software is to utilize less hardware resource evermore efficiently, so the HW sales guys are in for a loosing battle and we have no reason to be ashamed.

Btw, same is true for the engineering perspective in these comments: MySQL absolutely needs the multi-core scalability, and we are getting great help from Sun. Net, I think it is a clear win for sure.

(Since I don't have a livejournal account, I may as well stay anonymous for this post :-)

Anonymous

December 10 2008, 20:56:56 UTC 8 years ago

Hey Morgan,

I think there are at least some at Sun that recognize the value that DRBD provides. I forgot to mention, OpenSolaris provides similar functionality as DRBD through StorageTek Availability Suite. http://opensolaris.org/os/project/avs/

Matt M.
You can look at it the other way too -- they're hedging their bets. Some people will always buy huge systems, and some people will always buy Oracle. The opposite is true. Sun hardware is no longer in the niche of small businesses (if it was ever there) because it's so expensive. Especially considering that a new startup can just spin up some Amazon EC2 images and be done with it all.

So by owning part of the "bigger is better" market (their hardware) and a part of the "scale-out not scale-up' market (MySQL), they can bet on both sides.

A third way to look at it is the current buzzword of virtualization. Instead of getting a SAN, you can have a shared disk solution with a big server and multiple MySQL instances. This way if one database instance ends up needing way more than you realized, you don't have to run out and buy more disk. Plenty of RAM and CPU to go around too. (and as an added bonus, if the machine happens to die, you lose ALL INSTANCES!)

So there you have it -- one instance where conflict of interest is a good thing for the company overall (not putting all the eggs in one basket) and another instance where there actually isn't a conflict of interest.
Yeah, I can see your point - largely because at the moment I won't deploy on anything but EC2 ;)

Do you think it would be better to run the High and Low end more independently though? I'm thinking of a linksys and a cisco here.
It depends. I worked for a university, which was a Sun shop. This was back in 2001-2003, when OpenSolaris wasn't really around (it was, but you were a l33t hax0r if you did it)......anyway, we ran MySQL on Sun boxes. We ran Apache on Sun boxes. Heck, the IT desktops were sparcs.

I don't have real numbers, but I'm betting at least 60% of Sun hardware sales are to "Sun shops". Probably more like 75-80%. and I'd be willing to bet that most, if not all, those shops have MySQL installed.

Again, the Sun shops are going to buy Sun, no matter what's put on them. It would be hard to convince them to get, say, 1U Dells for MySQL Scale-out, even though that's a better solution.

The very small shops will never buy Sun. They just can't afford it, period.

I think it'd be a good deal if MySQL support were thrown in the deal, too. ie, say, 3 years of silver support if you buy a certain class of hardware, or 1 year of platinum support, etc.

You're also making the presumption that anyone buying Sun will have no problem buying Oracle as well. Which is not the case. If you're a sun shop with MySQL DBA's, you wouldn't buy Oracle -- you'd have the added expense of training your DBAs or hiring Oracle DBAs.

So honestly -- whether they're hedging their bets, concentrating on increasing scale-up capabilities of MySQL, banking on virtualizaton, or heck -- all of the above -- it's likely a good business decision because they can go in any and all the directions.