lj_backend: database update -- new machines, 64-bit, innodb

Brad Fitzpatrick (

bradfitz) wrote in

lj_backend,
@ 2004-10-01 15:56:00

database update -- new machines, 64-bit, innodb
After a month and a half of vendor and motherboard hell, we now have six new 64-bit database machines on their way:

Two dual 64-bit Intel Xeons (EM64T)
Two dual 64-bit AMD Opteron 246s (2.0 Ghz)
Two dual 64-bit Intel Itanium2 (1.4Ghz 1.5MB cache)

That's a total of 12 new 64-bit processors... the first 12 we've had.

Why is this notable? Because now our user clusters can run InnoDB well. We already run InnoDB on our global machines, and it kicks ass, but we've stuck with MyISAM (which is lame, but has its benefits) on the user clusters because 32-bit machines don't give a single process enough memory to run InnoDB the way we would've liked.

See, InnoDB maintains all its own caches in-process, whereas MyISAM only caches indexes, and the kernel caches data pages. On a 4GB or 8GB box on MyISAM, you can get 2GB of indexed cached in-process (because you only have 3GB of user address space on a 32-bit machine) and the rest of the memory on the box is used by the kernel to cache data pages.

But with InnoDB you only have that 2GB for everything... data and indexed. Sure, the kernel can still help out, but they step on each other's toes.

Plus InnoDB uses twice as much disk space as MyISAM, so MyISAM won there. And MyISAM is easier to sysadmin. But MyISAM has table-level locking, which sucks, but can be mitigated by having multiple databases per machine. (and with memcached it's not a big deal... only sometimes) In a nutshell, MyISAM's worked well enough for us so far, and we've come to be able to deal with (or tolerate) its deficiencies so far. But that's not to say we've been happy with MyISAM.

Anyway, we've been holding out for 64-bit for awhile now, waiting to run InnoDB effectively. Soon we'll be able to.

Also, the new machines feature:

-- 8, 12, or 16 GB of memory
-- twice as many disks in the RAID 10 as we've done in any other machine (instead of RAID 10 on 4 disks we'll have RAID 10 on 8 disks or 10 disks). and that's in addition to the RAID 1 for the operating system and DB logs volume

So it's all very exciting. Can't wait to get all the users moved to this new hardware.

(Post a new comment)

	revwry 2004-10-01 10:57 pm UTC (link)
	/me has hardware envy. (Reply to this)(Thread)

	ruthanolis 2004-10-01 11:53 pm UTC (link)
	Not the only one ... ‡ puddles of drool ‡ (Reply to this)(Parent)

	gloi 2004-10-01 11:04 pm UTC (link)
	sorry if this has been answered elsewhere, but why such a wide mix of hardware? (Reply to this)(Thread)

	bradfitz 2004-10-01 11:06 pm UTC (link)
	Two reasons: 1) We were having a bitch of a problem with vendors and availability, so we just ordered a ton of different stuff from different people, hoping to get something. 2) It just doesn't matter. We run Linux/Debian/MySQL, and they all run on anything. Hell, plus it lets us evaluate what we like for the future. (Reply to this)(Parent)(Thread)

	cuban321 2004-10-01 11:07 pm UTC (link)
	What vendor did you end up using? HP, Dell, IBM, or are they whitebox servers? (Reply to this)(Parent)(Thread)

bradfitz
2004-10-01 11:11 pm UTC (link)

HP sucked.. kept hanging up on us transferring us between departments. We were going to get the Itaniums from them but didn't.

Dell took forever to get back to me, and I kinda gave up on them.

IBM... yes, they were difficult, but we ordered the eServer 325 from them (the Opterons)

The Itaniums and Xeons are coming from Silicon Mechanics. Silicon Mechanics also built us some Opterons, but they had problems in burn-in (then didn't, then did...), so we passed on them when the IBM thing finally worked out.

(Reply to this)(Parent)(Thread)

	cuban321 2004-10-01 11:23 pm UTC (link)
	Next time you are looking for hardware email me. I can set you up a B2B account for HP and you can order and get quotes directly from the website with a discount. HP's sales people are generally pretty stupid, but the hardware is worth it (Proliants). cuban (Reply to this)(Parent)(Thread)

bdinger
2004-10-04 01:54 pm UTC (link)

HP's sales people are generally pretty stupid, but the hardware is worth it (Proliants).

I will echo this in volumes. I went direct to HP for a quote and almost jumped through the phone to choke the guy on the other end. Finally i gave up and hung up.

However... I puchased a Proliant through CDW and was quite happy. Told them what I needed and when I needed it, and it was done. I think that I'm unique in having a good account manager, I have spoke with other folks there and been unimpressed.

Anyway, yeah. Proliants are wonderful.

(Reply to this)(Parent)

	chipware 2004-10-01 11:11 pm UTC (link)
	InnoDB has transactions too.. do you guys plan to use them for any part of the system? (Reply to this)(Thread)

	bradfitz 2004-10-01 11:12 pm UTC (link)
	Yes, in the few hacky spots we had to do it by hand. MyISAM might even be deprecated and left unsupported over time. We've yet to decide a schedule for that. (Reply to this)(Parent)

	corneliousjd 2004-10-01 11:16 pm UTC (link)
	very nice. (Reply to this)

	waffqle 2004-10-01 11:59 pm UTC (link)
	Any particular reason why you went with the Itaniums? It's been my experience (and according to reviews, I'm not alone) that the performance is abysmal compared to other 64-bit chips even without taking price into account. (Reply to this)(Thread)

	toast0 2004-10-02 12:52 am UTC (link)
	I think it may have had to do with them being around relatively forever, so even if they suck performance wise, they should be readily available and working. (Reply to this)(Parent)(Thread)

	asher19 2004-10-02 05:10 am UTC (link)
	Itaniums haven't been around forever, they're practically the newest chip architecture on the block. (Reply to this)(Parent)(Thread)

toast0
2004-10-02 05:27 am UTC (link)

While the architecture is newer, the silicon is older. Itanium servers from compaq were press released as shipping as early as July 2001, but opterons started shipping in March 2004, and the intel x86-64 (or whatever you wanna call it) servers were announced last month.

Yes, compaq isn't truly representative of what's available, and they were a lot closer to launch of the processors for intel than for amd, but their press release archive is convenient.

(Reply to this)(Parent)(Thread)

asher19
2004-10-02 05:48 am UTC (link)

That's a pretty nonsensical argument. IA-64 is completely and radically different from any other ISA, and x86-64 is just a logically extended version of x86, which has been around for decades.

Further, AMD64/EM64T/x86-64 is outshipping IA-64/Itanium systems by over a factor of 10.

If anything, I'd expect the IA-64 systems to be far buggier, given how radically different the architecture is, how new all of the development tools are for it, etc. While x86-64 is technically newer (over a year old in the wild), it's likely far more bug-free than IA-64 and its tools are right now, since it's a simple and logical extension to x86.

(Reply to this)(Parent)

	dporowski 2004-10-02 12:12 am UTC (link)
	I'm assuming there's a reason for MySQL over any of the other options, but I'm wondering why MySQL in specific opposition to Postgres? (Since it's what I have to kick at all the time, and am thus most familiar with making it go.) I'm not too familiar with MySQL, so... Flexibility? Scalability? Speed, other performance reasons? Or just "We know this one already"? (Reply to this)(Thread)

	zzyzx 2004-10-02 07:21 am UTC (link)
	Here's a comparison tool thingy for you to play with: Database Server Feature Comparisons. ;) (Reply to this)(Parent)(Thread)

	dporowski 2004-10-02 08:46 pm UTC (link)
	Now that's pretty slick. Thanks! (Reply to this)(Parent)(Thread)

	zzyzx 2004-10-04 11:35 pm UTC (link)
	No problem. The main reason, I think, is probably because MySQL is generally thought to be faster. MySQL used to have a benchmark that I can no longer find. (Reply to this)(Parent)

	ydna 2004-10-02 02:34 am UTC (link)
	Think, "Go, Speed Racer! Go, Speed Racer! Go!" I love PostgreSQL for personal reasons, but MySQL has it for getting business done and it's a great company standing behind it (not to mention all the great people standing behind it too). The PostgreSQL folks have this stuff too, but MySQL is cheering for you all the way. (Have I had too much to drink or not enough?) (Reply to this)(Thread)

	ydna 2004-10-02 02:34 am UTC (link)
	Aw, shit. Clicked the wrong link. Now I have to have another drink. Stupid LJ drinking game. (Reply to this)(Parent)

	schnee 2004-10-02 08:37 am UTC (link)
	Out of curiosity, how many machines do you have in total now, anyway? (Reply to this)

krow
2004-10-02 07:22 pm UTC (link)

The theory is that the filesystem should be able to cash files better then what a record level cache can do. Hence this is why MyISAM uses the filecache, while Innodb has its own record cache.

Flip a coin, I have seen cases where both hold true. With large files though, and specific record needs on a 32bit system, I have seen Innodb hold its own. For ha_archive it seems to be able to outperform MyISAM just as soon as the files are too big for the filecache (I need about 10 more hours to test its new row level locks, hopefully I will do that this week).

(Reply to this)

	kevo295 2004-10-03 08:19 am UTC (link)
	I understand raid 10 is fast, but are there other benefits that would preclude using raid 50/51? I'm rather sure someone who knows the intimate bones of LJ (who's not necessarily on staff) will be glad to refresh my memory.... (Reply to this)(Thread)

luxography
2004-10-04 02:24 pm UTC (link)

If they built the RAID 1+0 correctly (e.g., mirrored pairs of drives together, then striped the pairs together... not the other way around), then it can withstand many more concurrent drive failures than RAID 5. There's also no parity to calculate or check in a 1+0, although that shouldn't be a bottleneck these days with hardware RAIDs. The penalty is the number of drives needed to meet a particular capacity requirement.

(Reply to this)(Parent)(Thread)

	jamesd 2004-10-04 05:07 pm UTC (link)
	On partially tuned systems I saw a 4GB RAM system with 6 disks in RAID 10 running at about three times the update rate of a 1GB 5 disk RAID 5 system. Same write caching controller, OS and CPUs, working to catch up with the same lagged replication data. InnoDB mostly and the extra RAM probably helped. Still, that's suggestive of the controller/disks being the bottleneck. (Reply to this)(Parent)

	luxography 2004-10-04 02:21 pm UTC (link)
	I would be interested to see how the different hardware platforms perform, assuming the OS, database, application and workload are similar. I'm cheering for AMD, having a dual Opteron workstation at home myself. :) (Reply to this)

	64 bits of iterwebiness imyourfault 2004-11-02 03:38 am UTC (link)
	Id like to see a picture of one of these servers (or all of them) when they actually arrive. Namely the inside of course. It would be like geek porn. I could print it out and put it inside my craptastic celeron 500 machine and pretend. Oh yes, how nice it would be, to run current software, and older software without burps and laggs and general disfunction... dreamy.... (Reply to this)

	fenixt 2005-01-01 06:13 pm UTC (link)
	so what kinda benchmarking have these machines done. how do these machines compare to one another? (Reply to this)

	Bow to the specs ryansthoughts 2005-01-04 05:45 pm UTC (link)
	Wow these specs are just awesome! The power....the power! This has been my 1st posting anywhere and only my 1st week at the site and what a great post 4 me to comment on! Thanks 4 the new servers and your awesome serve. See u all and have a great 05 everybody :) (Reply to this)

	thegreat271 2005-01-19 04:07 am UTC (link)
	sooo sweet, drools on new servers.. I dream of capability like that. (Reply to this)

	breadlord 2005-01-21 05:13 am UTC (link)
	What's the cost of the cheapest of these machines? Also, nice to hear you run Debian, for it is the king of Linux. (Reply to this)

(Reply from suspended user)

Username:		Create an Account Forgot your login? Login w/ OpenID
Password:
	Remember Me
	English • Español • Deutsch • Русский…