Arjen's Journal

InnoDB Performance Tuning (training course day for MySQL)

Fri, 25 Jul 2008 00:21:03 GMT

This DBA course day is in part based on info from Peter Zaitsev 's earlier explorations (thanks Peter), and of course new insights from the High Performance MySQL book (there's a lot of Baron in there, all credit to his great work!), wisdom from Mark Callaghan, with my own teach approach and style thrown in to turn it into a good yet fun educational day!

Just like with other topics, extending this to a full day allows the issue to be explored much better rather than getting skimmed over as a small part of a much bigger course. After all, if you understand the tool, how it does things and why, you can utilise it more optimally. The skimming wouldn't help you in this sense.

I still have some customers use MyISAM (out of habit mainly) but the default storage engine really is InnoDB, effectively, even if the server build does not see it that way by default ;-) But there is more to tune, things to know. Just converting to InnoDB does not make the application run any better necessarily. It might reduce lock contention in some cases (like update heavy apps), but the overall running might be slower without the right setup. While maintaining ACID, there's much that can be done.

The course is part of the DBA course days that Open Query has currently scheduled in cities around Australia, and in Auckland New Zealand. For dates that are still a few months away you can get up to $50 off (per person/per day) for early bookings.

PHPUnit, Selenium... Sebastian's workshops in Australia

Thu, 24 Jul 2008 03:04:09 GMT

Sebastian Bergmann's slides of his presentation at OSCON this week. Quality Assurance in PHP Projects is a workshop by Sebastian (author of PHPUnit) in Melbourne (4-6 August) and Brisbane (11-12,14 August), on my invitation. This was triggered by meeting up with him earlier this year at the MySQL Conf in Santa Clara, and a discussion I had a month or so later with Jonathan Oxer of IVT.

I've been promoting various aspects of development process improvements through the keynote selection at OSDC 2007, the distributed revision control talk at the combined MySQL/PHP user groups in Brisbane by Ian Clatworthy (Canonical/Bazaar), and of course test-driven development for PHP is another key aspect. PHP is pupular (essentially it runs Web 2.0) but so easy that coders can get very slack. On a larger project, this can become disastrous. I don't believe that PHP is fundamentally bad; given the right development environment, it's fine. It's just that what's regarded as standard in the Java and Ruby/Rails world, has been nonexistent, hard to use, or optional in the PHP realm.

Anyway, both workshops are almost full now, just one or two places still available. No this is not marketing blah to just make you think it's popular ;-) If I say almost full, I do mean just that. I gave Jonathan a place for the Melbourne workshop, as it's his birthday coming up this Saturday (Unfortunately I can't make it to the party next week). Jonathan is a special friend to me and Open Query as we bounce around ideas and exchange interesting books. IVT also happened to be Open Query's first customer.

Speaking of gifts, consider donating something for little Ivan Nikitin's operation: http://www.mysql.com/about/help-ivan.html. This is not another random sad chainletter, this is a real small smurf, son of a MySQL support engineer. Ivan and his parents have now already arrived in Germany for the tests and procedures that Ivan needs, however additional financial help is still very important.

MySQL and Drizzle

Wed, 23 Jul 2008 01:54:55 GMT

Today Brian launched Drizzle, something he's been working on for a number of weeks now, together with some other Sun/MySQL people and quite a few active MySQL community members. I scribbled some quick info and my own perspective on About Drizzle, with links to the various currently available resources.

I think it's an interesting and worthwhile development, and I understand it has the okidoki from inside Sun; Monty also noted this. We'll have to see how that goes though. It amounts to an internal fork, doesn't it... Drizzle is not directly a MySQL replacement, but it does kinda wipe the floor with MySQL's current development process, roadmap, licensing and business model. I do commend Sun on allowing a project such as this to exist within.

If you're a coder, do get involved. Grab the bzr tree, read the FAQ, and get in with the refactoring effort. I've been perusing the tree for a bit now, and having previous experience with the MySQL tree, I find it much easier to find things now that some of the fluff is gone. This should make other things easier as well... development and bug fixing, build process. It's a good exercise.

OQ Poll: which MySQL version do you use for development?

Mon, 21 Jul 2008 23:53:13 GMT

Related to my post earlier today about the use of MySQL 5.0 and the Open Query course on upgrading and new features... I thought it might be good to do a broader poll of the MySQL population and see what version people are using for development (not deployment) right now!

Use this link to go directly to this OQ Poll page: Which MySQL version do you use for development?

Development is not quite the same as production. It can be good to use a newer version in development, as you have new features (and fewer old bugs ;-) at your disposal. Regardless of the release status of a version, by the time your application reaches production stage, you will be comfortable with that version so deploying on it will be fine.

Development is the ideal time for being progressive with choice of version, since at any other stage it can be risky or costly (in terms of QA). So, I'm very curious whether developers are indeed being progressive in this respect.
Please enter your vote! Thanks

Are you using MySQL 5.0 yet, or any of its features?

Mon, 21 Jul 2008 22:26:45 GMT

MySQL 5.1 is nearly GA (not debating that fun now ;-), MySQL 6.0 has interesting new magic such as Falcon and online backup. But I still have many clients that are on MySQL 4.0 or 4.1 both in production and development. Works fine, overall, but quite a few problems would be nonexistent if they were on 5.0, and by making use of some changed or new features various aspects could be faster.

The Open Query MySQL 5.0 Upgrade and Advanced Features course day is for this group of developers and DBAs, but also for users moving to MySQL and going straight for MySQL 5.0 (as they should, please don't *start* with earlier versions!).

We delve into the differences and new features. This includes

Views — did you know they're updatable, and have you considered using them for smart access control? (tables, columns, rows)
INFORMATION_SCHEMA — users at my local MySQL group were initially not so excited about this, until I showed how it can be used to optimise your database! Several people have applied the techniques in their business deployments now, and reported excellent improvements.
Stored Procedures — not wanting to shift all your app into the db, that's unwise; but you can use dynamic SQL and cursors (and infoschema) to perform some tasks more efficiently inside the server.
Stored Functions — MySQL has lots of built-in functions, but sometimes you just really need one that's missing. You can build your own! UDFs are faster but require more effort and maintenance.
Making use of the fact that a VARCHAR can now be nearly 64k (in bytes, character length will depend on the character set) you can get rid of some text-type columns and thus reduce temp-disk-table usage (=speed).

MySQL 5.0 also fixes many bugs from the 4.x series; this is particularly relevant if you use features like replication. Even with MySQL 5.0's less-than-perfect track record, it's generally preferably over MySQL 4.1, and users can move from 4.0 straight to 5.0, provided some precautions are taken.

The course day costs $575 + GST which includes a suitable lab environment (Windows or Ubuntu, or you can even bring your own laptop!), lunch and morning and afternoon tea, and a most enjoyable as well as educational day. Like all other Open Query modules you can just sign up for this one day, if you want. It's scheduled in various cities around Australia, and in Auckland New Zealand.

Never attribute to malice... MySQL Training in Australia

Mon, 21 Jul 2008 04:24:53 GMT

Sometimes you don't quite know what to think... last month MySQL (the company) had no courses in Australia listed in their schedule. They do a few courses a year, schedule them, and they disappear from the schedule when they're full - which, given one active MySQL salesperson since the start of the year, is happening.

Anyway, all of a sudden there were some new ones in the schedule (that's ok, must publish some time) but they're fairly short notice (about one month, rather than 2-3 months ahead), and one is on a similar topic as one of the Open Query courses, in a location that MySQL has not previously been to. And aside from the short notice, the state in question had its winter school holidays two weeks of the period between announcing and the course, so that makes the lead time even shorter.

Now we know, "never attribute to malice what can easily be attributed to ignorance/stupidity." Shall we leave it at that? What do you think...

But, I hear you ponder, MySQL does not actually have any trainers in Australia! That's right. Because MySQL does not wish to partner/contract Open Query for training courses (since I also do my own training materials/courses), they actually fly in a trainer from the US or Europe. As I understand it, they spent USD 5k earlier this year for flights from the US for a one-week training course. Not sure about the economics of that ;-)

site tracking phone spam

Mon, 21 Jul 2008 02:18:37 GMT

Good idea: WhoCallsMe?. Tracks nuisance calls (globally), of course only if there's a CallerID. Check on the site if the number is already in the system, read the comments, add your own. Simple, but useful.

ulimit, taskset for mysqld

Mon, 21 Jul 2008 00:09:17 GMT

my.cnf can take the open_files_limit=N option in the [mysqld_safe] block. This doesn't do anything to mysqld directly, instead the mysqld_safe script calls ulimit -n <N>

These days, we might want to limit mysqld to use a maximum of 4 cores (using taskset on Linux) so InnoDB performance doesn't degrade. Of course this is only relevant on machines with more than 4 cores (like boxes with 2 quadcore CPUs), but still, it's nice to not have to hack mysqld_safe for such changes, as it makes upgrades less fussy and keeps config info in the one place (my.cnf).

Peter and I were discussing this, and I suggested two new options for mysqld_safe, one for pre-mysqld and one for post-mysqld commands that need to be run. Calling ulimit would be a pre-mysqld task, but taskset is post.
The one thing is that taskset needs to be passed the PID, so we must provide a way to pass that through the config dynamically.

We know that MySQL wants to get rid of mysqld_safe in favour of the Instance Manager, but that's been a long time and it hasn't happened yet.

Thoughts/ideas welcome, before we make a patch for this.

Oracle Out - MySQL In?

Sun, 20 Jul 2008 23:13:54 GMT

Interesting comment spotted on Jonathan Schwartz' blog:

[...] I work for a major Fortune company, and we're in the process of putting Oracle on a "sunset" list of restricted vendors. No new applications are allowed on Oracle, the only approved vendors are Sun/MySQL and Microsoft/SQL Server. So I don't know how Sun did that, but if their objective was to provide competition for Oracle, it appears to have worked with my management...

I know that MySQL and Oracle happily co-exist in many companies, I often train highly skilled Oracle developers and DBAs about MySQL specifics.

Open Query training for MySQL in Canberra

Fri, 18 Jul 2008 00:03:17 GMT

Already scheduled for some time, but since it's a new location for MySQL training it's probably worth an extra mention. Ye that's right, MySQL itself has never held a training course in Canberra either (I know since I was the trainer ;-) except for some custom stuff. The DBA days are pretty soon but there are still some seats.

MySQL DBAs

Mon 4 Aug: MySQL Installation, Security and User Management ($475)
Tue 5 Aug: MySQL Backup and Recovery ($475)
Wed 6 Aug: MySQL InnoDB Performance Tuning ($575)

You can pick individual days if you want. For the later courses/dates you can score a $50/person/day discount by booking early!

MySQL HA

Melbourne

MySQL Developers

Mon 10 Nov: MySQL Optimisation by Design ($475)
Tue 11 Nov: MySQL Storage Engine Optimisation ($475)
Wed 12 Nov: MySQL 5.0 Upgrade and Advanced Features ($575)

India's SMS GupShup Has 3x The Usage Of Twitter And No Downtime

Fri, 11 Jul 2008 04:13:11 GMT

Just spotted this article by Anand Rajaraman about an Indian version of Twitter that uses mostly mobile SMS rather than web access, and is built on Java and MySQL. It handles many more users and messages than Twitter currently does, and seems to be doing ok.

Anyway, have a read, the usual obvious architecture choices for scalable database apps are in there, but it's always good to read how people have actually applied these things and made something real work - serves better as an example for others... ;-)

One free OSCON 2008 ticket on offer!

Tue, 08 Jul 2008 00:00:28 GMT

Do you want to go to OSCON but can't afford the entrance fee? I can help. Being on the program committee, I receive a freebie ticket to hand out. Simply answer this question (provided by Brian): Where does name of the TINA storage engine come from, and which engine is it actually when used in MySQL?

Be sure to either login to Livejournal when you reply, or have some other way to identify yourself - an anonymous comment can't be contacted, and thus you wouldn't be able to win the prize...

Interview with MySQL's Marten Mickos in WSJ

Mon, 07 Jul 2008 23:38:41 GMT

Interesting interview with Marten Mickos in the Wall Street Journal today. I think he describes the overall environment really well, although I don't quite agree that the innovation spirit is all through MySQL. That depends on where the people come from and their own interests, as well as the drivers of their department.

Also, the number of developers is not really 12M. The 12M people could actually be a huge QA department, and used to be exactly that, but they aren't now. This because new code now goes into the Enterprise branch, and first gets tested by the few thousand Enterprise customers, before being let loose on the wider community (the 12M). I'm sure the Enterprise customers are really pleased about this (ask Mark Callaghan)... it's completely upside down. AKA utterly borked.

Thanks to James McPherson for that link.

On the note of business models, see this: What could Open Office do with a business model?.

Sakila the Tequila Dolphin?

Mon, 07 Jul 2008 23:11:29 GMT

A few years back, MySQL (the company) has let its trademark application on the name Sakila expire. As you may (or may not) remember, around 2002 there was a grand naming competition for the new logo dolphin, and Sakila name won, over other interesting names like Finn, and corny blah like Flipper (I know this because I was the person wading through the spreadsheet with all the submissions, extracting a shortlist for Monty and David).
(By the way, MySQL docs team lead Stefan Hinz named his son Finn. I don't know if there's a connection ;-)

Anyway, a bit later but still a few years ago already, the name Sakila was registered by the Jose Cuervo company, of global tequila fame. You can easily see this at uspto.gov.
So, I'm just curious does anyone know of a Cuervo product that uses that name now, or is it a dormant trademark? Is there a relevance to the name sakila in Mexican Spanish? The name originates in Africa.... if there's no Mexican connection, chances are there's a MySQL fan in the Cuervo company!

Two 5.0 bugs in a day

Mon, 07 Jul 2008 12:46:06 GMT

I filed these two MySQL bugs today:

Bug#37910 Dropping unrelated index fails if fk on first col
Bug#37925 Select with comma join pretends a column does not exist

They originate with clients, and occur on recent 5.0 versions. No fatal stuff, but annoying. Both are of the fairly awkward variety and I'm curious why they still exist so late in the 5.0 series.

For instance, the second one looks like a parser bug, and I'd have figured someone tripping over this ages ago. It's actually a regression as it worked fine in 4.x, the testcase is directly from an app that was in production on a 4.x server, then upgraded to 5.0. Kaboom on this. Quite unexpected, and very annoying as going through the entire app changing the join syntax is a bit of a pest for the developers.

(update on this below, with assistance from Mark Leith - thanks Mark!):

Being aware of
- http://bugs.mysql.com/13551
- http://bugs.mysql.com/14817

and the information from the Docs (http://dev.mysql.com/doc/refman/5.0/en/join.html):

'However, the precedence of the comma operator is less than of INNER JOIN, CROSS JOIN, LEFT JOIN, and so on. If you mix comma joins with the other join types when there is a join condition, an error of the form Unknown column 'col_name' in 'on clause' may occur. Information about dealing with this problem is given later in this section.'
[...]
'Previously, the comma operator (,) and JOIN both had the same precedence, so the join expression t1, t2 JOIN t3 was interpreted as ((t1, t2) JOIN t3). Now JOIN has higher precedence, so the expression is interpreted as (t1, (t2 JOIN t3)). This change affects statements that use an ON clause, because that clause can refer only to columns in the operands of the join, and the change in precedence changes interpretation of what those operands are.'

That still leaves us with a unclear error reporting. Technically it may be correct based
on parser internals, but users should not need to care about such black magic.

Abuse of LEFT JOIN

Mon, 07 Jul 2008 09:56:49 GMT

I teach the exact differences and use of the different join types and keywords in my courses (specifically the Optimisation by Design day). A few tidbits:

Left and right joins are outer joins.
A,B is an inner join (equivalent to a cross join if you don't have a join condition).

But the point of this post... when looking at apps, I often see LEFT JOINs abused. That is, the query does not actually require a left join. The result might be ok, but that entirely depends on the data contained in the tables; no guarantee.

Apart from understanding joins, I've found that there's a very simple but effective test for whether or not to use a left join.... "if you can't think of a specific reason, you don't" ;-)

FORCE INDEX

Thu, 03 Jul 2008 11:36:22 GMT

SELECT ... FROM
tblProduct as prd FORCE INDEX(vProductName,iCategoryId,eProductType,vProductImage,fPrice,eProductStatus,dProductDateAdded)
LEFT OUTER JOIN (SELECT bidjoin.* FROM tblProduct as prdjoin, tblBid as bidjoin WHERE bidjoin.iProductId = prdjoin.iProductId) bid ON prd.iProductId = bid.iProductId
, tblCategory as cat FORCE INDEX(vCategoryName,eCategoryStatus,dCategoryDateAdded) 
, tblAuctionProduct as aucprd FORCE INDEX(iRequiredBids,iProductId,eAuctionType,iFreeBidLevel,fAdminFee,dDateClosed)
, tblPurchaseProduct as purprd FORCE INDEX(dProductCloseDate,iProductId,iTotalInventory,dLastUpdated)

I know, unreadable. But the main issue today is the FORCE INDEX. This is just one query (real, with permission - I just adjusted some identifiers) from an app that is litterally full of queries using FORCE INDEX. So what does FORCE INDEX do? It forces the MySQL optimiser to use the specified index (or choose from one of the specified, if multiple), even if it reckons it's not the best choice. Likewise there's an IGNORE INDEX modifier that denies the optimiser the choice for using a specified index in a particular query.

I generally hold that these modifiers should only be used for testing things and tracking down optimiser issues (not that common these days, but 3.23 had plenty that 4.0 fixed). This because when you hardcode them into an app, perhaps fixing a real problem you see *now*, your data will still change over time and so the optimiser needs the freedom to change its choices too. It might be ok for a week, a month, or even a year, but at some point it's going to cost.

The above query takes about 5 times as long compared to the same query without the FORCE modifiers. Considering the app is chockers with it, I'm guessing that it might have been coded by someone who didn't know that MySQL actually has an optimiser at all. Who can say... anyway it serves here as an extreme example.

The take-away message is: use these modifiers with care, and be very hesitant ever adding it to production code. There's generally always a better way to resolve whatever problem you might be observing.

OSS-based appliances: Cybersource/datasafe

Tue, 01 Jul 2008 00:47:45 GMT

I'm pretty happy with my 1TB Apple Time Capsule. Bought it while at the MySQL Conf in April, and it does the right thing for my situation.

Con Zymaris and his crowd at Cybersource in Melbourne made something like this ages ago, aimed at small businesses: Cybersource/DATASAFE. That's a pretty neat solution, and an excellent example of how an OSS-based solution can be deployed in a business, regardless of what other technologies might already exist on the premises. The box will work just fine in a Windows environment.

Always focus on the solution (and what practical needs it solves for the client), not the technology (or the philosophy) - with a happy customer, you'll get plenty of opportunity (over time) to discuss what OSS is really about, and you're likely to find a very willing ear at that point.

Training schedule readability

Mon, 30 Jun 2008 23:36:43 GMT

A tricky meta-question. I've been experimenting with ways to display an overview of training days (or days grouped by audience/topics, like Developers or DBAs), locations, and schedule.

Product	Audience	BNE	SYD	CBR	MEL	ADL	AKL
MySQL	Dev	Sep	Aug		Sep	Aug
MySQL	DBA	Sep	Jul	Aug	Aug	Aug	Jul
MySQL	HA		Aug
PostgreSQL	DBA		Jul				Jul
PHP QA	Dev	Aug			Aug

With the number of locations I have now, it's easy to run out of space horizontally. And vertically can't make the list too long either, otherwise the nice simple overview is lost. It's a difficult problem and I haven't fully resolved it. Displaying the locations vertically and the topics horizontally doesn't seem to work either. Different pages on the Open Query web site now display things differently depending on context, and I hope to learn more from that experiment. The above is an example of this. Compact, but I'm not sure it conveys sufficient information. Within each group, people are able to sign up for individual course days. It's work in progress I suppose. Suggestions welcome!

Finding useless indexes

Mon, 30 Jun 2008 02:47:37 GMT

I'll say beforehand that the following is not very clean - for a number of reasons. I'll discuss some issues after the query. The query returns all indexes in a db where the cardinality is lower than 30% of the rows, thus making it unlikely that the server will ever use that index.

SELECT s.table_name,
       concat(s.index_name,'(',group_concat(s.column_name order by s.seq_in_index),')') as idx,
       GROUP_CONCAT(s.cardinality ORDER BY s.seq_in_index) AS card,
       t.table_rows
  FROM information_schema.tables t
  JOIN information_schema.statistics s USING (table_schema,table_name)
 WHERE t.table_schema='dbname'
   AND t.table_rows > 1000
   AND s.non_unique
 GROUP BY s.table_name,s.index_name
HAVING (card + 0) < (t.table_rows / 3);

Let's discuss...
The number of rows in a table used here will be accurate for MyISAM; for InnoDB it's essentiall a rough guess.
Cardinality in this context is the number of distinct values in a column. This statistic needs to be updated using ANALYZE TABLE, otherwise it might either not be available or just outdated.
Since there can be composite indexes (index on a,b or more columns, rather than just one column), the cardinality is per column and in the output we show this comma-separated; of course for anything but the first, it's relative to the previous columns. The HAVING clause just ends up using the cardinality of the first column, which statistically is not really correct. If someone has a bright idea on how to grab or calculate a sensible cardinality figure for composite indexes, please do comment.
And yes, some selected columns (like t.table_rows) fall outside the grouping, however they remain the same so that's "ok" for this quick hack and MySQL allows this unless you have sql_mode=ONLY_FULL_GROUP_BY enabled.

The fact that the calculated cardinality figure (for composite indexes) is dodgy may not actually be relevant for the purpose of this query. The point of the query is that generally, the server will not use an index if it needs to look at >30% of the rows anyway. That's not *actually* how it works inside the server (as Peter Zaitsev can explain in great detail ;-) but as an easy rule-of-thumb it's close enough to reality. That's not the whole story, because if all the columns referenced in a query are in the index you have a so-called covering index, and the server knows that scanning an index is quicker than scanning the table. So in that case, an index might still make sense, but this query can't know about that case.
Also, for very small tables, the server will prefer a table scan anyway; to weed out the worst of this, only tables with >1000 rows are shown in the output. Again, a rather crude filter with plenty of flaws. But again, it may suffice for this purpose.

Normally, what you'd do is pick out the obvious indexes on yes/no, male/female, active/deleted style columns. They're easy to spot. But if you run the above query, much more may show up. Do unused indexes hurt? Well, they slow down INSERT/UPDATE/DELETE and ALTER TABLE statements. So this query can be useful as a rough filter, then human brains can be used to decide which indexes to keep and which ones to get rid of.

mysql cmdline tricks: output control (\G, pager, tee)

Fri, 27 Jun 2008 05:13:55 GMT

Compared to SQL*Plus (Oracle's cmdline client) which can as I understand produce complete paginated reports, the 'mysql' tool has fairly simple output formatting. I'm not generally fussed about that, as there are perfectly good other tools do make snazzy reports with. Some very cross platform (like web based), output PDF, not a problem. You can even write your own quickly these days.

But the cmdline tool has its use. I always teach my students the basics of it, and insist that some exercises are done using this tool. Familiarity (which involves some practice) helps there. Why? If all else fails, you generally do have mysql available on a machine. It's not hindered by remote ssh logins, or anything else. And, of course, for quickly checking something. And that's where the proficiency helps again. So, normally, you get this type of ASCII table:

mysql> select "Hello, world!" AS foo;
 --------------- 
| foo           |
 --------------- 
| Hello, world! | 
 ---------------

which is nice. But for some commands, this type of formatting would make the output unreadable. Key examples are SHOW CREATE TABLE, SHOW CREATE TRIGGER, and SHOW WARNINGS (particularly after EXPLAIN). Now check this out:

mysql> select "Hello, world!" AS foo \G
*************************** 1. row ***************************
foo: Hello, world!

If, instead of a semicolon you specify \G, the output is formatted vertically. A separator line above each row, and then each column prefixed by its name/alias followed by the value of that column within the row. It's much more readable for cases where you get a one-row result set with only a few columns, at least one of which might be longer than the width of your console.

The above also teaches us another lesson: the semicolon is not actually part of the SQL command. It's merely a delimiter that tells the client code to send the query to the server. If you're in a programming or scripting language using the MySQL API, you wouldn't include it in the query string.

A related trick is to use the "pager" command. If you specify pager less on the mysql command prompt, the output will thereafter be run through less. So then you can scroll through a larger result set, search it, etc. (see man less).

Final one for this post, there's "tee". This command followed by a filename (or program/pipe!) lets you log everything you do, queries (what you type) as well as output. Very handy, particularly during training and testing, or when you need to make a bugreport or support request.

Website usability vs performance - Measly Mouse revisited

Fri, 27 Jun 2008 01:26:54 GMT

Peter Zaitsev wrote an interesting item on front-end performance of a website.

I've always tried to look at the front-end from the user perspective, rather than purely technical. Once you weed out what's not really necessary for the user, and also deal with issues like "how important is it that this number is live", you generally look at a fairly different site already ;-)

Before my time at MySQL, I wrote a little gizmo called Measly Mouse which leads a modest but still active life. When reading on from here, please remember it was designed in 2001 and hasn't really been changed since.

Measly Mouse retrieves a page and deals with redirects, CSS and other includes like images, and tries to apply some basic metric to see how sensible the page is. Basic usability testing shows that people cannot choose between more than about 7 or so items. So you can imagine how the brain desperately fails to deal with most websites (website creators often feel everything is so important it must be on the front page), or through training filters out most things than don't relate directly to what the person is looking for. The other key factor is size. The bigger the page, and the more includes, the longer it takes. That's annoying. So the Measly scoring formula is as follows: BYTES + ((REQUESTS + LINKS) * 1024) where:

BYTES - is the total page content including stylesheets, images, etc;
REQUESTS - is the total number of requests required (including redirects) to get all page content;
LINKS - is the total number of clickable items on the page, including forms fields.

Naturally all pages have some includes and some links, it's just a matter of finding balance to keep the site usable. Some sites use so many redirects.... nutty.

At the time there was quite a lot of debate on the simple methodology and the owners of some sites got pretty upset when someone ran their front page through Measly and it ended up in the top 10 ;-)
I still reckon the concept has merit though... please do make your own judgement and feel free to comment.
You may find some aggregating (like mailing list archive) sites in the top 10... I personally take those entries less serious, because they're generally focused on a niche (geek) group which does not conform to a general user profile. Still, it's quite possible their user interface could be improved!

As to how the tool works... it actually parses the pages in PHP using regexes (again remember the time it was built). Although it still works reasonably well, could be vastly improved now and catch more of a modern page.
But what would be really great, is if someone would care turn Measly Mouse into a Firefox plugin. Inside Firefox you have clean access to a page, so the analysis becomes extremely easy. For any page, the plugin could calculate the Measly Mouse (MM?) score, and perhaps optionally submit it to a central location. Who would like to pick this up?

Some impressive hw hacking of Eee PC

Fri, 27 Jun 2008 00:18:10 GMT

This fellow is pretty amazing: http://beta.ivancover.com/wiki/index.php/Eee_PC_Internal_Upgrades. To the already tiny Eee PC, he added (internally!):

USB hub
GPS with antenna
Bluetooth
Card reader w/ additioal SSD
Power switch (10 dip) for switching all extra foo on/off
Wifi upgrade 802.11n
FM transmitter
Modem (admittedly there's design space for that)
Touch screen
Temperature sensor
Heatsink

That's pretty cool...

Got my copy! High Performance MySQL (2nd edition)

Thu, 26 Jun 2008 02:05:31 GMT

Grand, I got my personal copy from the courier today!
Many people kept asking me over recent months "when is it coming, when can I get it"... so now you can. Officially out, and available.

Click on the book pic if you want to grab your copy via Amazon (which is probably one of the cheapest sources, even if you're overseas).

I'm also looking at acquiring a bundle of them through the author arrangement with O'Reilly, to give away to Open Query training students - well, for the courses related to MySQL at least ;-)

The grey art of eBay - the feedback process borked

Sat, 21 Jun 2008 05:01:47 GMT

Since a few months, sellers can no longer provide neutral or negative feedback to buyers. Not sure if I completely agree with this as I reckon buyers can be quite dodgy and how else to capture that. But the objective was to weed out retaliatory feedback. I.e. buyer has a bad experience, provides neutral or negative feedback, and in return for his honesty gets back similar feedback. So that's no longer possible. I suppose that's good.

I actually had such an experience at the beginning of the year, before the new policy. That one (in this case neutral) retaliatory feedback seriously impacts my rating to just over 90%. Not much of an issue as a buyer, but if I want to sell something, potential buyers might regard me as dodgy just on the basis of that number. Of course they could delve into the exact feedback comment and figure out what's been going on to some degree, but who bothers with such detail while browsing eBay for goodies?

The formula for the rating is positive / (positive+neutral+negative), and all that limited to the last 12 months. Let's take my simple case where I get one neutral. For 11 transactions over the last year, you get 10/(10+1) = 90%. Ok so I don't buy stuff on eBay all the time. Most people don't. But let's double the number of transactions and see how the formula works out: 20/(20+1) = 95%. Still not good for seller karma. Anyway, you get the picture; this kind of thing is damaging, and you'd really have to go on a buying spree to make up for it. With different buyers or on different weeks, otherwise it's counted as a duplicate and disregarded. And heaven beware if any of the many different sellers also decides to give you a neutral rating!

Ok ok, so the new eBay policy prevents this exact scenario from occurring again. But consider the mechanism if you're a seller. Buyers *will* provide you with feedback, and using the same formula a single malicious buyer can destroy your reputation. You suddenly have to sell a lot more to make up for it, except you may not be able to, because of your rating! In a nutshell, unless you're a powerseller moving dozens of items a month, you're pretty much an open target. If you, like I, sell the odd item once in a while, you don't have a chance.

But hang on, isn't there a policy that allows you to get feedback removed? Yes, but only in very specific cases. For this, the comment would have to be defamatory, referencing an unrelated item/transaction, a policy file number, a URL or other contact details, stuff like that. Also, a clearly negative comment on a positive rating qualifies. But not the other way round.
So, the loophole is a gaping one, and that is: you just scribble something nonsensical or at least meaningless; that way it won't qualify for removal. So you can just give a seller a neutral (just as damaging as negative, as far as the formula is concerned!) with a comment of "blah de blah" and that does the trick.

Someone please tell me that I went wrong with my logic - and where!