Monday, August 16, 2010

MySQL and AppArmor on Ubuntu

This is just a quick post that I hope will save some people some headache when they try to customize their MySQL setup on Ubuntu. I've spent some quality time with this problem over the weekend. I tried in vain for hours to have MySQL read its configuration files from a non-default location on an Ubuntu 9.04 server, only to figure out that it was all AppArmor's fault.

My ultimate goal was to run multiple instances of MySQL on the same host. In the past I achieved this with MySQL Sandbox, but this time I wanted to use MySQL installed from Debian packages and not from a tarball of the binary distribution, and MySQL Sandbox has some issues with that.

Here's what I did: I copied /etc/mysql to /etc/mysql0, then I edited /etc/mysql0/my.cnf and modified the location of the socket file, the pid file and the datadir to non-default locations. Then I tried to run:

/usr/bin/mysqld_safe --defaults-file=/etc/mysql0/my.cnf

At this point, /var/log/daemon.log showed this error:

mysqld[25133]: Could not open required defaults file: /etc/mysql0/my.cnf
mysqld[25133]: Fatal error in defaults handling. Program aborted

It took me as I said a few hours trying all kinds of crazy things until I noticed lines like these in /var/log/syslog:

kernel: [18593519.090601] type=1503 audit(1281847667.413:22): operation="inode_permission" requested_mask="::r" denied_mask="::r" fsuid=0 name="/etc/mysql0/my.cnf"
 pid=4884 profile="/usr/sbin/mysqld"

This made me realize it's AppArmor preventing mysqld from opening non-default files. I don't need AppArmor on my servers, so I just stopped it with 'service apparmor stop' and chkconfig-ed it off....at which point every customization I had started to work perfectly.

At least 2 lessons:

1) when you see mysterious, hair-pulling errors, check security-related processes on your server: iptables, AppArmor, SELinux etc.

2) check all log files in /var/log -- I was focused on daemon.log and didn't notice the errors in syslog quickly enough

Google didn't help when I searched for "mysqld Could not open required defaults file". I couldn't find any reference to AppArmor, only to file permissions.

6 comments:

Eusebiu Blindu said...

Delete the log files when mysql isn't starting---This helped me a lot of times

Grig Gheorghiu said...

In this case it had nothing to do with log files though.

bpk said...

Where was this blog post about a month ago when I had the same problem? I think it took me a few hours of Googling to realize it was Ubuntu+appArmor related and not an error in my mysql conf.

I was trying to store my data in a different dir. I found the apparmor config files in

/etc/apparmor.d/usr.sbin.mysqld

and added my data dir to this list.

Later on i decided the more appropriate approach was just to remove apparmor.

sudo apt-get remove apparmor

I hope your blog post gets enough hits to land high in Google page rankings.

Yaniv Aknin said...

As I was reading the post, the first thing that came to my mind is strace(1). Don't be afraid to strace even a 'large' application, like a database, especially since you know precisely what you're looking for.

As soon as you would have seen whatever system call fail for mysql as it was trying to open that file with something like EPERM (or whatever AppArmor uses to signify failure), you'd be well on your way to realizing what's wrong.

Grig Gheorghiu said...

Thanks for the tip, Yaniv! You're a pro ;-)

Anonymous said...

I also wasted a lot of time wondering what the h""l is wrong with my server...

Apparmor made me mad :)

Thanks

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...