My ultimate goal was to run multiple instances of MySQL on the same host. In the past I achieved this with MySQL Sandbox, but this time I wanted to use MySQL installed from Debian packages and not from a tarball of the binary distribution, and MySQL Sandbox has some issues with that.
Here's what I did: I copied /etc/mysql to /etc/mysql0, then I edited /etc/mysql0/my.cnf and modified the location of the socket file, the pid file and the datadir to non-default locations. Then I tried to run:
/usr/bin/mysqld_safe --defaults-file=/etc/mysql0/my.cnf
At this point, /var/log/daemon.log showed this error:
mysqld[25133]: Could not open required defaults file: /etc/mysql0/my.cnf
mysqld[25133]: Fatal error in defaults handling. Program aborted
It took me as I said a few hours trying all kinds of crazy things until I noticed lines like these in /var/log/syslog:
kernel: [18593519.090601] type=1503 audit(1281847667.413:22): operation="inode_permission" requested_mask="::r" denied_mask="::r" fsuid=0 name="/etc/mysql0/my.cnf"
pid=4884 profile="/usr/sbin/mysqld"
This made me realize it's AppArmor preventing mysqld from opening non-default files. I don't need AppArmor on my servers, so I just stopped it with 'service apparmor stop' and chkconfig-ed it off....at which point every customization I had started to work perfectly.
At least 2 lessons:
1) when you see mysterious, hair-pulling errors, check security-related processes on your server: iptables, AppArmor, SELinux etc.
2) check all log files in /var/log -- I was focused on daemon.log and didn't notice the errors in syslog quickly enough
Google didn't help when I searched for "mysqld Could not open required defaults file". I couldn't find any reference to AppArmor, only to file permissions.
6 comments:
Delete the log files when mysql isn't starting---This helped me a lot of times
In this case it had nothing to do with log files though.
Where was this blog post about a month ago when I had the same problem? I think it took me a few hours of Googling to realize it was Ubuntu+appArmor related and not an error in my mysql conf.
I was trying to store my data in a different dir. I found the apparmor config files in
/etc/apparmor.d/usr.sbin.mysqld
and added my data dir to this list.
Later on i decided the more appropriate approach was just to remove apparmor.
sudo apt-get remove apparmor
I hope your blog post gets enough hits to land high in Google page rankings.
As I was reading the post, the first thing that came to my mind is strace(1). Don't be afraid to strace even a 'large' application, like a database, especially since you know precisely what you're looking for.
As soon as you would have seen whatever system call fail for mysql as it was trying to open that file with something like EPERM (or whatever AppArmor uses to signify failure), you'd be well on your way to realizing what's wrong.
Thanks for the tip, Yaniv! You're a pro ;-)
I also wasted a lot of time wondering what the h""l is wrong with my server...
Apparmor made me mad :)
Thanks
Post a Comment