Best Practices for Linux Administration

By | 2013/04/14

Here are best practices for Linux Administration. By following these, one can be sure that a solid foundation is built which any IT staff present and future can easily follow.

1. Never login as root

The root password should be a 40 or more character string which is written down and kept secret and never used. For everyday administration, make use of a regular user account with sudo which allows for many features such as:

  • Auditing. Each sudo command is logged to audit responsibility. Otherwise if everyone uses the root password it is difficult to pin-down who made what change, and when.
  • Escalated privilege terminal timeout. Terminals have a buffer (default 5 minutes) which sudo ability then expires. This helps keep root terminals from sitting around open.
  • Avoids self-inflicted damage. Accidentally typing a single command in a root prompt could have irreversible damages. sudo provides a bit of protection against yourself. 🙂
  • Better ssh security. Use of sudo avoids having to allow root access via ssh to a machine.

2. Use the package manager or other vendor provided repository to install and manage software

Avoid pulling down a copy of software package X from sourceforge or other outlets. First determine if the Linux distro repository or a trusted vendor repository has software needed. This allows for less problems in operating system updates, package conflicts, as well as knowing that the repository provided package has been through some general QA and bugs to become ready for use in the Linux distro.

If one must compile software, take great research in determining dependent libraries required and build and produce an rpm or deb to easily install and uninstall with the package manager.

Debian has a ton of software. If a particular package is not available in Red Hat or EPEL but _is_ available and supported in Debian, one should consider the time, administration, and stability savings of just running Debian for that particular system or software choice.

Also the less custom software on a system, the easier it is to maintain with regular vendor patches and updates which will make a more reliable trouble-free system saving time and effort in the long run.

3. root mail – read it!

Linux by default sends errors, warnings, and other problems the system is having to root@localhost. It is critical to monitor and pay attention to root mail. Best practices are to send all root mail to an internal mailing list. Then, all Linux admins subscribe to that email list to receive the messages.

One example could be that a cron job has been failing for 5 months and the only way to have known this would be to check root’s mail for the cron error.

4. Monitor logs like a hawk. Use a log server.

Logging in Linux is a great feature and largely the source of how to check interactivity of a service, users, or other system messages. At a bare minimum, the app logwatch (included in all Linux distros) should be configured and set to run daily. Logwatch highlights problems or errors noted in all logs for the past 24 hours. Logwatch may be ok for a small handful of servers, but gets somewhat out of hand for more than say, six servers. For larger infrastructures, make use of rsyslog which has the ability to send logs to mysql in a central log server. This rsyslog server can be fit with various web interfaces to make log viewing across a large amount of servers easy.

If your CTO asks ‘How many times did this IP hit all of our servers at 5pm yesterday?’ that tasks would be almost impossible if not take your entire day without a central log server. However with a log server, one can simply search for that date and identify the information instantly.

5. Apply security updates regularly

Have a monthly maintenance window that is documented and on the calendar to apply security updates system-wide. For example in the Windows world, the typical patch window is anytime in the the Friday – Sunday weekend after patch Tuesday. However for Linux, security patches are made available all of the time from the Linux distro vendor. At the very minimum apply these updates once a month at a designated time determined by your IT staff.

The risk of not updating is that then a system is left in a state which after years is unable to be updated to a supported version and a more costly, drastic change is required which typically requires more downtime. Also the large amount of risk from vulnerabilities continue to mount on a system or service that is not maintained. The risk is both to the users and data of that system, as well as the entire network if a compromise occurs. This can generally be avoided if regular updates and maintenance are provided.

6. Use standard Linux directories

Each Linux vendor or distro has a particular layout of directory structure. It is best to follow this structure and not invent your own. For example:

  • Debian/Ubuntu: install websites and web applications in /var/www/
  • CentOS/Red Hat: install websites and web applications in /var/www/html/
  • User created scripts should only reside in /usr/local/bin/ or /usr/local/sbin/ . Do not place scripts or other content in the vendor directories /usr/bin/ or /usr/sbin/ both for security reasons as well as if an OS re-install is needed on those directories.
  • Do not create new random directories which are unknown to any others.

The more uniform and standard a system is, the easier it is for other staff to administer, to fix when the operating system is broken, as well as to keep the Linux security model in place.

OK! I could go on but that should be enough for starters! 🙂 Let me know if you would like to see more articles like this. Thanks,