The Unix Side of macOS

DONG Yuxuan @ Feb 01, 2020


Preface

I have been using macOS as my desktop system since 2015 when ZHOU Jamin introduced his MacBook Air to me. I choose it for 3 reasons:

The last two reasons are on the software side. However, if you were a Unix hacker who decides to use macOS for the two reasons, you may find Apple “thinks different”. If you check /etc/passwd you may not find your username. useradd doesn’t even exist. No default package manager, so you search online and find most people use Homebrew as their package manager so you install it and use it for a long time. Until one day you decided to create another privileged account and use the new account to install packages by Homebrew. Homebrew suddenly fails. You investigated the reason and it shocked you. Homebrew did some evil things to your /usr/local which a Unix hacker can’t endure. As the new account can’t use Homebrew, you want to install programs from compiling but Homebrew also takes /usr/local as its home so you can’t follow FHS. If you develop with macOS, there’re even more questions you want to ask. Where’re header files? Where are libraries? What are /private, /Library and other mac-specified directories?

All these problems I have also met. That’s why I decided to write this article to note the difference between macOS and common Unix. Wish it helpful.

Users and Groups

macOS manages users and groups by directory services instead of /etc/passwd, /etc/group. The directory services is yet another simple concept that becomes shrouded in terminology by theorists. Consider a local network, for example, the one in your office, there’re many computers in the network and each has its own files, accounts, etc.. A directory service organizes these distributed resources into one single index and makes them look like a directory hierarchy. For example, /Hosts/PC1 represents a host in the network and /Users/Smith represents a user in the network.1

macOS uses the dscl command to interact with directory services. To list all users on the system, we could run the following.

dscl . -list /Users

The first argument sets which datasource we want to retrieve from. A dot means the local datasource.

# List groups
dscl . -list /Groups

The -list option lets dscl print subitems of a resource. Subitems of /Users are all users. Subitems of /Groups are all groups.

If we want to check the information about the resource instead of listing subitems we should use -read. For example, we could check the information of the user Smith like the following.

# Retrieve the information of Smith
dscl . -read /Users/Smith

-read prints keys and values of the specified resource. We could also specify the keys we want.

# Get the primary group ID and default shell of Smith
dscl . -read /Users/Smith PrimaryGroupID UserShell

To add a resource, we use dscl . -create. The following command adds a new user belson.

sudo dscl . -create /Users/belson

It seems ok but it’s actually not. If we try to set a password for belson using sudo passwd belson, passwd will tell us the user doesn’t exist. Because dscl just adds a record to the data source but is doesn’t check the integrity. It doesn’t even assign a UID for belson. We must assign a UID and a primary group for belson after adding it to the datasource by ourselves.

sudo dscl . -create /Users/belson UniqueID 511	# UID 511 for example
sudo dscl . -create /Users/belson PrimaryGroupID 20	# The staff group

Now, belson is a Unix user and you can set a password for him using passwd. However, you still can’t login as belson for we don’t assign him a shell.

sudo dscl . -create /Users/belson UserShell /bin/bash

Well we can login as belson now but he doesn’t have a home directory. We must create the directory and specify it with dscl.

sudo dscl . -create /Users/belson NFSHomeDirectory /Users/belson
sudo mkdir /User/belson
sudo chmod -R belson:staff /User/belson

Sounds hell right? But it’s not over. Run dscl . -read /Groups/staff and you will find that neither the GroupMembers nor the GroupMembership attribute has any information about belson. We have specified his primary group to staff but as mentioned above, dscl doesn’t check the integrity. Bidirectional relationships must be maintained manually. This is the real hell.

To solves the user-group relationship problem, we could use dseditgroup. The following command adds belson to the staff group and it takes care of the integrity.2

sudo dseditgroup -o edit -a belson -t user staff

We should make a summary here. To create a user belson in the group staff, we need:

sudo dscl . -create /Users/belson
sudo dscl . -create /Users/belson UniqueID 511	# UID 511 for example
sudo dscl . -create /Users/belson PrimaryGroupID 20	# The staff group
sudo dscl . -create /Users/belson UserShell /bin/bash
sudo dscl . -create /Users/belson NFSHomeDirectory /Users/belson
sudo dseditgroup -o edit -a belson -t user staff
sudo mkdir /User/belson
sudo chmod -R belson:staff /User/belson

It is painful for a system administrator to type so many commands just to create a new user. We must miss adduser and addgroup in Debian so much. Since Mac OS X 10.10, the sysadminctl command is provided. It’s a high-level interface of dscl. Type sysadminctl in the terminal to see its usage3.

Usage: sysadminctl
	-deleteUser <user name> [-secure || -keepHome] (interactive || -adminUser <administrator user name> -adminPassword <administrator password>)
	-newPassword <new password> -oldPassword <old password> [-passwordHint <password hint>]
	-resetPasswordFor <local user name> -newPassword <new password> [-passwordHint <password hint>] (interactive] || -adminUser <administrator user name> -adminPassword <administrator password>)
	-addUser <user name> [-fullName <full name>] [-UID <user ID>] [-shell <path to shell>] [-password <user password>] [-hint <user hint>] [-home <full path to home>] [-admin] [-picture <full path to user image>] (interactive] || -adminUser <administrator user name> -adminPassword <administrator password>)
	-secureTokenStatus <user name>
	-secureTokenOn <user name> -password <password> (interactive || -adminUser <administrator user name> -adminPassword <administrator password>)
	-secureTokenOff <user name> -password <password> (interactive || -adminUser <administrator user name> -adminPassword <administrator password>)
	-guestAccount <on || off || status>
	-afpGuestAccess <on || off || status>
	-smbGuestAccess <on || off || status>
	-automaticTime <on || off || status>
	-filesystem status
	-screenLock <immediate || off> -password <password>

Pass '-' instead of password in commands above to request prompt.
'-adminPassword' used mostly for scripted operation. Use '-' or 'interactive' to get the authentication string interactivly. This preferred for security reasons

We use sysadminctl to delete belson first.

sudo sysadminctl -deleteUser belson -secure

Using dscl to check the directory service, we could find that everything we did is removed.

Now, we add belson again with sysadminctl:

sudo sysadminctl -addUser belson
sudo dseditgroup -o edit -a belson -t user staff

Two commands finish the job.

Filesystem Hierarchy

If you check the root directory, you will see some traditional Unix things like /usr, /var, and /etc. You may also find mac-specified things like /Library /Applications, and /cores.

A list of mac-specified directories with their purposes in the root directory is here4.

Directory Purpose
/private Contains the tmp, var, etc, directories. /tmp, /var, /etc are symbolic links to these subdirectories
/Library Contains support files for locally installed applications, among other things.
/System Contains a subdirectory, Library, that holds support files for the system and system applications, among other things.
/Network Contains network-mounted Application, Library, and Users directories, as well as a Servers directory that contains directories mounted by the auto mount daemon.
/Users Contains home directories for the users on the system. The root user’s home directory is /var/root (actually /private/var/root).
/Volumes Contains all visible mounted filesystems, including removable media and mounted disk images.

We could make a list to compare some of them with Linux.

macOS Linux
/System/Library /lib
/Library /usr/lib, /usr/local/lib
/Users /home
/Volumes /mnt, /media

You may be confused here for the differnce between /Library and /usr/lib. Which directory should we put libraries into?

/Library is for macOS-specified softwares, like the data of things you download from the App Store, and BSD programs shipping with macOS, like modules of pre-installed Perl and of pre-installed Python.

/usr/lib and /usr/local/lib are for Unix programs installed by yourself.

Booting

In modern Linux, the first program launched by the kernel is systemd. In macOS it’s launchd5.

launchd manages two types of services, LaunchDaemon and LaunchAgent. LaunchDaemon will be launched after the system is booted. LaunchAgent will be launched after a user is logged-in.

If a program needs to be launched after the system is booted, a .plist file is required to be placed into /Library/LaunchDaemons or /System/LaunchDaemons.

If a program needs to be launched after a user logged-in, a .plist file is required to be placed into /Library/LaunchAgents or /System/LaunchAgents, or ~/Library/LaunchAgents of the user.

The launchctl command is used to control services of launchd.

# to enable sshd
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

# to disenable sshd
sudo launchctl unload -w /System/Library/LaunchDaemons/ssh.plist

Development

You bought a Mac for development. People told you that you need to do the following two things:

You finished them, opened the terminal and found everything is familiar. You decided to write a classic Hello World with C to celebrate.

#include <stdio.h>

int main(void)
{
	printf("%s\n", "hello, world");
	return 0;
}

If you’re not lucky, especially if you use a 3rd party compiler, the compiler may tell you that stdio.h can’t be found.

No such fundamental file? You want to check /usr/include and you will find that even /usr/include doesn’t exist.

In newer Xcode like Xcode 10, header files are put into the package of Xcode. You can run xcrun --show-sdk-path to print the path6.

APX16:~ $ xcrun --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

Then you can use -isysroot to specify the path, for example:

gcc -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk hello.c

Compliers ship with macOS will care about this for you.

Package Manager

The first package installed to the system is usually a package manager. However, macOS has no default package manager. If you search online, you will find that Homebrew is the most popular package manager for macOS.

Homebrew is good for many people but not for one who wants things to go more like common Unix systems.

Homebrew can’t run as root. If you want to install python, you run brew install python instead of sudo brew install python. Homebrew installs packages to /usr/local. To achieve its goal about sudo-free, Homebrew changes the owner of all files and subdirectories under /usr/local to the user who installed Homebrew.

This brings security issues. Considering a malware that wants to fake a gcc to infect all the code you compile. Without what Homebrew does, it must be run as root to achieve its goal. But now, even it’s run as a normal user, it can create /usr/local/bin/gcc.

Besides the security issue, if you share the Mac with other people or you use it as a server, Homebrew can bring another issue. Two administrators both want to use Homebrew to install packages but /usr/local/bin can belong to only one of them, so another administrator will fail. If he or she tries to use Homebrew with sudo, he or she will also fail. Because Homebrew will check that if it’s run as root. If it is, Homebrew will refuse to work unless Homebrew itself belongs to root. OK, it seems the way to use Homebrew on a shared system is to install Homebrew by root and use it with sudo. However, dare we give the root privilege to a program that begs us not to? So, what could we do? I’ll quote the document of Homebrew here:

If you need to run Homebrew in a multi-user environment, consider creating a separate user account especially for use of Homebrew.

This means, every time we want to use Homebrew, we must switch the account first.

$ sudo -u brewuser brew install python

That’s too trouble. But if you do love Homebrew and you want to use it in a Unix way, this is it.

If you’re not a fan of Homebrew, there’re other package managers. MacPorts is one of them and it may be more welcome by Unix hackers. It requires sudo to install packages and installs them to /opt. Thus you can install packages that aren’t in the package manager to /usr/local.

References

  1. Andrew. Managing users and groups from the OS X terminal 

  2. How to add user to a group from Mac OS X command line? 

  3. Charles S. Edge. Using sysadminctl on macOS 

  4. Jepson; Rothman; Rosen. Mac OS X for Unix Geeks. 

  5. Apple Inc. Creating Launch Daemons and Agents 

  6. Apple Inc. Xcode 10 Release Notes #3035624