Joel's Writings

The annoying double “is”

Joel — Mon, 11 Oct 2010 07:37:13 +0000

Since I arrived in the United States, I heard a lot of people using a phrase that I always found very awkward. People tend to say things like:

The problem is is that the economy is not in good shape.

I’m not really trying to talk about the economy here, it was just an example. So what’s the problem in that sentence? The repetition of the word “is”. I don’t really know why, but a lot of people use that phrase around here. And it’s not just with sentences that start with “The problem is”: it often happens with sentences that start with “The thing is”, or “The question is”.

I heard this phrase so often that I almost thought it was proper English…

Fortunately, a few weeks ago, I listened to an episode of the Grammar Girl podcast about double words, which explains that this phrase is actually incorrect. She even has a great explanation for why people repeat the word “is” in such sentences:

We probably say such ungrammatical sentences because we’re stalling as we think of what to say next. Thinking longer before you speak is a better idea.

Grammar Girl

So, now, the mystery is solved: we should not repeat the word “is” in such sentences.

Thanks for clearing that up Grammar Girl!

Compute MD5 or SHA hash of large file efficiently on iOS and Mac OS X

Joel — Tue, 07 Sep 2010 06:18:56 +0000

Computing cryptographic hashes of files on iOS and Mac OS X using the CommonCrypto APIs is fairly easy, but doing it in a way that minimizes memory consumption even with large files can be a little more difficult… The other day, I was reading what some people were saying about this on a forum about iPhone development, and they thought they found the trick, but they still had a growing memory footprint with large files because they forgot something fundamental about memory management in Cocoa.

Updated

Friday, October 1, 2010: removed comment about the fact that I used character arrays on the heap with the more modular solution described at the end of the post; this is now fixed, and that more general solution is now as efficient as the simple one described here.
Sunday, October 17, 2010: added link to a simple GitHub repository that I created to show exactly how to integrate my function FileMD5HashCreateWithPath with a simple iOS or Mac application.

What was wrong with that solution?

Even though they had a solution to read bytes from the file progressively instead of reading everything at once, it did not improve the memory consumption of their program when computing hashes of large files. The mistake they made is that the bytes read in the while loop were in an autoreleased instance of NSData. So, unless they create a local autorelease pool within the while loop, the memory will just accumulate, until the next autorelease pool is drained. But I think it would be very inefficient to add an autorelease pool in the while loop, because you would end up allocating a new object in every pass of the loop.

So, in my opinion, the right question is: how do we read those bytes without getting an autoreleased object?

How to get around that problem?

I looked for a solution, and I couldn’t find anything that would do the same thing as -[NSFileHandle readDataOfLength:] at the Foundation level without returning an autoreleased object. So I thought: we have to go deeper. I looked for something similar in Core Foundation, and sure enough, I found the CFReadStream API.

And since I was going to do this using Core Foundation to read those bytes, I decided to go all the way with Core Foundation, with a solution in pure C.

Here’s how you can compute efficiently the MD5 hash of a large file with CommonCrypto and Core Foundation:

FileMD5Hash.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

// Standard library
#include
#include

// Core Foundation
#include

// Cryptography
#include

// In bytes
#define FileHashDefaultChunkSizeForReadingData 4096

// Function
CFStringRef FileMD5HashCreateWithPath(CFStringRef filePath,
size_t chunkSizeForReadingData) {

// Declare needed variables
CFStringRef result = NULL;
CFReadStreamRef readStream = NULL;

// Get the file URL
CFURLRef fileURL =
CFURLCreateWithFileSystemPath(kCFAllocatorDefault,
(CFStringRef)filePath,
kCFURLPOSIXPathStyle,
(Boolean)false);
if (!fileURL) goto done;

// Create and open the read stream
readStream = CFReadStreamCreateWithFile(kCFAllocatorDefault,
(CFURLRef)fileURL);
if (!readStream) goto done;
bool didSucceed = (bool)CFReadStreamOpen(readStream);
if (!didSucceed) goto done;

// Initialize the hash object
CC_MD5_CTX hashObject;
CC_MD5_Init(&hashObject);

// Make sure chunkSizeForReadingData is valid
if (!chunkSizeForReadingData) {
chunkSizeForReadingData = FileHashDefaultChunkSizeForReadingData;
}

// Feed the data to the hash object
bool hasMoreData = true;
while (hasMoreData) {
uint8_t buffer[chunkSizeForReadingData];
CFIndex readBytesCount = CFReadStreamRead(readStream,
(UInt8 *)buffer,
(CFIndex)sizeof(buffer));
if (readBytesCount == -1) break;
if (readBytesCount == 0) {
hasMoreData = false;
continue;
}
CC_MD5_Update(&hashObject,
(const void *)buffer,
(CC_LONG)readBytesCount);
}

// Check if the read operation succeeded
didSucceed = !hasMoreData;

// Compute the hash digest
unsigned char digest[CC_MD5_DIGEST_LENGTH];
CC_MD5_Final(digest, &hashObject);

// Abort if the read operation failed
if (!didSucceed) goto done;

// Compute the string result
char hash[2 * sizeof(digest) + 1];
for (size_t i = 0; i < sizeof(digest); ++i) {
snprintf(hash + (2 * i), 3, "%02x", (int)(digest[i]));
}
result = CFStringCreateWithCString(kCFAllocatorDefault,
(const char *)hash,
kCFStringEncodingUTF8);

done:

if (readStream) {
CFReadStreamClose(readStream);
CFRelease(readStream);
}
if (fileURL) {
CFRelease(fileURL);
}
return result;
}

Then, from your Objective-C code, you can just use that function like this:

1
2
3
4
5
6
7

NSString *filePath = ...; // Let's assume filePath is defined...
CFStringRef md5hash =
FileMD5HashCreateWithPath((CFStringRef)filePath,
FileHashDefaultChunkSizeForReadingData);
NSLog(@"MD5 hash of file at path \"%@\": %@",
filePath, (NSString *)md5hash);
CFRelease(md5hash);

Remember that FileMD5HashCreateWithPath transfers ownership of the returned string, so you must release it yourself.

I also created a small GitHub repository that may help you understand how to integrate that code in your project. It contains a very simple Xcode project, with a target for iOS and another one for Mac OS X. In both cases, the application just provides a simple button to compute the MD5 hash of the executable file (the binary). Here is where you can find that repository: FileMD5Hash GitHub repository.

Advantages of this solution

There are several nice things about this implementation:

first, it works as advertised: it computes the MD5 hash of the file correctly, and it doesn’t make the memory footprint of your app grow, even if you give it the path to a huge file;
even though the path argument is a CFStringRef, it’s really easy to use this from Objective-C, thanks to the fact that NSString and CFStringRef are toll-free bridged; cf. example above for usage;
it works just fine both on iOS and on Mac OS X;
by reusing sizeof(digest), I avoided the pitfall of exposing the real value of CC_MD5_DIGEST_LENGTH, which would make it more difficult to adapt to other cryptographic algorithms.

How about SHA1, SHA256, and others?

It’s really simple to adapt this function to other algorithms. Say you want to adapt it to get the SHA1 hash instead. Here’s what you need to do:

replace CC_MD5_CTX with CC_SHA1_CTX;
replace CC_MD5_Init with CC_SHA1_Init;
replace CC_MD5_Update with CC_SHA1_Update;
replace CC_MD5_Final with CC_SHA1_Final;
replace CC_MD5_DIGEST_LENGTH with CC_SHA1_DIGEST_LENGTH;

Or more simply, just do a find and replace to transform every occurrence of the string “MD5” with “SHA1“. Voilà, you got it!

Another way to extend this to other algorithms is to make this function more modular, and basically take all of those things as arguments. This is a little more difficult, but I did it for my project TagAdA. With this more advanced and more modular solution, you have a third argument that represents the algorithm that you wish to use, and you only have one instance of the code associated to that logic in your binary, even if you use several of those cryptographic algorithms in your app. I even went to great lengths using the preprocessor to minimize the amount of duplicated code in my source file.

Anyway, there you go! I hope you will find this useful.

Configure common command-line tools on Mac OS X

Joel — Thu, 29 Jan 2009 19:24:38 +0000

Even if you properly configured the Terminal application on your Mac, you may think that using it is difficult, and not intuitive at all. Indeed, with the default configuration for the common command-line tools, you won’t get syntax highlighting within vim, nor will you have a shell which facilitates your work, and so on. But since I started my journey in Mac universe after spending some time on Linux, I am used to configure some of these tools, and I simply decided to make my shell and my other command-line tools on my mac look like what I previously had in my Linux box.

So, let me tell you what I did to have really usable command-line tools on my Mac…

Updated

Sunday, September 5, 2010: adapted the tutorial to Snow Leopard; incorporated Louis Romero’s idea about hiding /opt for the Finder; other minor changes.

For your information, I did all these things on Mac OS X Snow Leopard 10.6.4. Let’s get started!

Install MacPorts

First of all, I noticed that some of the tools I liked were missing on my Mac. Basically, every command-line tool can be difficult to find, and at the same time, you may want to use a package management system to manage such software. So I could choose between MacPorts, and Fink. Although Fink seemed more powerful (in fact, more Debian-like), I had the impression that there wasn’t any kind of active development on Fink at the time I wanted to install it. So I chose MacPorts.

MacPorts is rather handy in fact, since it will automatically fetch the sources of what you want to install, and compile them with the right options. By the way, that’s why the first thing you need to do even before installing MacPorts is to install Apple Developer Tools. You can find them on Apple’s Developer website.

MacPorts is not perfect, and I already experienced some compilation failures with it, but it works most of the time, and it’s rather straightforward. One of the things I regret when I remember Debian’s aptitude, though, is its advanced dependency management system; indeed, I think MacPorts’ one is very basic. But hey, nothing is perfect.

Anyway, I just installed MacPorts using the Snow Leopard “dmg” disk image. Just refer to the download page of MacPorts. I also encourage you to read the MacPorts Guide, which explains with more details how to install MacPorts properly and how to use it. But you can just skip the part called “MacPorts and the Shell”, and instead do the following in a Terminal window:

sudo sh -c 'echo "/opt/local/bin\n/opt/local/sbin" > /etc/paths.d/macports'
sudo sh -c 'echo "/opt/local/share/man" > /etc/manpaths.d/macports'

Also, you may want the /opt directory (where MacPorts lives) hidden in the Finder, like a lot of other system directories. Here’s how you do that:

sudo SetFile -a V /opt

Now, just make sure you quit the Terminal application and relaunch it. Let’s continue…

Install some tools with MacPorts

Once MacPorts is installed, you can easily install some useful tools. For the rest of this tutorial, you will just need to do the following:

sudo port install figlet fortune par wget

The tools figlet and fortune are some fun tools, you will see later why we need them; par is very useful to wrap lines with vim, and wget can be used to fetch some files from the internet without using curl.

But for your information, here is the list of the other tools I usually install:

convmv: to re-encode some filenames;
nmap: to discover open ports on remote servers;
irssi: an excellent text based IRC client;
p7zip, pbzip2, unrar: compression utilities;
potrace: to vectorize bitmap image files;
pstree: to see in a fancy way the list of processes running on your Mac;
signing-party: to help you sign some GnuPG keys;
w3m: a text based web browser.

I also install the excellent text based mail client mutt, but this one is a little tricky; here is how you should install it:

sudo port install mutt-devel +gdbm +headercache +imap +pop +ssl

It will install the right version of mutt with the right variants to enable everything that can be useful for normal email usage.

That’s it for MacPorts!

Configure Terminal on Mac OS X

Joel — Wed, 28 Jan 2009 20:28:19 +0000

Every time someone asks me for help about something geek on a Mac, I often prefer to use the Terminal, because I never got used to editing a configuration file under /etc with a graphical text editor. Don’t misunderstand me, I am not a Terminal addict, and I often prefer using graphical tools that are extremely integrated with Mac OS X: for example, I wouldn’t code in C or C++ with something else than Xcode, which turns out to be an incredible IDE. But for some other things, I still launch this Terminal, and there are a few things in its default configuration I never liked, such as the behavior of the keys page up/page down, home/end, etc.

So, let me tell you how you can configure your Terminal like I did on my mac, running Mac OS X Snow Leopard 10.6.4.

General settings

First, launch Terminal (using Spotlight, my favorite!), open the Preferences window, and click on Settings. Here, you will see on the left a list of predefined themes with different colors. If you decide to change your default theme, just click on it, and then click the button Default. I always preferred a black terminal with white text, so I chose the theme called Pro. Then, to be sure this will work (specially if you already changed some preferences here), you can click on the cogwheel button next to the minus button, and choose Restore Defaults.

In order to have a much more beautiful text on your terminal, you should check the box Antialias text.

If, like me, you don’t understand why a Terminal window doesn’t close automatically once the shell exits (for example, when using the common shortcut ^+D), then you should go to the Shell tab, and under When the shell exits: select Close the window.

Keys and actions

Let’s go to the tricky part: keys and actions. Go to the Keyboard tab, and set the following keys to the corresponding actions:

Key & modifier	Action	Corresponding string	What to type?
control cursor left	send string to shell	\033b	esc B
control cursor right	send string to shell	\033f	esc F
home	send string to shell	\001	^+A
end	send string to shell	\005	^+E
page up	send string to shell	\033[5~	esc [5~
page down	send string to shell	\033[6~	esc [6~
shift home	scroll to start of buffer
shift end	scroll to end of buffer
shift page up	scroll to previous page in buffer
shift page down	scroll to next page in buffer

As you will see, when typing the corresponding string in the text area under send string to shell, every keystroke you type can create a weird output. So you should just type what is in the last column, knowing that esc represents the escape button, that ^ is the control button, and that + means press both keys at the same time.

What if you use a remote screen with ssh on a Linux server?

Everything should work properly now… except if you’re like me and you often use a screen on Linux boxes through ssh. If you didn’t even understand this sentence, you should just skip this part.

In this case, your backspace key won’t work properly on the remote screen, and it’s a shame. To solve this problem, the best solution I found was to upload the file /usr/share/terminfo/78/xterm-color of the Mac to the server (with scp for instance, or with the excellent Cyberduck application through SFTP) and to do the following on the server:

mkdir -p ~/.terminfo/x/
mv ~/path/to/xterm-color ~/.terminfo/x/

This trick works for me, but if you use a remote screen on many different Linux servers, then it can be a pain. Hopefully, like me, you will only need it for a single account on a single Linux server where you run irssi in a screen; in that case, you just found the solution for the backspace problem.

The end

And you’re done! Just quit the Terminal application, launch it again. That’s it!

One last word, you should also look at the feature called Window groups: it allows you to save the way several Terminal windows are on the screen, and it can be useful for someone who often uses more than one Terminal window at a time.

Enjoy!

Draggable Screen Ruler for Web Development

Joel — Sun, 25 Jan 2009 23:27:31 +0000

When doing some web development work, I used to use Screenshot Plus to measure the width of some elements. While this Mac OS X widget is very handy, it wasn’t very appropriate for this job: it required me to load Mac OS X’s Dashboard, activate Drag a picture in the widget, drag around what I wanted to measure, and finally drop the screenshot which was just taken. I thought it was a complex way to simply measure things in a web page, but I also thought there should a much better way to do that, which would be far more integrated within the web browser.

How to use the Draggable Screen Ruler bookmarklet

Well, I was right, since this Draggable Screen Ruler bookmarklet actually does the job quite efficiently, while also giving some useful information such as absolute position in the window. I wrote it a few days ago, and I actually use it every time I work on web design with CSS. You just have to drag and drop the previous link to your bookmarks bar, and you will be able to measure anything from any website you visit. To deactivate it, simply double click with your mouse, or press any key on your keyboard. Isn’t it nice?

How it all works

In fact, the bookmarklet will simply retrieve this much bigger Javascript file over the internet, which means that you won’t be able to use it when your are not connected to the internet. But this also guarantees transparent updates in case I enhance it, or fix a bug in the javascript file.

However, there is a simple solution if you really want to use it without internet connectivity, provided that you have a local web server activated (which I assume you have if you are a web developer): just download the Javascript file, and put it somewhere on your computer, so that it is available at local address, such as http://localhost/path/to/scripts/Ruler.js; then replace the content of the variable called url in the bookmarklet code by this local URL, and you’re all set.

This bookmarklet has been tested only with Firefox 3.0 and Safari 3.1.2, and it worked perfectly with those two browsers.

Where the idea came from

Actually, this bookmarklet is not one of my ideas: I found this Draggable Screen Ruler made by Kevin C. Smith by chance, and I thought it was a very good idea, but I regretted a few things about the implementation.

For example, the original bookmarklet didn’t allow us to measure something, deactivate it, and then reactivate it clicking on the bookmarklet without reloading the web page. It turns out the main reason for that was the fact that everything was done with global javascript variables which status was not properly updated in this case, and that forced us to reload the page to measure something else when the bookmarlet had already been deactivated.

I also regretted that the colors of the elements in the info box of the ruler could change with some web pages, which would result in bad contrast and poor readability. And finally, I found the bookmarklet code very difficult to understand: variables have very short names which are not explicit at all…

So I decided to come up with my own version of the bookmarklet, a version which solves all those issues. I think it’s much easier to understand, and therefore easier to maintain. Of course, I left a reference to the original bookmarlet web page next to the copyright notice.

I hope you will like this nifty tool as much as I do!

How can Windows be easier to use than other Operating Systems?

Joel — Sun, 06 Jul 2008 13:57:56 +0000

Before I got into graduate school, I almost knew nothing but Microsoft Windows as an Operating System. I even knew many tricks about Windows, and learned some of the things that would inevitably make it unstable… Then, I just thought that Computers had to be that way…

The real question…

Fortunately, I am not such a narrow-minded guy, and I definitely wanted to know why many scientists and engineers hated Windows, and chose to use something else, such as Linux. Entering VIA Centrale Réseaux, the organization which provides a high speed internet access to more than a thousand students in the Residence of the École Centrale Paris, was a real revolution to me… In order to be able to manage the organizations’ Debian Servers, as a member of the Root team, I had to learn quite fast what Linux was, and how to use it with a command-line interface. A few months later, I had become some sort of a geek who wouldn’t want to use Windows at all… But, putting the ideological fight free vs. proprietary aside, you might want to know what was so great with Debian or Ubuntu (the Linux distributions I use) for a simple user.

Windows is very buggy

I think that the best thing with Linux is its stability. Haven’t you heard any Windows user saying things such as “I need to format my C: drive once more…”? This is not a myth: as a former Windows geek, I knew that starting all over again was, most of the time, the only solution to get rid of Windows instability. What I mean by instability might simply be error messages which pop up just after login… And once again, I experienced this sort of problem: my Mom has a computer with both Ubuntu and Windows XP. For your information, I always make sure not to install software that is not really needed on Windows, and this computer hasn’t been really used on Windows yet: it’s just there, in case of an urgent need. Indeed, my Mom uses exclusively Ubuntu, which she likes very much.

A few months ago, I decided to boot Windows (such a rare thing now!), in order to upgrade it to the SP3. Not only did I have to install 15 upgrades and to do the classic reboot before Microsoft Update would suggest me to install the SP3, but after this very long upgrade another Microsoft program became buggy… I had the pleasure to welcome a wonderful “2771 error” from Windows Live Messenger… What can you expect from a firm which doesn’t really know how to upgrade its OS without affecting its other flagship products? Fortunately, after 5 minutes of googling about this error, I found out that I just had to uninstall Windows Live Messenger, and to install it again.

Yet, can we say that such problems must be accepted by all users? Can we really say to a grandma that using Windows is simple, and that she wouldn’t have any problem? Can we really say that such bugs are acceptable? You might say “but the solution to your problem was so easy… anyone would have thought about it!”. But I won’t agree with you: I progressively understood that most users are not so comfortable with technology as I am, and for some users, those things have no logic at all, and they may be completely clueless when facing such a bug.

Software management

Besides, haven’t you already heard someone saying to a Windows user, talking about a non-Microsoft program: “but this feature is available in the new version of the software, you just have to upgrade it”? Such answers simply reveal an enormous problem in Windows: software management is not uniform at all… What is so great about Debian, Ubuntu, or other famous Linux Distributions is indeed the Distribution concept itself.

From what I understood, a Linux Distribution is an operating system based on a Linux Kernel, and a set of coherent pieces of software, which may have been adapted to the distribution’s main choices and organization, with a common tool which makes software management easy. With Debian or Ubuntu, we can think about APT, or its Graphical User Interface Synaptic. Although Linux Distributions are definitely modular, the set of pieces of software of one Linux distribution is really coherent, and one piece of software just works well with the rest of the software of this distribution. The great thing about this way of managing software is that all software can be installed, removed, or upgraded with a single tool: the distribution’s package management system. Even better, APT is very good at managing dependencies, or conflicts between software. This way, with such a Linux Distribution, you can manage your software very easily, and be sure that everything is up to date at all time. Isn’t it simplicity itself?

Furthermore, when using Microsoft Update, I always get angry when I realize, once more, how slow upgrading Microsoft programs is. Indeed, APT is much faster for that, and it is able to upgrade the whole system without asking to reboot in order to install the rest of the available upgrades, which occurs quite frequently with Microsoft Update…

Installing Tomcat on Mac OS X

Joel — Tue, 13 May 2008 22:30:45 +0000

Having to use Tomcat for an important project I am working on, I installed it on my Mac. However, in the beginning, I didn’t make much effort: I just uncompressed the official archive in my /usr/local, and I was able to start and stop Tomcat manually, that’s all… But today I decided to learn a few things about Mac OS X, trying to install it properly. Let’s see what I did!

Updated

Friday, January 30, 2009: added instructions to create an admin user for Tomcat; corrected and enhanced instructions for permissions; added comments on deploying applications which need access to the filesystem; added link to Lambda Probe; corrected the StartupItem files; many thanks to Derk Norton for suggesting some of those corrections.
Saturday, September 4, 2010: replaced instructions about setting up Tomcat as a StartupItem by instructions to set it up as a launchd job; added a section about Tomcat 7.

This tutorial is based on my experience installing Tomcat 6.0.29 on a Mac OS X Snow Leopard 10.6.4 system, and I have not tested those instructions with other versions of the operating system.

Downloading and uncompressing Tomcat

First, download from Tomcat’s website the version of Tomcat you want to use. Choose the .tar.gz file in the Core section of the Binary Distributions. Let’s say you just downloaded it to your Downloads directory (usually /Users/username/Downloads/). Open a Terminal and execute the following commands:

sudo mkdir -p /usr/local
cd /usr/local
sudo tar xvzf ~/Downloads/apache-tomcat-6.0.29.tar.gz
sudo ln -s apache-tomcat-6.0.29 tomcat

You also need to create an admin user for Tomcat. I will assume you are familiar with vim, but you can use any other text editor. Let’s edit the right file:

sudo vim /usr/local/tomcat/conf/tomcat-users.xml

Then replace its contents by the following:

/usr/local/tomcat/conf/tomcat-users.xml

1
2
3
4
5
6

version='1.0' encoding='utf-8'?>

rolename="manager"/>
rolename="admin"/>
username="admin" password="pass" roles="admin,manager"/>
>

Of course, you can replace the word pass by any password you want to use.

Tomcat should already be working, but we should not run it as root, since it may be a huge security issue. That is why we need to first create an unprivileged user as well as its primary group.

Creating an unprivileged user

We will be using the command-line utility dscl for this. First, you need to choose the User ID and the Group ID, which must be a positive integer, lower than 500, since User IDs above 500 are reserved to normal users. You just have to pick a number between 0 and 500 which is not yet used by any other user or group. Check this out with the commands:

dscl . -list /Groups PrimaryGroupID
dscl . -list /Users UniqueID

For this tutorial, I will be using the number 107. We can create the group and the user with the following commands:

sudo dscl . -create /Groups/_tomcat PrimaryGroupID 107
sudo dscl . -create /Groups/_tomcat RealName "Tomcat Users"
sudo dscl . -create /Groups/_tomcat Password \*
sudo dscl . -create /Users/_tomcat UniqueID 107
sudo dscl . -create /Users/_tomcat PrimaryGroupID 107
sudo dscl . -create /Users/_tomcat HomeDirectory /usr/local/tomcat
sudo dscl . -create /Users/_tomcat UserShell /usr/bin/false
sudo dscl . -create /Users/_tomcat RealName "Tomcat Administrator"
sudo dscl . -create /Users/_tomcat Password \*

Choosing /usr/bin/false as the UserShell, and setting the Password to “*” turns this account unusable as a standard user account.

Setting permissions

Since Tomcat will be run as the _tomcat user, you must change the ownership of some directories and files. Just run the following commands:

cd /usr/local/tomcat
sudo chown -R root:wheel .
sudo chmod 644 conf/*
sudo chown root:_tomcat conf/tomcat-users.xml
sudo chmod 640 conf/tomcat-users.xml
sudo mkdir conf/Catalina
sudo chown _tomcat:_tomcat conf/Catalina
sudo chown _tomcat:admin logs temp webapps work
sudo chmod 2770 logs temp webapps work
cd -

This way, Tomcat should have the right permissions for all the directories it needs to write into, and at the same time, you won’t need to use root privileges to read the logs, and so on.

But be careful when developing an application which needs to have access to some directories in the filesystem (for storing files inside the web application for example): you might need to give Tomcat specific permissions on those directories for this to work.

However, I think that a good way to avoid a lot of Unix permissions headaches is to deploy your web applications only using tools such as the Tomcat Manager, or the excellent Lambda Probe.

Setting Tomcat as a launchd job

If we want to have Tomcat start up automatically at boot time, or if we simply want to easily start it and stop it without having to think about details like paths and permissions, it’s a good idea to use it as a launchd job.

The first big thing we need to do for that is to create a wrapper script that will start Tomcat and not return until the Tomcat process exits, which is not what the startup.sh and catalina.sh scripts do, as they launch Tomcat as a background process.

To do that, let’s first create a new shell script in Tomcat’s bin directory:

sudo touch /usr/local/tomcat/bin/tomcat-launchd.sh
sudo chmod +x /usr/local/tomcat/bin/tomcat-launchd.sh
sudo vim /usr/local/tomcat/bin/tomcat-launchd.sh

Then put the following content:

/usr/local/tomcat/bin/tomcat-launchd.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

#! /bin/sh

# tomcat-launchd.sh
#
# Wrapper script that starts Tomcat and waits for the Tomcat process
# to exit. This is needed for proper interaction with launchd.

#---------------------------------------------------------
# Helper functions
#---------------------------------------------------------

# NOTE: We are inheriting CATALINA_HOME from launchd, because its value
# was defined in the launchd plist configuration file.

function shutdown() {

# Bye Tomcat!
echo "Shutting down Tomcat... "
$CATALINA_HOME/bin/catalina.sh stop
echo "done."

# Cleaning up the temporary file
rm -f $CATALINA_PID

}

function startup() {

# Define the file where we want the Tomcat process ID to be stored.
export CATALINA_PID=$(mktemp /tmp/`basename -s .sh $0`.XXXXXX)
if [ $? -ne 0 ]
then
echo "$0: Failed to create temporary file. Aborting."
exit 1
fi
rm -f $CATALINA_PID

# Let's go!
echo "Starting up Tomcat... "
. $CATALINA_HOME/bin/catalina.sh start

# Register the shutdown function as callback to execute when a signal
# is sent to this process.
trap shutdown HUP INT QUIT ABRT KILL ALRM TERM TSTP
echo "done."

}

function wait_for_tomcat_to_exit() {
echo "Waiting for Tomcat to exit (PID: `cat $CATALINA_PID`)... "
wait `cat $CATALINA_PID`
echo "done waiting for Tomcat to exit."
}

#---------------------------------------------------------
# Let's go
#---------------------------------------------------------

startup
wait_for_tomcat_to_exit

Then, we need to create a configuration plist file for the Tomcat launchd job:

sudo vim /Library/LaunchDaemons/org.apache.tomcat.plist

Then put the following content:

/Library/LaunchDaemons/org.apache.tomcat.plist

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

version="1.0" encoding="UTF-8"?>

"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
version="1.0">
>
>Label>
>org.apache.tomcat>
>ServiceDescription>
>Tomcat Servlet/JSP Server>
>UserName>
>_tomcat>
>GroupName>
>_tomcat>
>EnvironmentVariables>
>
>CATALINA_HOME>
>/usr/local/tomcat>
>JAVA_HOME>
>/System/Library/Frameworks/JavaVM.framework/Home>
>
>ProgramArguments>
>
>/usr/local/tomcat/bin/tomcat-launchd.sh>
>
>StandardOutPath>
>/usr/local/tomcat/logs/launchd-stdout.log>
>StandardErrorPath>
>/usr/local/tomcat/logs/launchd-stderr.log>
>RunAtLoad>
/>
>KeepAlive>
/>
>
>

Using launchd to manage Tomcat

The first thing we need to do is load this configuration file:

sudo launchctl load /Library/LaunchDaemons/org.apache.tomcat.plist

Tomcat should be launching as the user _tomcat, and if you go to http://localhost:8080/ using your favorite browser, you should get to the homepage of your new Tomcat server.

With the above launchd configuration file, Tomcat is pretty much guaranteed to always be running on your system: it will be restarted automatically for you, by launchd, even if it exits for any reason (for example, a crash, or a reboot). The key that controls the fact that Tomcat is launched at boot time is actually the key named RunAtLoad. So, if you don’t like this behavior, you can always set that key to false, and reload the configuration:

sudo launchctl unload /Library/LaunchDaemons/org.apache.tomcat.plist
sudo launchctl load /Library/LaunchDaemons/org.apache.tomcat.plist

You should run those two commands to reload the configuration anytime you make a change to the plist file.

Now, let’s say that you want to just restart the Tomcat server (maybe because you changed some of Tomcat’s configuration files). With the above launchd configuration file, all you have to do is this:

sudo launchctl stop org.apache.tomcat

Indeed, you don’t even need to do an equivalent start call, because launchd will take care of relaunching the Tomcat server. That is because we set the key KeepAlive to true. If you prefer to be able to really shut down Tomcat, then you can just change that key to false, reload the launchd configuration, and then, to achieve the same result, you’ll have to run two commands now:

sudo launchctl stop org.apache.tomcat
sudo launchctl start org.apache.tomcat

Of course, if you omit the start command, then the server will be permanently shut down (until the next reboot, if you still have the key RunAtLoad set to true).

You can customize the behavior of launchd with regard to Tomcat even further, but I suggest you just check out launchd plists’s man page if you want to learn more about that.

Update for Tomcat 7

Tomcat 7 is around the corner. It’s still considered beta software at the moment (at time of writing, the latest released version of Tomcat is 7.0.2), but it seems to work pretty well. I tried to see if those instructions would work with Tomcat 7, and apparently, they do, except for one thing. The content you need to put in the tomcat-users.xml file is a little different:

/usr/local/tomcat/conf/tomcat-users.xml

1
2
3
4
5
6

version='1.0' encoding='utf-8'?>

rolename="manager-gui"/>
rolename="admin"/>
username="admin" password="pass" roles="admin,manager-gui"/>
>

The difference is that I had to replace the role manager by manager-gui. Tomcat’s migration page explains that difference very well.

Other than that, you should be able to install Tomcat 7 on your Mac using those instructions.

Enjoy!

mmv on Mac OS X

Joel — Mon, 12 May 2008 13:18:02 +0000

Since I switched from Debian and Ubuntu to Mac OS X, I got used to most of the differences between those operating systems. Yet one very important thing for me was still lacking: A command-line utility called mmv…

Updated

Friday, January 30, 2009: added a small command to rebuild the whatis database after install and uninstall.
Wednesday, September 1, 2010: replaced the version of mmv distributed by a newer one, compiled differently, and specially targeted at Mac OS X Snow Leopard; also added a small update about the fact that mmv is now available through MacPorts.

Description

According to this article, which describes very well what mmv can do for you:

mmv is a command-line tool which allows the user to move, rename, copy, append and link large amounts of files with a single command. The tool is especially useful when you need to rename a lot of files that have similar filenames, yet subtle differences.

Debian Package of the Day

The problem, and my idea…

Unfortunately, mmv is not available through MacPorts, which I tend to use when I want to install some specific command-line tools on my Mac. But then I thought that I could try recompiling myself the source code for mmv that is used to create the mmv package for Debian.

Update: At the time of initial writing, this was true, but it is no longer the case: I just discovered that mmv is now available through MacPorts. However, looking at the version number they advertise, my version is more recent.

Sure enough, it wasn’t that complicated, I quickly got a 3-way Universal Binary (x86_64, i386 and ppc) version of mmv for Mac OS X. Finally, mmv on the Mac! Then, I thought: Why wouldn’t I make an archive of all those files, and distribute my binary version of mmv for Mac OS X? According to mmv‘s license, there is no reason why I couldn’t do that.

Download

Therefore, here is my version of mmv:

Binary version of mmv for Mac OS X Snow Leopard.

Install

First, download the .tar.bz2 file, and be sure to let it in your Downloads directory (usually /Users/username/Downloads/).

Then, launch the Terminal application and execute the following commands:

cd /usr/local/
sudo tar xvjf ~/Downloads/mmv-binary-MacOSX-10.6.tar.bz2
sudo sh -c '/etc/periodic/weekly/320.whatis 2> /dev/null; exit 0'

That’s it!

Uninstall

Launch the Terminal application and execute the following commands:

cd /usr/local/
sudo rm -R -f bin/m{ad,cp,ln,mv} share/doc/mmv/ share/man/man1/m{ad,cp,ln,mv}.1
sudo sh -c '/etc/periodic/weekly/320.whatis 2> /dev/null; exit 0'

That’s it!

Important Notes

As the distributor of this binary version of mmv, I want to add to mmv‘s copyright notice my own warning:

This software is provided by the distributor “as is” and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the distributor be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this software, even if advised of the possibility of such damage.

Another important thing is that I only tested this version of mmv on Mac OS X Snow Leopard 10.6.4, but it should work with all versions of Snow Leopard, and most likely with future major versions of the Mac OS X. If you find that it works with older versions of the operating system, please let me know.

I also wrote a README file containing all the important things you should know about this package. Once the software installed, it should be located at /usr/local/share/doc/mmv/README.

One more thing…

Enjoy!

Japanese in LaTeX documents in Unicode with MacTeX

Joel — Sat, 10 May 2008 13:51:31 +0000

Since I discovered LaTeX, I have become a real fan of this typesetting system. Yet, something bothered me very much: I could not use easily Japanese with LaTeX using a normal encoding, that is to say using UTF-8. Using CJK is not too difficult, but getting CJKutf8 to work is much harder. I just couldn’t resign myself to use some strange encoding popular in Japan, mostly for Windows Addicts…

But I just found how to do it. Since it is not so easy, I want to do a very simple tutorial explaining precisely how to install UTF-8 Japanese support in MacTeX. So, just follow the guide…

Important Note

Even though this tutorial worked just fine a while back (with Mac OS X Leopard 10.5.2, I believe), it stopped working for some reason with later updates (including the latest software updates for Leopard and Snow Leopard). The problem is that at the end of the whole process, the PDF generated using pdflatex uses crazy glyphs instead of Japanese ones.

However, I know that copying the generated files from another machine on which this procedure has worked makes it also work on other machines. So I may end up publishing those files someday.

Updated

Friday, November 14, 2008: updated for MacTeX 2008 support on Leopard; small “bugfix” of the tutorial.
Monday, December 1, 2008: replaced wget by curl -O.

Of course, installing MacTeX is a prerequisite. This tutorial has been tested on Mac OS X 10.5.2 (Leopard) with both versions 2007 and 2008 of MacTeX, and it is just the adaptation of a procedure described in a Debian README file (/usr/share/doc/latex-cjk-common/README.Debian.gz in Debian Etch)…

The problem when we want to type Japanese directly in UTF-8 encoding in a LaTeX document is that we have to use the package CJKutf8 instead of CJK, and by default, there is no font with Japanese symbols adapted for UTF-8 typesetting in LaTeX. Therefore, the main goal of this tutorial is to install the font Cyberbit into your LaTeX distribution. Please, be sure to acknowledge the font’s license before proceeding with this tutorial.

First, in a terminal, download and unzip the font file:

curl -O ftp://ftp.netscape.com/pub/communicator/extras/fonts/windows/Cyberbit.ZIP
unzip Cyberbit.ZIP
rm -f Cyberbit.ZIP

Then, we need to become root in order to prepare some directories and files:

sudo -s # Enter your password if needed
mkdir -p /usr/local/share/fonts/truetype/bitstream/
mv Cyberbit.ttf /usr/local/share/fonts/truetype/bitstream/cyberbit.ttf
mkdir -p /usr/src/cyberbit-fonts/
exit # After this you should be back in a shell without superuser privileges
sudo chown "$USER":staff /usr/src/cyberbit-fonts/
cd /usr/src/cyberbit-fonts/

/usr/src/cyberbit-fonts/ will be our build directory. We must now prepare some more files in that directory:

ln -s /usr/local/share/fonts/truetype/bitstream/cyberbit.ttf
cp /Library/TeX/Root/texmf-dist/source/latex/CJK/utils/subfonts/subfonts.pe .

You mustn’t forget the final “.”, since it means the working directory. Then, you have to run the following command:

fontforge -script subfonts.pe cyberbit.ttf cyberbit \
/Library/TeX/Root/texmf/fonts/sfd/Unicode.sfd

This will take a very long time, probably several hours, so if you have something else to do, go for it! Then you must generate a file called cyberbit.map:

touch cyberbit.map
for i in *.pfb
do
echo "$(basename $i .pfb) $(basename $i .pfb)" >> cyberbit.map
done

Then, we need root‘s privileges for pretty much all the commands we need to run, so type the following command:

sudo -s

in order to become root, and type your password. Then do the following:

mkdir -p /usr/local/texlive/texmf-local/fonts/map/dvips/cyberbit/
mkdir -p /usr/local/texlive/texmf-local/fonts/{afm,type1,tfm}/cyberbit/
cp cyberbit.map /usr/local/texlive/texmf-local/fonts/map/dvips/cyberbit/
cp *.afm /usr/local/texlive/texmf-local/fonts/afm/cyberbit/
cp *.pfb /usr/local/texlive/texmf-local/fonts/type1/cyberbit/
cp *.tfm /usr/local/texlive/texmf-local/fonts/tfm/cyberbit/
echo "Map cyberbit.map" >> /Library/TeX/Root/texmf-var/web2c/updmap.cfg
cd ..
export PATH="/usr/texbin:$PATH"
texhash
updmap
updmap-sys

Then you have to replace a file of the MacTeX distribution by another one:

rm -f /Library/TeX/Root/texmf-dist/tex/latex/CJK/UTF8/c70song.fd
mkdir -p /usr/local/texlive/texmf-local/tex/latex/CJK/UTF-8/
cd /usr/local/texlive/texmf-local/tex/latex/CJK/UTF-8/
touch c70song.fd && chmod -R go+w .
open -a TextEdit c70song.fd

Now, a new TextEdit window should have opened, with an empty file. Copy-paste the following text into the empty file:

/usr/local/texlive/texmf-local/tex/latex/CJK/UTF-8/c70song.fd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

%%%%%%
% This is the file c70song.fd of the CJK package
% for using Asian logographs (Chinese/Japanese/Korean) with LaTeX2e
%
% created by Werner Lemberg
%
% Version 4.6.0 (11-Aug-2005)

\def\fileversion{4.6.0}
\def\filedate{2005/08/11}
\ProvidesFile{c70song.fd}[\filedate\space\fileversion]

% character set: Unicode U+0080 - U+FFFD
% font encoding: Unicode

\DeclareFontFamily{C70}{song}{\hyphenchar \font\m@ne}

\DeclareFontShape{C70}{song}{m}{n}{<-> CJK * cyberbit}{}
\DeclareFontShape{C70}{song}{bx}{n}{<-> CJKb * cyberbit}{\CJKbold}

\endinput
%%%%%%

Then save the file. Lastly, run:

chown -R root:wheel . && chmod -R go-w .
texhash

That’s it! Now you can enter the following LaTeX code as UTF-8 (double check the preferences of your LaTeX editor) in a new LaTeX file and compile it with pdflatex:

Example.tex

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

\documentclass[a4paper,12pt]{article}

%%%%%%%%%%%%%%%%%%%%%%%
% Encodings, fonts... %
%%%%%%%%%%%%%%%%%%%%%%%

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Oriental languages specificic options %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\usepackage[T1]{CJKutf8}

\usepackage[overlap,CJK]{ruby} % Furigana support
\renewcommand{\rubysize}{0.5} % Furigana size
\renewcommand{\rubysep}{-0.3ex} % Spacing between Furigana and Kanji

%%%%%%%%%%%%%%%%%
% Title, author %
%%%%%%%%%%%%%%%%%

\title{Japanese in UTF-8 with \LaTeX}
\author{Joel \textsc{Lopes Da Silva}}

\begin{document}

\begin{CJK}{UTF8}{song}

\maketitle
\thispagestyle{empty}

\section{タイトル}

おめでとうございます。\ruby{自分}{じぶん}でできましたね。

\bigskip

これから、日本語で \LaTeX の\ruby{資料}{しりょう}を\ruby{書}{か}けます。

\section{Another title}

You can even use English words within a
日本語の\ruby{資料}{しりょう}~(document in Japanese).

\bigskip

Have fun!

\end{CJK}

\end{document}

Once compiled with pdflatex, you should get the expected result: a nice PDF document generated by LaTeX with Japanese text. Enjoy!

A matter of language

Joel — Mon, 03 Mar 2008 10:44:25 +0000

As you can see, although English is not my native language, I chose to write articles in English for my personal website. The reason why I chose to write in English is quite simple: English is the most spoken language in the world, and I didn’t want to translate every article in some secondary language.

Yet, I may use French for some articles which have nothing to do with non-French people. I hope you will understand that!

However, my English surely is not perfect nor genuine. Therefore, if you see some mistake in my articles, please report it with the Contact form. By the way, don’t hesitate to make comments on my website or my articles: I do like having feedback, and I try to keep my articles up to date.