Introduction

As developers, when one of our applications crashes, we would like to gather enough information about the crash such that we can reason about its cause and (ideally) fix it. Crash reports generated and provided by tools and services such as iTunes Connect, PLCrashReporter, HockeyApp or others can look a bit daunting at first. We will demystify the important aspects of such reports, so that after reading this article one should be able to use crash report data more effectively.

We are going to be talking mainly about crashes of iOS applications, but some of the concepts can be carried over to crashes on other platforms.

Also, there are some reasons for crashes that are of special interest for iOS developers, such as when a watchdog timeout happens, or when the application is killed due to memory constraints. We are not going to cover those here, because others have already done that.

A First Look

Let us dive right in and consider a full, real-world crash report from a production app. There is a lot of information shown there. We are going to break it up into more easily digestable fragments and discuss those individually.

The Header

At the start of every crash report, you’ll find the basic header:

Incident Identifier: A8111234-3CD8-4FF0-BD99-CFF7FDACB212
CrashReporter Key: A54D28AF-3010-5839-BBA6-FE72C8AFCC2E
Hardware Model: iPod3,1
Process: OurApp [476]
Path: /Users/USER/OurApp.app/OurApp
Identifier: coop.plausible.OurApp
Version: 48
Code Type: ARM
Parent Process: launchd [1]

Date/Time: 2013-03-30T04:42:07Z
OS Version: iPhone OS 5.1.1 (9B206)
Report Version: 104

Most of these fields are self-explanatory, but a few deserve note:

• Incident Identifier: Client-assigned unique identifier for the report.
• CrashReporter Key: This is a client-assigned, anonymized, per-device identifier, similar to the UDID. This helps in determining how widespread an issue is.
• Hardware Model: This is the hardware on which a crash occurred, as available from the “hw.machine” sysctl. This can be useful for reproducing some bugs that are specific to a given phone model, but those cases are rare.
• Code Type: This is the target processor type. On an iOS device, this will always be ‘ARM’, even if the code is ARMv7 or ARMv7s.
• OS Version: The OS version on which the crash occurred, including the build number. This can be used to identify regressions that are specific to a given OS release. Note that while different models of iOS devices are assigned unique build numbers (eg, 9B206), crashes are only very rarely specific to a given OS build.
• Report Version: This opaque value is used by Apple to version the actual format of the report. As the report format is changed, Apple may update this version number. In PLCrashReporter, we generate and store reports in our own structured protobuf-based format, and generate Apple-compatible reports on-demand.

The Stack Trace

On iOS, an application is a single process that typically contains multiple active threads, including the main UI thread, the dispatch manager thread, and other worker threads for things like I/O. Each thread executes code, and a thread’s stack trace shows a list of the function calls the thread took to ultimately end up at its current place. A good crash report contains a stack trace for every thread at the time of crashing, and will also tell us in which thread the crash occurred. We read this from the bottom up, with the topmost entry being the top stack item, i.e. the innermost function that was called. The following shows an example crash on the main thread:

Thread 0 Crashed:
0 libsystem_kernel.dylib 0x3466e32c ___pthread_kill + 8
1 libsystem_c.dylib 0x3526829f abort + 95
2 OurApp 0x0015dfc3 uncaught_exception_handler + 27
3 CoreFoundation 0x3601b957 __handleUncaughtException + 75
4 libobjc.A.dylib 0x30f91345 __objc_terminate + 129
5 libc++abi.dylib 0x34d333c5 __ZL19safe_handler_callerPFvvE + 77
6 libc++abi.dylib 0x34d33451 __ZdlPv + 1
7 libc++abi.dylib 0x34d34825 ___cxa_current_exception_type + 1
8 libobjc.A.dylib 0x30f912a9 _objc_exception_rethrow + 13
9 CoreFoundation 0x35f7150d _CFRunLoopRunSpecific + 405
10 CoreFoundation 0x35f7136d _CFRunLoopRunInMode + 105
11 GraphicsServices 0x31876439 _GSEventRunModal + 137
12 UIKit 0x3192ecd5 _UIApplicationMain + 1081
13 OurApp 0x000938e7 main (main.m:16)

Thread 1:
...

If we eyeball that stack trace, we can easily see that the cause of the crash is an unhandled exception. (If you find yourself getting a lot of these, you might want to set up an Exception Breakpoint in Xcode, which takes you to the function that throws the exception when testing your application during development.)

It is probably safe to say that a crash report’s stack traces are what most programmers first consider, as they are relatively easy to comprehend. Often but not always a stack trace is all that’s required to understand the underlying cause of a crash.

Debug Symbols

It is important to note that at the time an application is built for deployment, the debug symbols (which associate constructs of the programming language with the machine code that the compiler generated from them) will be stripped so as to produce a smaller build product. For iOS applications, it is therefore important to keep a copy of the .dSYM bundle that is generated alongside the binary, as it cannot easily be recovered even if the same set of source files is compiled again. The .dSYM bundle is generated by the dsymutil program, which crafts it from the executable and its intermediate object files (.o, which still contain the DWARF debug information).

Using a binary’s .dSYM, crash reporting services can later map the symbol addresses from the binary to more comprehensible, human-readable symbol names, and even file and line numbers.

Let’s illustrate the difference that symbolication makes using two short examples from stack traces.

// Before symbolication
8 OurApp 0x000029d4 0x1000 + 6612

The first column gives us the index of the stack frame in the stack trace. The second column indicates the name of the binary the function belongs to. The third column tells us the address of the function that was called in the process’ address space. The last column divides this into a base address for the library’s binary image (see section Binary Images) and an offset.

We can see that this is not all that intuitive. Let’s look at the same example after symbolication:

// After symbolication
8 OurApp 0x000029d4 -[OurAppDelegate applicationDidFinishLaunching:] (OurAppDelegate.m:128)

The first three columns are the same. The last column however, contains the actual file name, line number, and function name, which we humans can more easily look up.

The Exception Section

The exception section of a crash report provides us with the exception type, the exception codes, and the index of the thread where the crash occurred:

Exception Type: SIGABRT
Exception Codes: #0 at 0x3466e32c
Crashed Thread: 0

When we talk about ‘exceptions’ in this context, we do not refer to Objective-C exceptions (although those may be reason for a crash), but Mach Exceptions. The example also shows a UNIX signal, SIGABRT, which many UNIX programmers will find familiar. There is a whole ecosystem of APIs built around Mach Exceptions and UNIX signals (e.g. to attach custom signal/exception handlers to given types of UNIX signals/Mach Exceptions), that we’re not going to cover here. If you’re interested, see the Further Reading section below.

The kernel will send such exceptions and signals under a variety of circumstances. For the sake of brevity, we limit our discussion to the most common ones that either lead to process termination or that are otherwise of interest in the face of crash analysis.

Signals

The following is a list of commonly encountered, process-terminating signals and a brief description:

A signal has either a default signal handler, or a custom one (if the program set it up using sigaction). As the second argument to the signal handler, a siginfo_t structure is passed that contains further information about the error that occurred. Of special interest is the si_addr field, which indicates the address at which the fault occurred. The following is a quote of a comment from the kernel’s bsd/sys/signal.h file:

When the signal is SIGILL or SIGFPE, si_addr contains the address of the faulting
instruction. When the signal is SIGSEGV or SIGBUS, si_addr contains the address of
the faulting memory reference. Although for x86 there are cases of SIGSEGV for
which si_addr cannot be determined and is NULL.

Exceptions

On Darwin, UNIX signals are built on top of Mach Exceptions, and the kernel performs some mapping between the two. For a more comprehensive list of exception types, see osfmk/mach/exception_types.h. Again, we list only the most important exception types:

It is also possible for an exception to have an associated exception code that contains further information about the problem. For instance, EXC_BAD_ACCESS could point to a KERN_PROTECTION_FAILURE, which would indicate that the address being accessed is valid, but does not permit the required form of access (seeosfmk/mach/kern_return.h). EXC_ARITHMETIC exceptions will also include the precise nature of the problem as part of the exception code.

Example

The example exception section shown above is an excerpt from this crash report. We can see that the reason for the crash is a SIGABRT, which makes us think that this crash might have been caused by a failing assertion. If we inspect the exception code, we can see that the kernel included the address of the instruction in question (0x3466e32c), and the crashed thread’s index. Sure enough, if we search for that address in the report, we’ll find it in both the program counter register (see below), and the crashing thread’s stack trace:

0 libsystem_kernel.dylib 0x3466e32c ___pthread_kill + 8

In this example, we can see that there’s even more to discover in the ‘Application Specific Information’ section, which tells us that a NSInternalInconsistencyException (a Foundation exception) occurred which was not caught and led to a call to abort(), which is ultimately why we saw the SIGABRT signal.

Binary Images

At the end of a crash report, we find a list of the loaded binary images, which in essence tells us which libraries were loaded by the application, and what their address space is within the process. Each entry in this list also shows the UUID for the respective binary, which is generated and set by the linker as part of the build process. It is stored in the Mach-O binary and identified by the LC_UUID command. The UUID is the same for the binary and the .dSYM bundle generated for it, which ensures that there’s no mismatch during symbolication.

For example, we might find the following in the list of binary images:

0x35f62000 - 0x36079fff CoreFoundation armv7 /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation

The first two hexadecimal numbers indicate the beginning and end of the address space that the CoreFoundation image is loaded into. If we consider a line from a stack trace such as the following, we can see that the function that was traced falls into this library’s address space:

9 CoreFoundation 0x35fee2ad ___CFRunLoopRun + 1269

When is this information useful? Imagine that for some reason we desire to analyze the assembly of one of the libraries that our application is using, let’s say CoreFoundation. CoreFoundation is not statically linked (i.e. it’s not part of the application’s own binary), but is dynamically loaded at runtime. When such loading occurs, the library’s binary image ends up at some arbitrary location in the process’ address space.

Let’s now assume that we know the value of the program counter (PC, i.e. the address of the instruction to be executed next) of the process, and that the PC refers to an instruction somewhere in CoreFoundation’s address space, relative to the process. If on our development machine we disassemble a local, on-disk copy of the CoreFoundation binary that the application previously loaded, we’d not be able to map the process-relative PC to the address space of the local copy, given that CoreFoundation was mapped to some arbitrary address. If, however, we know the offset the CoreFoundation binary image was at during the process’ lifetime, we can easily map the process-relative PC to the corresponding value for the on-disk binary.

As an example, if CoreFoundation’s binary image was loaded into our process with an offset of 0x35f62000, and the PC is 0x35fee2ad, then we can compute the actual address of the CoreFoundation instruction as:

0x35fee2ad - 0x35f62000 == 0x8c2ad

In our locally disassembled CoreFoundation binary, we can now inspect the instruction at address 0x8c2ad.

Register State

Further down in the crash report we find the ARM thread state of the crashed thread, which is essentially a list of the CPU registers and their respective values at the time of the crash. The section may look like so:

Thread 0 crashed with ARM Thread State:
r0: 0x00000000 r1: 0x00000000 r2: 0x00000001 r3: 0x00000000
r4: 0x00000006 r5: 0x3f09cd98 r6: 0x00000002 r7: 0x2fe80a70
r8: 0x00000001 r9: 0x00000000 r10: 0x0000000c r11: 0x00000001
ip: 0x00000148 sp: 0x2fe80a64 lr: 0x3526f20f pc: 0x3466e32c
cpsr: 0x00000010

A crashing thread’s register state is not always required to read a crash report, but there are certainly instances where this information can be very useful. For instance, if the crashing instruction tries to access a register that has a value of 0x0, and the thread tries to access memory at that register’s address or with only a small offset the failure cause is extremely likely to be a NULL dereference. That’s because the entire page from 0-4095 is mapped with read/write/execution permissions disabled, i.e. no access is allowed.

In the following section, we’ll give another more elaborate example.

An Example

Let us now consider a contrived example that illustrates why it can be handy to have the register state of a crashing application.

Assume that the crash report tells us the thread that the crash happened in, and that we’ve identified the line in our program that causes the crash. Imagine that the line reads as such:

new_data->ptr2 = [myObject executeSomeMethod:old_data->ptr2];

If we consider the UNIX signal that got sent (SIGSEGV), we can guess that the crash happened due to the application trying to dereference a memory address that for some reason is invalid. In this example, there are two pointers being evidently dereferenced (new_data and old_data). The question then becomes: Which one is responsible for the crash?

Assembly to the Rescue

If we still have a copy of the crashing binary, we can disassemble it and look at the exact instruction that was being executed when the application crashed (the address of which is made available as part of the SIGSEGV’ssi_addr).

Assume that the application at the time of crashing was executing the following ARM instruction:

str r0, [r1, #4]

There are two registers being used here: r0 and r1. Imagine that r1 points to the address of a C struct with the following declaration:

typedef struct { void *ptr1, void *ptr2 } data_t;

Then, given that pointers on ARM have a width of 4 bytes, we know that r1 plus four refers to the struct memberptr2.

In the form above, the str instruction takes the value stored in register r0 and attempts to store it at the address pointed to by r1, plus four. We could read the assembly like so:

*(r1 + 4) = r0;

That is, we can think of str being the equivalent of a C assignment here.

We now know two things:

• The application received a SIGSEGV (invalid memory access we already knew this; see above).
• The crash happened while trying to store a value to some address with an offset of 4.

At this point we should be able to suspect that the pointer value of new_data might not be what we were expecting it to be.

Looking at the ARM thread state (the registers and their respective values), we can confirm this theory:

r1: 0x00000000

In other words, we can now be certain that we are trying to dereference an address (which was computed based on an offset and a NULL pointer) which in all likelihood points to invalid memory. This was ultimately why the application crashed. If this was a real example, the next step would be to look at our code and determine what path the code could have taken such that we ended up on the crashing line with new_data == NULL, and to fix that.

Additional Notes

To drive the point home, dereferencing old_data can’t be the cause of the crash, given that the access to it only involves reading the value, not writing it (i.e. we wouldn’t be seeing a str instruction). That said, one should not be confused when looking at a more complete list of ARM assembly instructions, where one might see an instruction such as

str [r2, #4], r0

This is an example of how the argument is passed to the subroutine (assuming r2 points to a struct of typedata_t). That is, the value of the second struct member is stored in register r0 prior to jumping into the subroutine, which then performs its work.

When the subroutine is about to return (assuming the Apple ARM ABI), the return value of the function is placed in r0 (if it fits there). In the above case, that would be the value returned by the executeSomeMethod: call, which by the time the crashing instruction executes is already stored in r0.

Conclusion

A fair number of crashes are easy to comprehend. For many crashes, however, comprehensive data is required for crash analysis. In those cases, a reliable crash reporting facility is desirable. Since you can’t know in advance when a crash will occur, and because you might not be able to narrow the cause down without a report, it’s advisable to set up a crash reporting solution for your app early during development. Using iTunes Connect does get you crash reports, but you can’t use it before your app is on the App Store, which means it’s not suitable for beta testing.

Using a service such as HockeyApp has the following advantages over iTunes Connect:

• You can get crash reports even during beta testing, before the app is on the iTunes Store.
• You can access crash reports easily through a convenient web interface.
• You can get notified via mail as soon as a crash happens.
• Crash reports have already been symbolicated for you (assuming proper setup).
• Users often opt-out of Apple’s data gathering, which means you wouldn’t get any crash reports. This is because Apple asks the user for permission to “improve its products and services by automatically sending daily diagnostics and usage data” once when a device is first used, which is a global setting that doesn’t easily convey the effect of declining the request.

It is therefore a good idea to find a crash reporting solution that works for you as soon as you deploy your application. (In the interest of full disclosure: HockeyApp has been a great sponsor of our open-source work on PLCrashReporter).

We hope that we have succeeded in shedding some light on the more advanced information provided by crash reports. In case you need some expert guidance regarding (our) crash reporting solutions, please note that we’re available for hire.

Further Reading

• A WWDC 2010 video on understanding crash reports (login required).
• Apple tech note 2151 on understanding crash reports.
• On Mach Exceptions and Signals, we recommend Amit Singh’s Mac OS X Internals book and Landon’s guest article on Mike’s Friday Q&A blog. As Landon highlights, please note that Mach Exceptions Handlers are a partially private API on iOS.

Prologue: Old Hardware Hacking

In my spare time in the Plausible Lab, I like to play with old Mac and video game hardware — it’s fun, appeals to my strong sense of nostalgia, and if I screw up, I won’t feel quite so terrible as I would if happened to destroy a piece of expensive modern hardware. Fortunately, I’ve yet to actually destroy anything, and actually have been able to fix a few things, like a failed Mac IIci power circuit.

In many cases, you can also find schematics for the hardware in question, if not actual vendor documentation. Official development manuals are available out there if you look around a bit, and crazy folks such as BoMarc Services sell reverse-engineered schematics for everything from the Super Nintendo to the Macintosh Quadra 840av.

This leads me to my small Friday evening project in getting a PC version of the Radeon 7000 64MB working on a PCI Power Macintosh, for which no 64MB cards were released. This requires desoldering, replacing, and reflashing the flash chip on the card, as recently documented by a friend of mine, Rob Braun.

PCI Option ROMs

Have you ever wondered how PCI cards are able to perform basic operations — such as accessing a disk, displaying graphics, or booting off the network — before their drivers have been loaded? Or why cards are platform specific, despite the fact that everyone is using standardized PCI interfaces?

The answer actually lies on the card itself, inside a bit of addressable flash memory called the “Option ROM”, which contains executable code that’s located and run at boot time. This code is responsible for interfacing with the underlying BIOS implementation, and providing the services required for disk, network, graphics, or similar access. Given where and when the code is running, there’s no real constraint — it could also display custom UI, or vend services other than what you might expect, such as a ‘kernel’ that runs entirely from a PCI option ROM.

There are also security implications; malware that acquires sufficient privileges could re-image your PCI cards’ option ROMs to contain persistent code that re-infects the machine no matter how many times you re-image your system. One of the goals of the Secure Boot initiave is to require signing of firmware, preventing this sort of attack (… until another bug is found in the firmware responsible for validating the boot process).

The PCI option ROM is also why PCI cards are platform specific — the ROMs contain either native code, or an architecture neutral bytecode, and that code is targeted to the actual firmware API specification of the host architecture (e.g., BIOS, UEFI, or OpenFirmware).

On legacy Power Macintosh systems, a combination of Open Firmware bytecode (called FCode) and native PPC code is included on the PCI Option ROM to provide *both* boot-time and run-time drivers for cards; this approach of including the OS drivers is what allowed Mac OS graphics cards to be run without additional drivers being installed. On older Macs, the FCode is used at boot time to provide basic services required at boot time, and then the OS loads and uses Mac OS drivers loaded from the PCI option ROM at runtime.

This approach ties the hardware fairly closely to a specific operating system, and while this sort of driver bundling was the norm on Mac OS for years, modern systems tend to rely on installable drivers after boot time.

Converting a Radeon 7000 64MB

This leads us to the original goal, which was to get the PC version of the Radeon 7000 64MB card working on a PCI Macintosh. There was never a 64MB version of the card released, but there *was* a 32MB version of the card provided for Mac OS. Assuming that the drivers written for the 32MB card are compatible with the 64MB card, it should simply be a matter of reflashing the PCI Option ROM with the Mac OS drivers — something we can do using the vendor’s own flashing software.

Unfortunately, there’s just one hitch. The Mac ROM is 128KB, and the PC version of the card only shipped with a 64KB flash ROM. This was presumably done to save money; pennies add up, and the PC ROM only needs to provide basic BIOS services, which can fit easily within 64KB of flash. The Mac ROM actually needs to provide two different drivers; one in FCode, one for Mac OS, and these won’t fit on the PC cards’ smaller flash chip.

Thus, to re-image the card with Mac-compatible firmware, we need to desolder or clip the 64KB Flash chip from the PC card, solder in a new compatible 128KB flash chip, and then reflash the ROM using the vendor’s tools. Here’s the flash chip I replaced:

Replacing the chip turns out to be pretty easy. The original flash chip conforms to an industry standard pinout and package size, for which many compatible replacements are available. The chips are communicated with over the defacto standard SPI Bus (Serial Peripheral Interface Bus), and use a common wire protocol. The protocol itselfuses 24-bit addressing; in theory, there’s no issue with swapping out a smaller 64KB chip for a larger 128KB chip, and having the card address the additional space. I was able to find a compatible replacement for the original Atmel chip from Digi-Key.

The first step is to remove the previous flash chip. If preserving the chip is important to you, you might use hot air rework or similar to lift the chip without damaging it. Given that you can easily copy a PCI option ROM’s contents from a running system, there’s no real need to preserve the actual chip; I went the more destructive approach of clipping the leads using a pair of small flush cutters, and then removing the remaining solder and pins with solder wick.

With the chip removed, the next step is to solder in the replacement. Soldering surface-mount components can be a bit daunting at first, but I’ve found it’s actually quite a bit easier (and less time consuming) than working with through-hole. Lately I’ve been getting the hang of hot air rework using solder paste and the Hakko FM-206 we have in the lab, but for replacing this chip, I wanted to minimize the risk of heat damage to surrounding components.

My approach for resoldering the chip was to use a standard soldering iron with a “hoof” soldering tip. These tips have a broad, flat or concave tip that can be easily dragged over a set of pins. After tacking down opposing corners of the chip, you can simply drag the tip across the pins, letting surface tension wick the solder around the pins. I’d recommend watching the EEVBlog’s video tutorial for more details on hand-soldering surface mount components, including drag soldering.

After dropping the card into my Power Mac 9600, I was able to reflash the new ROM using ATI’s tools. One reboot later, and I could sit in awe of System 7.6 running at 1920×1200 over DVI. Now I just need to get copy of Marathon running…

In my previous post, I described the work benches I built for our new electronics lab at Plausible Labs. With the benches in place, the final missing piece was shelving, and after I spent a long weekend and a few weeknights hanging the shelving, I’m pretty happy with the result:

It’s surprisingly easy to fill a set of shelves, so I wanted to make sure we had a ton of capacity, and that we could easily adjust the shelving positions to fit our needs. I decided to use standard “double slot” shelving tracks, with 31″ x 14″ individual shelves, spanning the full 18′ of the desk surface area.

To ensure sturdy mounting — and simplify leveling of the shelves — I decided to mount a single hanging track from which the shelving standards are hung. To level the track I used a combination of a 4′ level and a standard chalk line reel. There are fancier laser levels available, but I was pretty happy with the results with a simple chalk line. Once the line was in place and level, I marked out the studs using a standard stud finder, to be used when lining up the track for mounting.

The walls were built with metal studs positioned at standard 16″ intervals, so I used sheet metal screws to actually mount the hanging track to the wall studs. This required drilling pilot holes in the studs, but was far less damaging to the drywall than using toggle bolts or similar. The track itself did not perfectly fit the required 18′ of space and needed to be shortened; a standard hacksaw made quick work of it.

Once the track was mounted, I was able to mount the actual shelving standards. I decided to space them out such that every other track is mounted directly into a wall stud, hopefully providing additional support than just the hanging track alone.

Parts List

Plywood & Hardwood

The shelves themselves are simple plywood. For increased strength (and improved aesthetics), you can glue or nail a hardwood strip to the front of the shelves — I haven’t done this yet, but it’s on my TODO list. Really.

I once again hit up our local Lumber City & Building Supply, based in my neighborhood in Williamsburg. These guys are fantastic.

Plywood is generally available in 8′ x 4′ sheets, and with my 31″ x 14″ shelving, I was able to get 9 shelves per sheet. With 18′ to span and 3 levels of shelving, this came to a total of 21 shelves.

Hardware

I purchased all my hardware from McMaster-Carr, including the shelving hardware itself. Although they don’t list a manufacturer in the McMaster-Carr catalog, the shelving hardware I purchased was manufactured by the John Sterling Corporation.

The full parts list follows, with links to McMaster-Carr. The parts list is for a full 18′ x 4′ of shelving, so you’ll need to adapt it for your own needs.

• 80″ hanging track - 3 total
• Shelving Standards, 48″ long - 15 total
• Shelf Brackets, 13 7/8″ Depth - 42 total
• Sheet Metal Screw, No 10 Size, 1-1/4″ Length - 2 boxes. Used to mount hanging track into metal studs and drywall.
• Sheet Metal Screw, No 10 Size, 2″ Length - 2 boxes. Used to mount standards into metal studs and drywall.

If you’re mounting into wood studs, you’ll need to swap the sheet metal screws for standard wood screws. While I re-used the sheet metal screws for mounting into drywall, it should be noted that these aren’t load-bearing, and aren’t a substitute for real anchoring. I’m relying on the hanging track and stud-mounted standards to bear the load, and merely using the sheet metal screws to loosely fasten the remaining standards in place.

Tools

• Power Drill & Bits: The bits depend on the studs you’re mounting the shelving to. I used #10 sheet metal screws, but you’ll want different hardware for wood studs. If you’re planning on using finishing nails to mount hardwood strips to the edge of your shelves, you’ll also need a bit to drill pilot holes.
• Chalk line reel: You’ll need a chalk reel to mark a straight edge for the hanging shelving track.
• 4′ Level: Comes in particularly handy when leveling the hanging track.
• Hacksaw: If you need to cut the hanging track to fit, you’ll find that a hacksaw can make short work of it.
• Quick-grip Bar Clamps: If you’re planning on using a hardwood strip to strengthen the shelves, you’ll need at least two of these to hold the shelves together while you glue and/or use finishing nails to permanently affix the hardwood to your shelves.
• Nail Punch: This is required to drive finishing nails flush or below the surface of the wood, if you plan to affix hardwood to your shelves using finishing nails.

Ever since we founded Plausible Labs in 2008, I’ve felt a mild sense of chagrin at the fact that — despite the ‘Labs’ in our name — we don’t actually have a lab. We’ve decided to finally rectify this oversight by building out an in-office electronics lab — we set aside lab space in our Brooklyn office, and I spent this last weekend building out an 18′ x 3′ electronics workbench to fit:

Given my lack of woodworking experience, I wanted a simple, sturdy design that could be built with only a drill and some simple tools. Thus, the design I used can be built with only screws, metal brackets, and pre-cut lumber, and it’s based entirely on eyeballing David L. Jones’ work benches in his awesome new lab. Any stupid design decisions are, of course, my own. If you haven’t yet discovered David’s EEVBlog, check it out! It’s worth watching!

In terms of design, the 18′ work surface is composed of three individual benches, each topped by 6′ by 3′ of Canadian Birch plywood. I kept the tables separate just in case we ever want to re-organize the room by breaking up the large bench. I also decided to use 30″ high surface, rather than a higher standing/stool height, but I did include adjustable feet that can raise the surface a bit. If we find ourselves really wanting a higher bench, I might have to get a bit clever to modify these.

Each bench is covered with rubber ESD mats, conveniently sold by All-Spec in matching 6′ x 3′ sizes. If you’re not familiar with ESD mats, they’re used to help prevent ESD damage to components by bleeding off any electrostatic charge through their dissipative top layer, via their conductive bottom layer. The rubber surface itself is a tough, heat-resistent material, which is perfect for a work bench on which you’ll be dealing with hot soldering irons, sharp objects, and occasional chemicals. You can see the blue mats on the benches in the photo above, although I’ve yet to attach them to ground.

Here’s the design I sketched out for the benches:

In the final build, I moved the top support beam up to sit flush with the desk surface, to help prevent the plywood from bowing across the rather long 6′ span. Given the number of supports and the size of the lumber, the benches are ridiculously sturdy — possibly more so than they have to be.

If you’d like to replicate these benches (or some variation), I’ve included my parts list below. Additionally, you can browse the gallery of photos I took while building the benches.

Parts List

Lumber & Plywood

You can pick up the lumber and plywood for these benches at your local lumber yard. Here in Brooklyn, I went to Lumber City & Building Supply, based in my neighborhood in Williamsburg. They cut all the wood to size and delivered to our loading dock the next business day, which saved me a ton of work, and meant I didn’t need any power tools besides my drill.

Be aware that lumber uses nominal sizes, which you need to take into account when sketching out your own designs. A 2″ by 4″, for instance, is usually more like 1.5″ x 3.5″.

For each bench as described above, you’ll need the following lumber:

• Legs: 28.5″ x 4″ x 4″ - 4 total
• Side frame: 36″ x 2″ x 4″ - 2 total
• Rear frame: 6′ x 2 ” x 4″ - 2 total
• Top (plywood) surface: 6′ x 36″ x 1.5″ - 1 total

Additionally, if you use a plywood surface, I would recommend attaching veneer edge-banding to cover the rough sides. You can buy pre-glued banding that can be applied with a simple iron — you’ll need enough to cover the circumference of the desk, and I’d suggest a width at least 1/2″ greater than that of your plywood.

Hardware

I purchased all my hardware from McMaster-Carr — it’s like a Lego store for grown-ups. The support structure of the bench is screwed directly together, and attached to the bench surface via L brackets. I also drilled out holes in the legs of the bench so that I’d be able to install leveling mounts (feet).

The full parts list follows, with links to McMaster-Carr. The part counts are for a single 6′ x 3′ work bench.

Bench Hardware:

• Brackets to attach legs to top: 2-1/2″ Length Of Sides, 1-1/2″ Width L Bracket - 4 total
• Phillips Wood Screw, No 8 Size, 1-3/4″ Length - 1 box
• Phillips Wood Screw, No 8 Size, 5/8″ Length - 1 box
• Phillips Wood Screw, No 8 Size, 2-1/2″ Length - 1 box

It should be pretty obvious which screws go where based on their length.

Leveling mount hardware:

• Leveling mounts: 3/8″ #16 x 4″ thread length with 2-3/8″ Base Diameter - 4 total
• Threaded inserts: Surface-Mount Insert for Wood 3/8″ - 16 Internal Thread - 4 total
• Wood Screws: #6 5/8″ length wood screws - 1 box

Tools

• Power Drill & Bits. You’ll need drill bits for all the listed thread sizes, as well a 1″ bit to drill out channels for the leveling mounts. I highly recommend investing in a quick-change bit, as you’ll be switching back and forth quite a bit. I have a cordless Dewalt that worked out great.
• Countersink bits. Used to allow the head of the countersunk screws to sit flush with the wood surface.
• Clothes iron. Used to apply the veneer edge-banding. Any clothes iron will do, but you probably won’t want to use it on your clothes afterwards. I bought a $15 iron off of Amazon.
• Single-cut, mill bastard file. Used to trim the veneer edge-banding.

We often receive inquiries about how Plausible Labs works, what we do here, and what our goals were in deciding to form as a cooperative.

Since people are interested in how we’re organized and what it’s like to work here (more so now that we’re hiring), and our bylaws can be a bit overwhelming to read, I decided to put together a brief FAQ on what our cooperative is, how it’s run, and what our goals are as a company.

How are employees compensated?

Our goal is for salaries to be competitive and market rate. End of year profits are split between the cooperative and it’s members — last year we invested 25% of year-end profits back in the company, and distributed 75% to members. Profit sharing is entirely independent of rank or seniority at the company.

How is the cooperative different from traditional company?

Our main goal in forming the cooperative was to find a way to align employee’s own interests more closely with the corporation’s interests. In terms of legal structure, we’re very similar to a standard corporation — the main differences we’ve adopted to meet this goal are in terms of governance and shareholders.

In a standard stock corporation, the CEO is beholden to the board of directors, and the board is ultimately beholden to the shareholders. The same applies to our co-operative; we have both a yearly-elected board and a CEO, as well as shareholders. However, unlike most stock corporations, only members (employees) of the co-operative may be shareholders, and they may only hold one share.

The board is free to make day-to-day business decisions — while the position of CEO is required by California’s corporate law, the CEO can not act independently of the board. Likewise, there are certain decisions — such as modifying the bylaws — that require a full vote of all shareholders.

Since the board is beholden to the shareholders, and the shareholders may only be employees of the organization, it creates a unique alignment of the corporate, shareholder, and employee interests. Profit sharing is one facet of this — in deciding how to invest the corporation’s resources and time, the board and shareholders must also take into account the employee’s personal interests and the resulting affect on their end-of-year dividends.

What’s the work/life balance like?

It really depends on the state of the company and the member’s own short-term goals, but generally speaking, we stick to a 40 hour work week. Sometimes the opportunity demands more — back in March, we all agreed to fly out to NYC and spent two weeks of 12 hour days working on the Comics iPad application.

What kind of work do you do?

We’re currently focused on contracting work for iPhone and Android, and we’re especially interested in work related to publishing, media, and local/community involvement. We also release a number of our libraries and tools as open source.

While we’re very interested in middleware, tools, and other product development, it requires funding. Our corporate structure requires us to be self-funded; we’ve engaged in contracting to build up our reserve of capital and bring on additional coworkers.

As we’ve grown, we’ve been able to better establish ourselves in the contracting market, charge reasonable rates for our services, and take on larger projects. It’s our goal to leverage this position to hire additional engineers and dedicate part of our contracting profits to product development, allowing us to eventually move the company to being more (but not entirely) product-focused.

How do you become a member/employee? Do you have to buy your share?

After an initial six month candidacy period, our bylaws require that — subject to a consensus vote — we either offer an employee full membership, or extend the candidacy period. If the candidate declines or we do not wish to offer full membership at that time, than the cooperative must either terminate the candidate’s employment or extend the candidacy period.

Each full member must be a share-holder in the cooperative corporation — a capital contribution (currently $1k) purchases the member’s share in the corporation. When a member leaves the cooperative, the coop will automatically redeem this amount in exchange for debt:

Article IX (F): When a member’s membership in the corporation is terminated for any reason, the amount in her or his Member Account (including the original capital contribution) will automatically be redeemed in exchange for debt. The corporation shall repay the debt within five years of the membership termination, with interest accruing at the discount rate — as set by the Federal Reserve Bank of San Francisco — plus two percent (2%), on the amount outstanding at the end of each fiscal year.

No capital contribution is required until a candidate’s application for full membership is offered and accepted, which will not occur until after a six month candidacy period. This is intended to provide the co-op members suitable time to evaluate whether they wish to extend an offer of full membership, and the candidate time to evaluate whether they wish to invest in their share of the co-operative. Like any investment in a corporation, the capital contribution provides a cooperative with additional working capital — however, in the case of Plausible Labs, the current contribution amount of $1,000 is not sufficiently large as to provide a significant percentage of our available capital, but it does help offset a small percentage of the hiring costs.

Some cooperatives — such as Arizmendi Bakery — tend to rely much more heavily on member contributions for working capital, and as such may have higher contribution requirements.

More Questions?

Have a question that isn’t covered here? Feel free to email us at contact@plausiblelabs.com.

Plausible Blocks (PLBlocks) provides a drop-in runtime and toolchain for using blocks in iPhone 2.2+ and Mac OS X 10.5 applications. We’ve started using PLBlocks in shipping iPhone applications, and soon-to-ship Mac OS X applications, and have been working to add support for some key features — the new PLBlocks 1.1-beta2 is the first release to include:

• Objective-C Garbage Collection Support
• C/C++ support

We’ve been using the new beta release for our own day-to-day development for a few weeks now, and hope you’ll help us test out some of the new features. The updated SDK is available for Leopard and Snow Leopard from the PLBlocks project page.

If you’d like to read more about using blocks in your own software, we recommend:

• Joachim Bengtsson’s comprehensive Programming with C Blocks
• Mike Ash’s Series on Blocks and PLBlocks (Part 1 and Part 2)
• My Own “Using Blocks” Series (Parts 1 and Part 2)

iPad Support

We’ve verified that both PLBlocks 1.0 and 1.1-beta2 work with the iPad SDK, and plan on integrating Apple’s beta iPhoneOS compiler updates in a future preview release of PLBlocks for iPad developers.

The new iPad compiler also appears to support the use of blocks on iPhone OS 3.2+; we’ll be very excited to see official block support for the platform. The iPhone OS 3.2 SDK GCC sources can be found at http://opensource.apple.com/tarballs/seeds/.

The Plausible Labs Engineering Team has doubled!

We are extremely pleased (and somewhat giddy) to announce that the incredibly talented Chris Campbell has joined the cooperative. Chris brings with him a decade of experience working at Sun, where he wrote the OpenGL-accelerated backend for Java2D and led the design and development of the Prism graphics engine for JavaFX.

You can learn more about Chris on our About page, or on his blog: http://labonnesoupe.org

We’ve been working our way through our open source TODO list, and we’ve finished up some new releases of our projects that we’re pleased to announce:

PLBlocks 1.0

Plausible Blocks (PLBlocks) provides a drop-in runtime and toolchain for using blocks in iPhone 2.2+ and Mac OS X 10.5 applications. Both the runtime and compiler patches are direct backports from Apple’s Snow Leopard source releases.

The final 1.0 release includes two fixes for issues reported against the beta release:

• Fixed support for using pre-compiled headers with blocks.
• Work-around for rdar://7189835 - Xcode rewrites all occurrence of ‘gcc’ in a
  compiler path when linking using g++

We’re now using PLBlocks 1.0 for our own internal and client projects. The updated SDK is available for Leopard and Snow Leopard from the PLBlocks project page.

If you’d like to read more about using blocks in your own software, we recommend:

• Joachim Bengtsson’s Programming with C Blocks
• Mike Ash’s Series on Blocks and PLBlocks (Part 1 and Part 2)
• My Own “Using Blocks” Series (Parts 1 and Part 2)

PLCrashReporter 1.0

Plausible CrashReporter provides an in-process crash reporting framework for use on both the iPhone and Mac OS X.

• iPhone 3GS-optimized (armv7) binaries
• Mac OS X 10.5+ PowerPC and experimental x86-64 support.

The latest release may be downloaded from the PLCrashReporter project page.

Since crash reports are handled internally to your iPhone application, it supports crash reporting for in-development application versions, allows users to provide additional feedback when submitting a report, and even provides the opportunity to inform users of known issues and the need to upgrade.

If you’d like to include PLCrashReporter in your own application, we recommend perusing some of the open-source usage examples:

• E-mail submission example from Mover. Submits PLCrashReporter reports using iPhone OS 3.0’s MFMailComposeViewController
• Localized HTTP-based submission library and PHP server component from the developers of WorldView. Supports detection known crashes and informing users if the crash is known, or if a fix is available in a later version of the application. Additionally provides a web UI for managing crash reports.

PLDatabase 1.2.1

Plausible Database is an SQL database access library for Objective-C, initially focused on SQLite as an application database. The library supports both Mac OS X and iPhone development.

The new 1.2.1 release includes:

• Support for in-application schema migrations.
• Binary release for Mac OS X 10.5+ x86-64

The latest release may be downloaded from the PLDatabase project page.

PLInstrument 1.0

This is the first release of PLInstrument, a reproducible instrumentation library modeled on xUnit. The library is intended to facilitate the instrumentation of performance critical code, and provide easily comparable results over the lifetime of the code base.

We use PLInstrument to provide reproducible measurements of performance critical sections in our applications and libraries.

The 1.0 release may be downloaded from the PLInstrument project page.

Example Usage

// If for some reason you wanted to measure the runtime of
// CGAffineTransform
- (PLInstrumentResult *) instrumentMirrorTransform {
    PLIAbsoluteTime start, finish;
    int iterations = 25000;

    start = PLICurrentTime();
    for (int i = 0 ; i < iterations; i++) {
        CGAffineTransform mirrorTransform;
        mirrorTransform = CGAffineTransformMakeTranslation(0.0, 200.0f);
        mirrorTransform = CGAffineTransformScale(mirrorTransform, 1.0, -1.0);
    }
    finish = PLICurrentTime();

    return [PLInstrumentResult resultWithStartTime: start
                                           endTime: finish iterations: iterations];
}

Results:

Instrumentation suite 'PLCoreGraphicsDemoInstruments' started at
    2008-12-20 18:51:46 -0800
Instrumentation case -[PLCoreGraphicsDemoInstruments instrumentMirrorTransform]
    completed (0.710000 us/iteration) at 2008-12-20 18:51:46 -0800
Instrumentation suite 'PLCoreGraphicsDemoInstruments' finished at
    2008-12-20 18:51:53 -0800

We’ve just released Plausible Blocks 1.0-beta2, which provides a drop-in runtime and Xcode-compatible compiler for using Blocks in your iPhone 2.2+ and Mac OS X 10.5 applications.

Changes

This release was focused on expanding the supported host and target architectures (thanks to everyone who provided testing!).

• iPhone OS 2.2 and later are now supported.
• The runtime is now available as an iPhone 3gs optimized armv6/armv7 universal binary.
• Development is now supported on PowerPC systems.

Download

The initial beta of Plausible Blocks is available for Leopard and Snow Leopard:

• Leopard: plblocks-1.0-beta2-leopard.dmg
• Snow Leopard: plblocks-1.0-beta2-snowleopard.dmg
• Project page: http://code.google.com/p/plblocks/

This beta release is provided for developer testing and experimentation. Plausible Blocks supports targeting Mac OS X 10.5 (PPC, i386, x86-64), iPhone OS 2.2+ (armv6, armv7), and iPhoneSimulator 2.2+ (i386). Garbage collection and Mac OS X 10.4 are currently unsupported.

Further Reading

If you’re interested in learning more about blocks, here are a few articles to get your started:

• Blocks Examples: NSOperationQueue and UIActionSheet and Using Blocks: Understanding the Memory Management Rules
• Mike Ash’s Friday Q&A on Blocks
• Jim Dovey’s Blocks Life-Cycles and Blocks Episode 1

Introduction

Update Sept 3rd 2009: Check out our more recent posts on PLBlocks.

If you’ve been following the wide variety of developer features planned for Snow Leopard, you may have noticed Apple’s introduction of Blocks, which add closures to C and Objective-C, along with preliminary support for C++.

Blocks are a great addition to Objective-C, but unfortunately, are only available in Mac OS X 10.6. We have a quite a bit of code that could be greatly simplified using blocks, and so I decided to spend some time back-porting block support to iPhoneOS 3.0 and Mac OS X 10.5.

The result, Plausible Blocks, provides a drop-in runtime and Xcode compiler for using blocks in your iPhone and Mac OS X 10.5 applications, based on Apple’s Snow Leopard blocks runtime and compiler support.

Closures are not a new idea — originally conceived nearly 40 years ago, they’re a staple of many languages, from Lisp to JavaScript. If you’ve used a functional language (or a language that borrowed some ideas from one), chances are very good that you’ve made use of closures:

Ruby

books.each { |book| puts (library.to_s + ': ' + book.title) }

JavaScript

function setClickMessage (button, message) {
    button.addEventListener("click", function() {
        /* This is a closure */
        alert(message);
    }, false);
}

Scala

def findBooks (title:String): Seq[Book] = books.filter { book =>
    /* This is a closure */
    book.title == title
}

C and Objective-C Blocks

NSArray *result = [values mapConcurrent: ^(id value) {
    /* Execute closure concurrently on available CPUs, collecting the results */
    return ExpensiveComputation(value);
}];

Closures are well suited to modeling a wide variety of higher-level programming constructs, and in doing so, can greatly simplify your code and enable functionality that would simply be too cumbersome to implement any other way. For some additional information on blocks, and why they’re interesting, I’d suggest reading Mike Ash’s great Friday Q&A on Blocks, and my own Using Blocks post, which includes some sample code to get you started.

Download

The initial beta of Plausible Blocks is available for Leopard and Snow Leopard:

• Leopard: plblocks-1.0-beta1-leopard.dmg
• Snow Leopard: plblocks-1.0-beta1-snowleopard.dmg
• Project page: http://code.google.com/p/plblocks/

This beta release is provided for developer testing and experimentation, and should not be used for production software prior to further testing and review. Plausible Blocks supports targeting Mac OS X 10.5 (PPC, i386, x86-64), iPhone OS 3.0 (armv6), and iPhoneSimulator 3.0 (i386). Garbage collection and Mac OS X 10.4 are currently unsupported, and the SDK requires an Intel Mac.

The Plausible Blocks runtime makes use of custom, prefixed symbol names to avoid any binary conflicts that could occur should Apple add block support to iPhone OS, or when running your binaries on 10.6. The provided compilers are based directly on the Apple standard, stable compiler versions as shipped with the Mac OS X and iPhone SDKs.

Installation & Use

Plausible Blocks is composed of two pieces:

• Plausible Blocks SDK: Supplies custom compilers for use in Xcode.
• PLBlocks Runtime: A runtime library required by all applications making use of Plausible Blocks.

To install the SDK, simply install the included “Plausible Blocks SDK” package. Once installed, an additional “GCC 4.2 (Plausible Blocks)” compiler will be available for selection in your Xcode project and targets. To build with the new compiler, simply select in either your project’s build settings, or on a per-target basis:

In addition to the SDK, you’ll need to include the PLBlocks runtime framework in your application. It is provided as an embeddable framework for Mac OS X, and as a static framework for iPhone applications.

To include in your project:

1. Copy the Mac OS X or iPhone PLBlocks.framework to your project directory
2. Within Xcode, select “Add -> Existing frameworks” and add the copied PLBlocks.framework to your project.
3. Ensure that PLBlocks.framework has been added to your targets “Link Binary With Libraries” section.
4. (Mac OS X Only) Add a new “Copy Files” build phase to copy PLBlocks.framework to your application’s “Framework” directory.
5. Set the project, or a specific target, to use the “GCC 4.2 (Plausible Blocks)” compiler.

Development

The full PLBlocks source code is available from the PLBlocks project page. If you’d like to contribute to the SDK, runtime, or simply discuss programming with blocks, please consider joining the development list at: http://groups.google.com/group/plblocks-devel

To build PLBlocks, select one of the following targets:

• Disk Image: Builds entire project and generates a distribution DMG.
• Package SDK: Builds SDK package, including all compilers, and Xcode plugins
• Runtimes: Builds all runtimes

The project should build on Mac OS X 10.5 and 10.6. In addition to Xcode 3.1.3 or 3.2, the iPhone 3.0 SDK is required.

Building the full SDK, including compilers, may take an extraordinarily long time. Unless you are experimenting with the compiler toolchain — or you are the type to enjoy watching paint dry — building only the Runtime targets during development is highly recommended.