Reading CVPixelBuffer in Objective-C

I wanted to inspect individual pixels in a CVPixelBuffer, but I couldn’t find a recipe online, so after figuring it out, here it is. You can adjust this code to iterate through the CVPixelBuffer too if that’s what you need!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
    CVPixelBufferRef pixelBuffer = _lastDepthData.depthDataMap;
   
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
   
    size_t cols = CVPixelBufferGetWidth(pixelBuffer);
    size_t rows = CVPixelBufferGetHeight(pixelBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow( pixelBuffer );
    uint8_t *baseAddress = CVPixelBufferGetBaseAddress( pixelBuffer );
   
    // This next step is not necessary, but I include it here for illustration,
    // you can get the type of pixel format, and it is associated with a kCVPixelFormatType
    // this can tell you what type of data it is e.g. in this case Float32

    OSType type = CVPixelBufferGetPixelFormatType( pixelBuffer);
   
    if (type != kCVPixelFormatType_DepthFloat32) {
        NSLog(@"Wrong type");
    }
   
    // Arbitrary values of x and y to sample
    int x = 20; // must be lower that cols
    int y = 30; // must be lower than rows
   
    // Get the pixel.  You could iterate here of course to get multiple pixels!
    int baseAddressIndex = y  * bytesPerRow + x  * sizeof(Float32);
    const Float32 pixel = *(Float32*)(&baseAddress[baseAddressIndex]);

    CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );

Note that the first thing you need to determine is what type of data is in the CVPixelBuffer – if you don’t know this then you can use CVPixelBufferGetPixelFormatType() to find out. In this case I am getting depth data at Float32, if you were using another type e.g. Float16, then you would need to replace all occurrences of Float32 with that type.

Note that it’s important to lock and unlock the base address using CVPixelBufferLockBaseAddress and CVPixelBufferUnlockBaseAddress.

Python in SPSS

Using Python in SPSS is great if you want to do any complex calculations, without having to leave the SPSS environment. Python is much more flexible than SPSS syntax, and it’s actually very easy to use. It is especially useful when you are collaborating with people who are not willing to do all their analysis in python (e.g. with Spyder), yet require complex data processing steps in their analysis – for example converting between colour spaces. The documentation online is actually pretty good, but I thought I’d post a very simple use case, converting a colour from sRGB to LAB colour space. Doing this in SPSS syntax would be very tiring, but in python I can just cut and paste code into a loop and it’s done. Here is the program:

1
2
DELETE VARIABLES LAB_L LAB_A LAB_B.
OUTPUT CLOSE *.

I start off with some syntax that deletes any variables with the same names as those I am about to create. This is useful especially when developing as you might run the script many times and don’t want to have to delete the variables manually each time.

1
2
3
4
5
6
7
8
9
10
11
BEGIN PROGRAM Python.
import spss


spss.StartDataStep()
datasetObj = spss.Dataset()

# Manipulation of variables goes here!

spss.EndDataStep()
END PROGRAM.

This is the standard boilerplate code needed in most Python SPSS scripts. BEGIN PROGRAM and END PROGRAM determine the area in which you are writing python. You then ‘import spss’ and start a data step. Finally you get a dataset object which allows you to iterate through rows and perform manipulations

1
2
3
4
# Create the new variables
datasetObj.varlist.append('LAB_L',0)
datasetObj.varlist.append('LAB_A',0)
datasetObj.varlist.append('LAB_B',0)

Here I create the new variables, initialised with 0. LAB is a colour space with three dimensions, L, A and B.

1
2
3
4
5
6
7
# Get the variable names
rIndex = datasetObj.varlist['r'].index
gIndex = datasetObj.varlist['g'].index
bIndex = datasetObj.varlist['b'].index
LIndex = datasetObj.varlist['LAB_L'].index
AIndex = datasetObj.varlist['LAB_A'].index
BIndex = datasetObj.varlist['LAB_B'].index

In order to perform data manipulations you need the index of the variable in the dataset. I get all these indexes at the start and store them in their own variables.

1
2
3
4
5
6
7
8
9
10
11
12
for idx, row in enumerate(datasetObj.cases):
    r = row[rIndex]
    g = row[gIndex]
    b = row[bIndex]
    if r >= 0 and g >= 0 and b >=0:
        LAB = calc_LAB(r,g,b)
    else:
        LAB = [None, None, None]
       
    datasetObj.cases[idx, LIndex] = LAB[0]
    datasetObj.cases[idx, AIndex] = LAB[1]
    datasetObj.cases[idx, BIndex] = LAB[2]

Here I iterate each row (or case), pulling the R, G and B values into python variables, using the variable indexes (i.e. rIndex, gIndex and bIndex). Then, if each is above 0, I send them to a function calc_LAB (which I will define later). Finally I take the LAB values and put them into the correct place in the dataset.

You can see how powerful this can be, calc_LAB is actually a lengthy function that would be a real chore to program in syntax. Here is the full program with the function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
* Encoding: UTF-8.
*Importing all of the data.
DELETE VARIABLES LAB_L LAB_A LAB_B.
OUTPUT CLOSE *.
BEGIN PROGRAM Python.
import spss


spss.StartDataStep()
datasetObj = spss.Dataset()

# Create the new variables
datasetObj.varlist.append('LAB_L',0)
datasetObj.varlist.append('LAB_A',0)
datasetObj.varlist.append('LAB_B',0)

# Get the variable names
rIndex = datasetObj.varlist['r'].index
gIndex = datasetObj.varlist['g'].index
bIndex = datasetObj.varlist['b'].index
LIndex = datasetObj.varlist['LAB_L'].index
AIndex = datasetObj.varlist['LAB_A'].index
BIndex = datasetObj.varlist['LAB_B'].index


def calc_LAB(R,G,B):
    var_R = float(R) / 255.0
    var_G = float(G) / 255.0
    var_B = float(B) / 255.0


    if ( var_R > 0.04045 ):
        var_R = pow(( ( var_R + 0.055 ) / 1.055 ), 2.4)
    else:                  
        var_R = var_R / 12.92;
    if ( var_G > 0.04045 ):
        var_G = pow(( ( var_G + 0.055 ) / 1.055 ), 2.4)
    else:
        var_G = var_G / 12.92;
    if ( var_B > 0.04045 ):
        var_B = pow(( ( var_B + 0.055 ) / 1.055 ), 2.4)
    else:
        var_B = var_B / 12.92

    var_R = var_R * 100
    var_G = var_G * 100
    var_B = var_B * 100


    X = var_R * 0.4124 + var_G * 0.3576 + var_B * 0.1805
    Y = var_R * 0.2126 + var_G * 0.7152 + var_B * 0.0722
    Z = var_R * 0.0193 + var_G * 0.1192 + var_B * 0.9505

    var_X = X / 95.047
    var_Y = Y / 100.000
    var_Z = Z / 108.883

   
    third = 1.0/3.0
   
    if ( var_X > 0.008856 ):
        var_X = pow(var_X,third)
    else:
        var_X = ( 7.787 * var_X ) + ( 16.0 / 116.0 )
    if ( var_Y > 0.008856 ):
        var_Y = pow(var_Y,third)
    else:
        var_Y = ( 7.787 * var_Y ) + ( 16.0 / 116.0 )
    if ( var_Z > 0.008856 ):
        var_Z = pow(var_Z,third)
    else:
        var_Z = ( 7.787 * var_Z ) + ( 16.0 / 116.0 )


    L = ( 116 * var_Y ) - 16
    A = 500 * ( var_X - var_Y )
    B = 200 * ( var_Y - var_Z )


    return [L, A, B]


for idx, row in enumerate(datasetObj.cases):
    r = row[rIndex]
    g = row[gIndex]
    b = row[bIndex]
    if r >= 0 and g >= 0 and b >=0:
        LAB = calc_LAB(r,g,b)
    else:
        LAB = [None, None, None]
       
    datasetObj.cases[idx, LIndex] = LAB[0]
    datasetObj.cases[idx, AIndex] = LAB[1]
    datasetObj.cases[idx, BIndex] = LAB[2]

spss.EndDataStep()
END PROGRAM.

Draggable Piechart JS class

For a recent project I made a draggable piechart. I thought this might be useful, so I have made it open source for others to use: https://github.com/jamesalvarez/draggable-piechart

Here is the default example:

Your browser is too old!

The default behaviour gives each slice a minimum width. You can change colours and set it to allow collapsing pies:

Your browser is too old!

However you need to provide a way to uncollapse the segments, as below. You can also override the drawing routines, providing your own format arguments for each segment, and create a UI for it to interact with:

Your browser is too old!

Random non-overlapping circles in MATLAB

For a recent project I had to generate a stimulus set consisting of random non-overlapping circles in MATLAB.  My solution was to keep a list of the outside of a gradually increasing outer ring of circles, and add new ones, expanding the ring.  This requirement is similar to circle-packing, except I needed to have gaps between the circles.

If you use especially large gaps, or the ratio between your maximum and minimum circle sizes is great, I detected the occasional overlap.  But for the parameters I needed, I tested over 10000 tests of 400 circles, with no overlaps.

Here is how it works:

1
2
3
4
5
6
% Parameters for the circles
setup.min_circle_radius = 15;
setup.max_circle_radius = 20;
setup.min_gap = 10;
setup.max_gap = 20;
setup.n_circles = 400;

First I set up some parameters – minimum and maximum circle radiuses and gaps, plus the number of circles I want to generate.

1
2
3
4
5
6
7
8
9
10
11
12
function [ circles ] = CirclePacking( setup )
    % Start with a circle in the middle
    first_circle = generate_random_radius_circle( setup );
    first_circle.position = [ 0, 0 ];
 
    % Add second circle at random angle
    second_circle = generate_random_radius_circle( setup );
 
    gap = randi([setup.min_gap, setup.max_gap]);
    total_distance = first_circle.radius + gap + second_circle.radius;
    [x, y] = pol2cart(2 * pi * rand, total_distance);
    second_circle.position = [x, y];

Next I begin with two circles, one in the centre, and another at a random angle from the first.

1
2
3
function [circle] = generate_random_radius_circle( setup )
    circle.radius = randi([setup.min_circle_radius, setup.max_circle_radius]);
end

I represent circles as structs, first defined solely by radius, but then by position. All circles are first generated using this function which makes a random radius.

Back to the main function:

1
2
3
4
5
6
7
8
9
first_circle.before = 2;
first_circle.after = 2;
second_circle.before = 1;
second_circle.after = 1;

% Setup doubly linked list
first_circle.id = 1;
circles(1) = first_circle; % have to set first circle manually
circles = save_circle(second_circle, circles);

I set up a doubly linked list, that is, each circle keeps track of the circle before and after it. With just two, before and after are the same.

1
2
3
4
5
6
function [circles, circle] = save_circle(circle, circles)
    if (~isfield(circle, 'id'))
        circle.id = size(circles,2) + 1;
    end
    circles(circle.id) = circle;
end

‘save_circle’ adds or saves a circle to the main circles array. I couldn’t work out after a google search how to initialise an empty array of structs, so I didn’t waste time doing so.

Back to main function:

1
2
3
4
current_position = 1;
for i = 1:setup.n_circles
    [circles, current_position] = add_new_circle( setup, circles, current_position );
end

To finish the main function, I just call ‘add_new_circle’, maintaining a variable ‘current_position’ which points to a particular circle in the circles array.

This is the end of the main function, most of what happens is going on inside ‘add_new_circle’, but before I go into it, I will show a few more important helper functions:

1
2
3
4
5
6
7
8
9
10
function circles_overlap = circles_overlap(circle1, circle2, min_gap)
    dist = edist(circle1.position, circle2.position);

    circles_overlap = dist < circle1.radius + circle2.radius + min_gap;
end

function edist = edist(a, b)
    % euclidean distance
    edist = sqrt((a(1)-b(1))^2+(a(2)-b(2))^2);
end

The above functions are self explanatory, just checking whether circles overlap, and a function for euclidean distance.

1
2
3
4
5
6
7
function next_circle = next_circle(circle, circles)
    next_circle = circles(circle.after);
end

function previous_circle = previous_circle(circle, circles)
    previous_circle = circles(circle.before);
end

The above functions allow me to get the next and previous circles – this is not necessary really, but I find it’s easier on the eyes using these functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
function new_circle = poisiton_new_circle(new_circle, circle1, circle2, gap1, gap2)

    distance1 = new_circle.radius + gap1 + circle1.radius;
    distance2 = new_circle.radius + gap2 + circle2.radius;
    distance_between = edist(circle1.position, circle2.position);

    % Calculate angle to new circle
    angle = get_law_cosines_angle_a(distance1, distance2, distance_between);

    % Calculate angle of first two circles
    vector_first_to_second = circle2.position - circle1.position;
    angle_first_to_second = atan2(vector_first_to_second(2), vector_first_to_second(1));

    angle_of_new_from_first = mod(angle_first_to_second + angle, 2 * pi);
    [x, y] = pol2cart(angle_of_new_from_first, distance1);
    new_circle.position = [x, y] + circle1.position;
end

function angle = get_law_cosines_angle_a(ab,bc,ca)
    num = (ab * ab) + (ca * ca) - (bc * bc);
    den = 2 * ab * ca;

    angle = acos(num / den);
end

The ‘position_new_circle’ function puts a circle next to circle1 and circle2, taking into account the gaps passed in too. It always results in the anticlockwise order circle1, new_circle, circle2. It is using the law of cosines to do this – I won’t explain the maths here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
function [circles, first_circle_index] = add_new_circle( setup, circles, first_circle_index )
    % circles must contain 2 or more circles

    gap_with_first = randi([setup.min_gap, setup.max_gap]);
    gap_with_second = randi([setup.min_gap, setup.max_gap]);

    first_circle = circles(first_circle_index);
    second_circle = next_circle(first_circle, circles);
    new_circle = generate_random_radius_circle(setup);

    % Check backwards for overlaps
    for i = 1:8;

        new_circle = poisiton_new_circle(new_circle, first_circle, second_circle, gap_with_first, gap_with_second);
        before_circle = previous_circle(first_circle, circles);
        number_to_check = min(size(circles,2) ,8);

        overlaps = false;

        for j=1:number_to_check
            % Check if circle overlaps
            if (before_circle.id == second_circle.id)
                break;
            end

            if (circles_overlap(new_circle, before_circle, setup.min_gap))
                first_circle = before_circle;
                first_circle_index = first_circle.id;
                overlaps = true;
                break;
            end

            before_circle = previous_circle(before_circle, circles);
        end

        if (~overlaps)
            break;
        end
    end

    % Update linked list
    circles = update_list(circles, first_circle, second_circle, new_circle);
end

This is the main part of the algorithm that works out where to put a new circle. It starts by setting the gaps, and taking the key circle, and the circle after that. ‘first_circle_index’ indexes the key circles, all circles are generated around this circle, until overlaps. Often, a new circle will overlap one of the previous circles in the chain. When this occurs I make the overlapping circle the new key circle, and generate circles around that.

I found that checking 8 circles back was more than enough (on the idea that you can fit roughly 6 circles around another circle – if they are similar size) – but this can be modified to check all circles, at a performance hit.

So after positioning a circle between the key circle and the next one, I check circles before the key (making sure I don’t check the same circle). If the circles overlap, I set the key circle to the one that overlaps.

If there are no overlaps, then I update the linked list with that circle.

1
2
3
4
5
6
7
8
9
10
11
function circles = update_list(circles, circle1, circle2, new_circle)
    new_circle.before = circle1.id;
    new_circle.after = circle2.id;
    [circles, new_circle] = save_circle(new_circle, circles);

    circle1.after = new_circle.id;
    circles = save_circle(circle1, circles);

    circle2.before = new_circle.id;
    circles = save_circle(circle2, circles);
end

Note this can very obviously be massively sped up – but remembering that ‘premature optimization is the root of all evil’ – I have not yet needed to so.

A simple introduction to Core Audio

I had to learn Core Audio for a project recently, which despite being notoriously difficult, has been great fun. Starting out, I would have killed for a basic example audio player that didn’t have any unnecessary bells and whistles to just get the basics.  I ended up creating this project, which specifically does two things:
 
Loads an audio file entirely into memory: 
The audio files for my project were very short, but many could be playing at the same time, which means that loading them from disk could be a bottleneck.
Plays the loaded data with the Audio Unit API: 
This is the most low-level way to play audio, thus offers the most control and lowest latency.
 
I have created a project which does this here: 
https://github.com/jamesalvarez/iosCoreAudioPlayer

The code is very straightforward once you become familiar with the API, but in this post I’ll go over the above tasks, with some extra notes which could be useful.  You need to know basic audio processing terms like samples, channels, frames etc, as well as the C language.

 

AudioStreamBasicDescription

This struct contains information that defines a specific format for an audio data stream.  It can get very complicated, but thankfully you can just define the one you want and load the data and play it back using this format.  I chose to use interleaved floats, which means the data comes in a single buffer of floats, with the left and right channels alternating – thus the two samples per stereo frame are always next to each other.  Non-interleaved means you get separate data buffers for the left and right channels.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#define CAP_SAMPLE_RATE 44100
#define CAP_CHANNELS 2
#define CAP_SAMPLE_SIZE sizeof(Float32)

AudioStreamBasicDescription const CAPAudioDescription = {
    .mSampleRate = CAP_SAMPLE_RATE,
    .mFormatID = kAudioFormatLinearPCM,
    .mFormatFlags = kAudioFormatFlagIsFloat,
    .mBytesPerPacket = CAP_SAMPLE_SIZE * CAP_CHANNELS,
    .mFramesPerPacket = 1,
    .mBytesPerFrame = CAP_CHANNELS * CAP_SAMPLE_SIZE,
    .mChannelsPerFrame = CAP_CHANNELS,
    .mBitsPerChannel = 8 * CAP_SAMPLE_SIZE, //8 bits per byte
    .mReserved = 0
};

When using other data formats you can sometimes have more than one frame per packet, but here this is not the case so the values are straightforward to fill out, using the size of Float32.

 

ExtAudioFile

It takes quite a few lines to load a file with ExtAudioFile, but the result is that you get your data in whatever format you like.  In the Github example, I add error checking but here for more clarity I will just call the functions without checking that they were successful. When dealing with large audio files, it is better to use a ring buffer, where you load more data from the file into memory as it is played rather than loading it all at once. For short files it’s ok to load them completely as I do here.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
ExtAudioFileRef audioFile;

// Open file
ExtAudioFileOpenURL(url, &audioFile);

// Get files information
AudioStreamBasicDescription fileAudioDescription;
UInt32 size = sizeof(fileAudioDescription);
ExtAudioFileGetProperty(audioFile,
    kExtAudioFileProperty_FileDataFormat,
    &size,
    &fileAudioDescription);

// Apply audio format
ExtAudioFileSetProperty(audioFile,
    kExtAudioFileProperty_ClientDataFormat,
    sizeof(CAPAudioDescription),
    &CAPAudioDescription);

The first command is to load the audio file with a URL from a CFURLRef, which bridges directly from a NSURL*.  Next we get the the AudioStreamBasicDescription of the file.  We don’t use this for any other purpose than to calculate the length of the file in frames when allocating buffers to load the file into.  Next we set our predefined AudioStreamBasicDescription on the file, so now when we request data, it will come in this format.

 

1
2
3
4
5
6
7
8
9
10
// Determine length in frames (in original file's sample rate)
UInt64 fileLengthInFrames;
size = sizeof(fileLengthInFrames);
ExtAudioFileGetProperty(audioFile,
    kExtAudioFileProperty_FileLengthFrames,
    &size,
    &fileLengthInFrames);

// Calculate the true length in frames, given the original and target sample rates
fileLengthInFrames = ceil(fileLengthInFrames * (CAPAudioDescription.mSampleRate / fileAudioDescription.mSampleRate));

Here, we get the number of frames of the file (in the original sample rate), and calculate the number of frames for the file using our desired sample rate and the original sample rate.

 

1
2
3
4
5
6
7
8
9
10
11
// Prepare AudioBufferList: Interleaved data uses just one buffer, non-interleaved has two
int numberOfBuffers = 1;
int channelsPerBuffer = CAPAudioDescription.mChannelsPerFrame;
int bytesPerBuffer = CAPAudioDescription.mBytesPerFrame * (int)fileLengthInFrames;

AudioBufferList *bufferList = malloc(sizeof(AudioBufferList) + (numberOfBuffers-1)*sizeof(AudioBuffer));

bufferList->mNumberBuffers = numberOfBuffers;
bufferList->mBuffers[0].mData = calloc(bytesPerBuffer, 1);
bufferList->mBuffers[0].mDataByteSize = bytesPerBuffer;
bufferList->mBuffers[0].mNumberChannels = channelsPerBuffer;

Here we create an AudioBufferList, which is used to store the loaded audio. Before doing so we need to know the number of buffers, the number of channels per buffer and the number of bytes per buffer. Since we are using interleaved data, we only need one buffer, which contains two channels. The number of bytes is simply the number of bytes per frame times the file length in frames. The Github example contains more complex code which can handle non-interleaved data – also in this snippet I have excluded checking to see if the malloc and calloc are successful – just for clarity.

 

1
2
3
4
5
// Create a stack copy of the given audio buffer list and offset mData pointers, with offset in bytes
char scratchBufferList_bytes[sizeof(AudioBufferList)];
memcpy(scratchBufferList_bytes, bufferList, sizeof(scratchBufferList_bytes));
AudioBufferList * scratchBufferList = (AudioBufferList*)scratchBufferList_bytes;
scratchBufferList->mBuffers[0].mData = (char*)scratchBufferList->mBuffers[0].mData;

Next we create a second AudioBufferList which is a copy of the first. This is used to load the data in piece by piece. After loading in a chunk of data, the pointers on the scratchBufferList are changed to point to the next section of data on the heap, ready to load the next chunk of data. I copied this technique from the excellent TAAE library.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// Perform read in multiple small chunks (otherwise ExtAudioFileRead crashes when performing sample rate conversion)
UInt32 readFrames = 0;
while (readFrames < fileLengthInFrames) {
    UInt32 framesLeftToRead = (UInt32)fileLengthInFrames - readFrames;
    UInt32 framesToRead = (16384 < framesLeftToRead) ? framesLeftToRead : 16384; // Set the scratch buffer to point to the correct position on the real buffer
    scratchBufferList->mNumberBuffers = bufferList->mNumberBuffers;
    scratchBufferList->mBuffers[0].mNumberChannels = bufferList->mBuffers[0].mNumberChannels;
    scratchBufferList->mBuffers[0].mData = bufferList->mBuffers[0].mData + (readFrames * CAPAudioDescription.mBytesPerFrame);
    scratchBufferList->mBuffers[0].mDataByteSize = framesToRead * CAPAudioDescription.mBytesPerFrame;

    // Perform read
    ExtAudioFileRead(audioFile, &framesToRead, scratchBufferList);

    // Break if no frames were read
    if ( framesToRead == 0 ) break;

    readFrames += framesToRead;
}

Now we loop whilst reading in frames to the scratch buffer list, which is updated each time to point to the next section of data. There is a maximum number of frames, this seems to be the done thing, I’m not 100% sure why.

 

1
2
3
4
5
6
// Clean up
ExtAudioFileDispose(audioFile);

// BufferList and readFrames are the audio we loaded
audioPlayer->bufferList = bufferList;
audioPlayer->frames = readFrames;

The last thing is to clean up, which just consists of calling ExtAudioFileDispose. I save bufferList and readFrames, to a custom struct which will be used later during the render callback when playing audio.

 

Audio Unit Output

It takes slightly fewer lines to set up the most basic stream for output using Audio Units. Since we only have one Audio Unit we don’t need to use a graph or anything like that. We simply create an output component, set the stream to the correct format, set the render callback and switch it on.

 

1
2
3
4
5
6
7
8
9
10
11
// Description for the output AudioComponent
AudioComponentDescription outputcd = {
    .componentType = kAudioUnitType_Output,
    .componentSubType = kAudioUnitSubType_RemoteIO,
    .componentManufacturer = kAudioUnitManufacturer_Apple,
    .componentFlags = 0,
    .componentFlagsMask = 0
};

// Get the output AudioComponent
AudioComponent comp = AudioComponentFindNext (NULL, &outputcd);

In this first step we create an AudioComponentDescription, which is a struct that describes a particular Audio Unit. In this case, we choose type: kAudioUnitType_Output and sub type: kAudioUnitSubType_RemoteIO to get the AudioUnit which deals in outputting audio to the device. AudioComponentFindNext finds this AudioComponent, so we can begin to use it.

 

1
2
3
4
5
6
7
8
9
10
11
// Create a new instance of the AudioComponent = the AudioUnit
// outputUnit is type AudioUnit
AudioComponentInstanceNew(comp, &outputUnit);

// Set the stream format
AudioUnitSetProperty(outputUnit,
    kAudioUnitProperty_StreamFormat,
    kAudioUnitScope_Input,
    0,
    &CAPAudioDescription,
    sizeof(CAPAudioDescription));

In this step, we create a new instance of the AudioComponent, which gives us the AudioUnit itself, and then we set the stream format using the same stream as we set the file we loaded. This makes it easy when it comes to filling the buffers as no conversion is needed.

 

1
2
3
4
5
6
7
8
9
10
11
12
// Set the render callback
AURenderCallbackStruct callBackStruct = {
    .inputProc = CAPRenderProc,
    .inputProcRefCon = player
};

AudioUnitSetProperty(outputUnit,
    kAudioUnitProperty_SetRenderCallback,
    kAudioUnitScope_Global,
    0,
    &callBackStruct,
    sizeof(callBackStruct));

Here we create a struct that contains the name of our render callback ‘CAPRenderProc’ and void* pointer to anything we like, that will be passed in each time the callback is called. I created a struct which amongst other things points to the buffer of data that was loaded earlier.

 

1
2
3
4
5
// Initialize the Audio Unit
AudioUnitInitialize(outputUnit);

// Start the Audio Unit (sound begins)
AudioOutputUnitStart(outputUnit);

Finally we initialize the audio unit and start it. This will begin calling the render callback for new audio data.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
static OSStatus CAPRenderProc(void *inRefCon,
    AudioUnitRenderActionFlags *ioActionFlags,
    const AudioTimeStamp *inTimeStamp,
    UInt32 inBusNumber,
    UInt32 inNumberFrames,
    AudioBufferList * ioData) {

    CAPAudioOutput *audioOutput = (CAPAudioOutput*)inRefCon;
    CAPAudioPlayer *audioPlayer = &audioOutput->player;

    UInt32 currentFrame = audioPlayer->currentFrame;
    UInt32 maxFrames = audioPlayer->frames;

    Float32 *outputData = (Float32*)ioData->mBuffers[0].mData;
    Float32 *inputData = (Float32*)audioPlayer->bufferList->mBuffers[0].mData;

    for (UInt32 frame = 0; frame < inNumberFrames; ++frame) {
        UInt32 outSample = frame * 2;
        UInt32 inSample = currentFrame * 2;

        (outputData)[outSample] = (inputData)[inSample];
        (outputData)[outSample+1] = (inputData)[inSample + 1];

        currentFrame++;
        currentFrame = currentFrame % maxFrames; // loop
    }

    audioPlayer->currentFrame = currentFrame;

    return noErr;
}

This is the render callback (designed for clarity as performance isn’t really going to be an issue yet!). Here I am just copying interleaved samples from the previously loaded buffer to the output buffer. I have a struct called CAPAudioPlayer which contains the AudioBufferList, the number of frames, and the current frame, which I first extract. Then I set pointers to the two data buffers. Next I loop over the number of frames the callback has requested, copying from the stored AudioBuffer to the output buffer. Finally I updated the currentFrame, so that audio picks up from the correct spot next time the callback occurs.

The project contains more code, especially how to detect errors, dispose of things, and also how to deal with non-interleaved data. This is about the most basic start to using Core Audio, I hope it was useful!