Episode 6a. Exploring the second processor.
Originally I was planning to do "using the 940" as episode 6 in this series. However, since it's such an interesting area there's no way I can cover it in one article. So instead episode 6 will itself be a four part series:
Episode 6a: Getting code running on the second processor
Episode 6b: Practical issues: debugging and interprocessor communication
Episode 6c: Performance analysis
Episode 6d: Complete example
Although the situation is improving, there has been a lot of confusion about the second processor; I've seen it called a "video processor" for example. In reality, it is a complete normal Arm processor that is roughly equivalent to the "main" processor, with two exceptions: First, it has smaller caches (4k vs 16k). That's actually a huge difference and impacts the types of code we'd choose to run on that processor. Second, it does not have a virtual memory system -- it just uses physical memory addresses. It does have a "memory protection unit" but that isn't of much use to us.
In theory, the two processors share exactly the same memory. In practice, the first 32MB of memory is used by linux on the first processor and cannot be easily used by the second processor. That last sentence is not necessarily true depending on your definition of 'easily' and there a few ugly but fascinating tricks we might be able to use in this regard, but I will save that discussion for episode 6c.
Each processor has a special use for memory near address ZERO. Specifically, there is an interrupt table down there, and execution after a reset of the processor starts at location 0. The first processor has been set up during the machine's boot process (otherwise linux wouldn't be running at all). So if the physical addresses on the second processor had exactly the same meaning as the addresses on the first processor, the second processor would try to boot into linux as soon as you started it up! This is because it would start to execute from exactly the same code as the first processor. Although it may seem amusing to try it, it actually wouldn't get very far, primarily because of the differences in virtual memory. The point of this is that both processors should NOT have the same physical memory corresponding to "address 0".
Really what we would like to do for now is make the first 32MB of memory off-limits to the second processor and instead have address ZERO on the second processor be somewhere up in the upper 32MB of memory. Because of this obvious desire, there is a register field in the MMSP2 at register address 0x3B48 that we can set in order to do this. By setting the field to 2 we cause all addresses generated by the 940 to have the value 0x2000000 added to them, which basically moves the memory up to the upper 32MB. We could also just use the topmost 16MB by setting the field to 3 which causes 0x3000000 to be added to each address.
The following pictures illustrates the memory layout:
So that's part of what we have to do to get the second processor going. Here's a list of EVERYTHING we have to do:
Pause the 940, to make sure it isn't running
Code:
// the low bit of register 0x0904 controls whether the 940 is running or not
g_pusRegs[0x0904>>1] &= (~(1<<0));
Put the 940 in a reset state. This is done using a bit in register 0x3B48, which is the same register used for the memory address shifting described above.
Code:
g_pusRegs[0x3B48>>1] = (1<<7) | 2; // (1<<7) for reset, 2 for the memory address shift
Copy the code that we want the 940 to use to the spot where it runs
More on that below
Make sure that interrupts connecting the two processors are disabled, since we won't use them
Code:
g_pusRegs[0x3B40>>1] = 0;
g_pusRegs[0x3B42>>1] = 0;
Take the second processor out of the 'reset' state
Code:
g_pusRegs[0x3B48>>1] = 2; // 2 for the memory address shift
Enable the second processor
Code:
g_pusRegs[0x0904>>1] |= (1<<0);
And that's all there is to it (heh)! Next, let's figure out what code we should copy to the second processor. As mentioned previously, the code near address ZERO has a special meaning. In particular, the first 8 32-bit memory locations are entry points for various interrupts, exceptions, etc, as follows:
Address 0x00: Reset vector
Address 0x04: Undefined Instruction vector
Address 0x08: Software Interrupt vector
Address 0x0C: Prefecth Abort vector
Address 0x10: Data Abort vector
Address 0x14: Reserved
Address 0x18: IRQ vector
Address 0x1C: FIQ vector
We don't really care much about these in particular, but the hardware does so we should should set it up properly.
There's no getting around it -- to do the first baby steps on the 940 we'll have to use assembly language. We won't have to use much assembly though. Here's the beginning of our assembly code to go at address zero on the 940:
Code:
asm ("b .CodeEntryPoint"); // have all the entry points jump to the start of the code
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm (".CodeEntryPoint:");
We could have used other instructions there; like "ldr pc, ..." or just any old no-op for that matter. If we wanted to build a little mini-operating system for the 940 we might be interested in handling the different vectors in a more reasonable way. But we won't be doing anything THAT complicated so this will be fine.
Ok, now when the second processor is enabled and reset, we manage to get it running at our symbol .CodeEntryPoint. Before we go off doing astonishing audiovisual demo effects, there's still some more setup that needs to be done. In particular, we have to set up the memory ranges and initialize our program "stack".
Through a slightly complicated configuration process, we get to decide whether each part of the memory is cached or not, and we have to think carefully about that choice. If memory is cached, it is very fast to access memory that has been recently accessed because it gets loaded into a fast cache on the processor. However, if that memory gets changed it does not immediately get saved to the "main" memory. A simple and basically accurate rule is that any data we wish to communicate off the processor (video memory or variables shared with the other processor for example) should be non-cached and everything else should be cached.
The next step is to map out exactly how we will use the memory, so we know how to configure it. For our purposes here let's just say that the first 16MB of the memory as viewed by the 940 (the physical memory range 32MB-48MB) will be cached. That will include all of the code and normal variables, as well as the system stack which we'll put at 1MB -- 1 meg for code, normal program data, and stack should be plenty for anything we'd do on the 940. The 15 megabytes between 1MB and 16MB are available then for cached data -- maybe we'd write a malloc() system to use it or maybe we'd just divide it up in a fixed way depending on what our program might do.
Then the upper 16MB of memory as viewed by the 940 (physical memory range 48MB-64MB) will be uncached and used for communication with the first processor, and maybe for further data storage if needed.
Finally, we need to do a little more configuration of the chip. The following code takes care of all this. You can go do more research to understand exactly what each line is doing or just take my word for it. the 'mcr' instruction is used to configure "coprocessor 15". The ARM documentation gives more information, though maybe my comments will be of some use.
Code:
asm (".CodeEntryPoint:");
asm ("mov sp, #0x100000"); // set the stack top (1M)
asm ("sub sp, sp, #4"); // minus 4, just to be tidy
// set up memory region 0 -- the whole 4GB address space, uncached
// bit 0 = enable
// bits 1-5: 5-bit region size (N). Size = 2^(N+1). For this partition, set all bits for maximum size
// bits 12:31: base address. Both regions in this example have a base address of zero
asm ("mov r0, #0x3F"); // region data
asm ("mcr p15, 0, r0, c6, c0, 0"); // data region 0
asm ("mcr p15, 0, r0, c6, c0, 1"); // code region 0
// set up region 1 which is the first 16 megabytes.
// bit 0 = enable
// bits 1-5: as above. 16 MB is 2^24 so N should be 23. In hex, 23 is 0x17.
// (0x17<<1) | 1 = 0x2F
asm ("mov r0, #0x2F"); // region data
asm ("mcr p15, 0, r0, c6, c1, 0"); // data region 1
asm ("mcr p15, 0, r0, c6, c1, 1"); // code region 1
// set region 1 to be cacheable (so the first 16M will be cacheable)
asm ("mov r0, #2"); // 2 means region 1 (1 would have meant region 0, 4 would mean region 2, etc)
asm ("mcr p15, 0, r0, c2, c0, 0"); // turn on data cache
asm ("mcr p15, 0, r0, c2, c0, 1"); // turn on instruction cache
// set region 1 to be bufferable too (only data)
asm ("mcr p15, 0, r0, c3, c0, 0"); // turn on write buffer
// set protection on for all regions
asm ("mov r0, #15"); // more than we actually need
asm ("mcr p15, 0, r0, c5, c0, 0"); // set full data permission
asm ("mcr p15, 0, r0, c5, c0, 1"); // set full code permission
asm ("mrc p15, 0, r0, c1, c0, 0"); // fetch current control reg
asm ("orr r0, r0, #1"); // 0x00000001: enable protection unit
asm ("orr r0, r0, #4"); // 0x00000004: enable D cache
asm ("orr r0, r0, #0x1000"); // 0x00001000: enable I cache
asm ("orr r0, r0, #0xC0000000"); // 0xC0000000: async+fastbus
asm ("mcr p15, 0, r0, c1, c0, 0"); // set control reg
// ok, now we're ready to rock
The last thing we need to work out for this article is how to build the program that will run on the 940. There are many ways to do this. We could compile the relevant code as part of our 920 program then copy the relevant parts to the 940, for example. We could get a "generic" Arm compiler and use that to make the executable (I am told that the compiler for the GP32 can be used in this way). The approach I will use here uses the normal gcc for the gp2x from the standard devkit available on the wiki. Just like the rest of the demo code from this series. In fact there isn't a lot of difference in the way it's used -- for this series on demo development we're not using any of the libraries so most of the issues that could arise aren't important. There's really only two issues that we have to deal with.
Issue 1: making sure that the correct code gets put at memory location zero, so it gets run when the processor is reset. Let's make a complete source file for what we have so far, which is just a little stub (saved in code940.c):
Code:
void code940(void)
{
asm ("b .CodeEntryPoint"); // have all the entry points jump to the start of the code
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm (".CodeEntryPoint:");
asm ("mov sp, #0x100000"); // set the stack top (1M)
asm ("sub sp, sp, #4"); // minus 4, just to be tidy
// set up memory region 0 -- the whole 4GB address space, uncached
// bit 0 = enable
// bits 1-5: 5-bit region size (N). Size = 2^(N+1). For this partition, set all bits for maximum size
// bits 12:31: base address. Both regions in this example have a base address of zero
asm ("mov r0, #0x3F"); // region data
asm ("mcr p15, 0, r0, c6, c0, 0"); // data region 0
asm ("mcr p15, 0, r0, c6, c0, 1"); // code region 0
// set up region 1 which is the first 16 megabytes.
// bit 0 = enable
// bits 1-5: as above. 16 MB is 2^24 so N should be 23. In hex, 23 is 0x17.
// (0x17<<1) | 1 = 0x2F
asm ("mov r0, #0x2F"); // region data
asm ("mcr p15, 0, r0, c6, c1, 0"); // data region 1
asm ("mcr p15, 0, r0, c6, c1, 1"); // code region 1
// set region 1 to be cacheable (so the first 16M will be cacheable)
asm ("mov r0, #2"); // 2 means region 1 (1 would have meant region 0, 4 would mean region 2, etc)
asm ("mcr p15, 0, r0, c2, c0, 0"); // turn on data cache
asm ("mcr p15, 0, r0, c2, c0, 1"); // turn on instruction cache
// set region 1 to be bufferable too (only data)
asm ("mcr p15, 0, r0, c3, c0, 0"); // turn on write buffer
// set protection on for all regions
asm ("mov r0, #15"); // more than we actually need
asm ("mcr p15, 0, r0, c5, c0, 0"); // set full data permission
asm ("mcr p15, 0, r0, c5, c0, 1"); // set full code permission
asm ("mrc p15, 0, r0, c1, c0, 0"); // fetch current control reg
asm ("orr r0, r0, #1"); // 0x00000001: enable protection unit
asm ("orr r0, r0, #4"); // 0x00000004: enable D cache
asm ("orr r0, r0, #0x1000"); // 0x00001000: enable I cache
asm ("orr r0, r0, #0xC0000000"); // 0xC0000000: async+fastbus
asm ("mcr p15, 0, r0, c1, c0, 0"); // set control reg
// ok, now we're ready to rock
}
I'll compile this with:
...gcc -O0 -c code940.c
I stick in the -O0 to turn off optimization, just to make sure that the assembly we so carefully constructed doesn't get mangled by some well-meaning optimizer. This makes our code940.o file.
Now we need to make that .o into an executable, so we try to do it with:
...ld code940.o -o code940.gpe
This produces a troublesome looking warning:
ld: warning: cannot find entry symbol _start; defaulting to 00008074
The loader wants to start the program at the symbol _start, but we don't have one! The answer is to tell it
to use our "code940" function as the entry point:
...ld -e code940 code940.o -o code940.gpe
It works!
But should we trust it? let's see what the assembly code actually generated by gcc is, by running
...gcc -O0 -S code940.c
Here's the beginning of the resulting code940.s:
Code:
.file "code940.c"
.text
.align 2
.global code940
.type code940, %function
code940:
@ args = 0, pretend = 0, frame = 0
@ frame_needed = 1, uses_anonymous_args = 0
mov ip, sp
stmfd sp!, {fp, ip, lr, pc}
sub fp, ip, #4
#APP
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
b .CodeEntryPoint
.CodeEntryPoint:
Hey, wait a minute! The compiler stuck in three instructions before the code we so carefully wrote:
mov ip, sp
stmfd sp!, {fp, ip, lr, pc}
sub fp, ip, #4
It did this because code940 is a C function. Luckily, we can tell the compiler to not do that, and here is the final source file after that:
Code:
void code940(void) __attribute__((naked));
void code940(void)
{
asm ("b .CodeEntryPoint"); // have all the entry points jump to the start of the code
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm ("b .CodeEntryPoint");
asm (".CodeEntryPoint:");
asm ("mov sp, #0x100000"); // set the stack top (1M)
asm ("sub sp, sp, #4"); // minus 4, just to be tidy
// set up memory region 0 -- the whole 4GB address space, uncached
// bit 0 = enable
// bits 1-5: 5-bit region size (N). Size = 2^(N+1). For this partition, set all bits for maximum size
// bits 12:31: base address. Both regions in this example have a base address of zero
asm ("mov r0, #0x3F"); // region data
asm ("mcr p15, 0, r0, c6, c0, 0"); // data region 0
asm ("mcr p15, 0, r0, c6, c0, 1"); // code region 0
// set up region 1 which is the first 16 megabytes.
// bit 0 = enable
// bits 1-5: as above. 16 MB is 2^24 so N should be 23. In hex, 23 is 0x17.
// (0x17<<1) | 1 = 0x2F
asm ("mov r0, #0x2F"); // region data
asm ("mcr p15, 0, r0, c6, c1, 0"); // data region 1
asm ("mcr p15, 0, r0, c6, c1, 1"); // code region 1
// set region 1 to be cacheable (so the first 16M will be cacheable)
asm ("mov r0, #2"); // 2 means region 1 (1 would have meant region 0, 4 would mean region 2, etc)
asm ("mcr p15, 0, r0, c2, c0, 0"); // turn on data cache
asm ("mcr p15, 0, r0, c2, c0, 1"); // turn on instruction cache
// set region 1 to be bufferable too (only data)
asm ("mcr p15, 0, r0, c3, c0, 0"); // turn on write buffer
// set protection on for all regions
asm ("mov r0, #15"); // more than we actually need
asm ("mcr p15, 0, r0, c5, c0, 0"); // set full data permission
asm ("mcr p15, 0, r0, c5, c0, 1"); // set full code permission
asm ("mrc p15, 0, r0, c1, c0, 0"); // fetch current control reg
asm ("orr r0, r0, #1"); // 0x00000001: enable protection unit
asm ("orr r0, r0, #4"); // 0x00000004: enable D cache
asm ("orr r0, r0, #0x1000"); // 0x00001000: enable I cache
asm ("orr r0, r0, #0xC0000000"); // 0xC0000000: async+fastbus
asm ("mcr p15, 0, r0, c1, c0, 0"); // set control reg
// ok, now we're ready to rock
}
Give it a try and look at the code940.s produced to verify that it is really what we want
Ok, after the 'ld' command with this new source file, we have a 1064 byte code940.gpe file. Let's peek inside:
...objdump -s code940.gpe
That gives us this:
Code:
code940.gpe: file format elf32-littlearm
Contents of section .text:
8074 060000ea 050000ea 040000ea 030000ea ................
8084 020000ea 010000ea 000000ea ffffffea ................
8094 01d6a0e3 04d04de2 3f00a0e3 100f06ee ......M.?.......
80a4 300f06ee 2f00a0e3 110f06ee 310f06ee 0.../.......1...
80b4 0200a0e3 100f02ee 300f02ee 100f03ee ........0.......
80c4 0f00a0e3 100f05ee 300f05ee 100f11ee ........0.......
80d4 010080e3 040080e3 010a80e3 030180e3 ................
80e4 100f01ee ....
Contents of section .comment:
0000 00474343 3a202847 4e552920 342e302e .GCC: (GNU) 4.0.
0010 3200 2.
Looking at the .text section, you can see the pattern of branches at the beginning. BUT: oh no! The
text section starts at address 8074! That's not what we want! We want it at address 0! So once
again we have to modify the 'ld' command:
Code:
...ld -e code940 -Ttext 0x0 code940.o -o code940.gpe
Now the result of objdump is:
Code:
code940.gpe: file format elf32-littlearm
Contents of section .text:
0000 060000ea 050000ea 040000ea 030000ea ................
0010 020000ea 010000ea 000000ea ffffffea ................
0020 01d6a0e3 04d04de2 3f00a0e3 100f06ee ......M.?.......
0030 300f06ee 2f00a0e3 110f06ee 310f06ee 0.../.......1...
0040 0200a0e3 100f02ee 300f02ee 100f03ee ........0.......
0050 0f00a0e3 100f05ee 300f05ee 100f11ee ........0.......
0060 010080e3 040080e3 010a80e3 030180e3 ................
0070 100f01ee ....
Contents of section .comment:
0000 00474343 3a202847 4e552920 342e302e .GCC: (GNU) 4.0.
0010 3200 2.
That looks good!
Issue 2: making sure that the executable file is in the correct format. By default, the gnu tools for the gp2x generate executables in the 'elf' format -- and a good thing too, because otherwise linux wouldn't know how to load them! But we don't want the elf format for our 940 program, we just want raw instructions.
Luckily, the tool suite contains a program called 'objcopy' which can easily convert the elf format to a raw binary. Just add the following to the makefile after the link line:
...objcopy -O binary code940.gpe code940.bin
Looking at the 116-byte result file, code940.bin, in a hex editor, we see:
Code:
060000ea 050000ea 040000ea 030000ea
020000ea 010000ea 000000ea ffffffea
01d6a0e3 04d04de2 3f00a0e3 100f06ee
300f06ee 2f00a0e3 110f06ee 310f06ee
0200a0e3 100f02ee 300f02ee 100f03ee
0f00a0e3 100f05ee 300f05ee 100f11ee
010080e3 040080e3 010a80e3 030180e3
100f01ee
So the resulting file, code940.bin, is exactly the data that we need to copy over to the second processor. We could read this secondary file off of the SD card and put it there, or we could compile the file directly into our main .gpe executable.
In the next article, I'll take care of that last detail and discuss how the demo program on the main processor can coordinate with the program on the second processor to get some real work done. There's no sample program with part 6a because we're not quite ready to do anything yet, but by the end of 6b we'll have sample code that works. Until then, everybody get to work on your demos for the competition!
If I've made any errors or explained anything poorly, please let me know. I'll also be happy to answer any questions on related topics if I know the answers.
