alexrp’s blog

ramblings usually related to software

Epiphany Notes

This page contains various notes on the Epiphany processor architecture that I have collected while writing a simulator for it. Most of this comes from the amazingly helpful folks at Adapteva and the Parallella forums, while other things are from manual trial and error and scavenging through binutils and CGEN.

Many of these things will probably eventually be documented in the architecture manual, so I’m just putting this page up for folks that are looking for this information in the meantime.

Feel free to comment if you spot anything that looks wrong.

Undefined Behavior

First of all, it’s important to realize that the Epiphany architecture actually has undefined behavior. If you come from C, this might not sound so bad, but it actually is. When undefined behavior occurs on an Epiphany chip, anything at all can happen. That means the chip could simply hang, it could branch to some completely arbitrary location, random registers could be overwritten, etc. This sound bad yet? The thing is that Epiphany isn’t designed to be used like the average host CPU architecture; rather, it’s meant to be used as an accelerator platform that the host CPU can use, in a similar vein to GPUs. The host has complete control over everything the Epiphany chip does. For that reason, security and reliable behavior on nonsensical code was not a huge concern when the architecture was designed.

In short, if the chip enters a state of undefined behavior, all bets are off. You should assume everything is broken and write to the RESETCORE register.

Undocumented Features

The architecture has a number of undocumented and/or unsupported features that haven’t been finished or tested fully yet. If you are a regular user of the architecture, you can ignore these. They’re mostly relevant for simulators and other such tools.

Don’t use these and expect them to work until they’re documented. Especially not across architecture generations.

SWI Instruction

The architecture has an undocumented SWI instruction which raises a software exception. It sets bit 1 of ILAT and sets the EXCAUSE bits in STATUS to 0b0001 (for Epiphany III) or 0b1110 (for Epiphany IV).

The instruction is encoded as 0x01E2 which does leave room for an operand of some kind. Interestingly, e-as allows an operand that is exactly 0, and nothing else, and encodes it as if no operand was given.

UNIMPL Instruction

There is an undocumented instruction called UNIMPL which raises a so-called unimplemented exception. It’s not clear where this would be useful, but it is nonetheless there. The instruction has no operands or configuration bits and is simply encoded as 0x000F000F. It sets bit 1 of ILAT and sets the EXCAUSE bits of STATUS to 0b0100 (for Epiphany III) or 0b1111 (for Epiphany IV).

User/Superuser Mode

Bit 2 of the STATUS register is documented as reserved, but actually means user/superuser mode, where it being cleared means user mode. This bit only has significance if (the documented as reserved) bit 25 of the CONFIG register is on, which tells the core to use user/superuser mode at all.

When user/superuser mode is in effect, an interrupt sets bit 2 of STATUS, and an RTI instruction unsets it. Among other things, when user/superuser mode is in effect, user mode is not allowed to issue GIE, GID, and RTI, and is not allowed to access system registers.

Andreas Olofsson of Adapteva has promised to write more documentation on this sometime in January, 2014.

Instruction Encoding

This section documents some peculiarities and undocumented aspects of encoding instructions in Epiphany.

General-Purpose Registers

General-purpose registers are encoded in the obvious way: r0 is 0b0000, r1 is 0b0001, r2 is 0b0010, and so on. The 4 bits leave exactly enough room to encode all 64 registers.

System Registers

It’s not documented how exactly system registers should be encoded for the MOVFS and MOVTS instructions. In Appendix B of the manual, each register has an address. The system registers start at 0xF0400 (CONFIG sits here). To get the register number used when encoding an instruction, take the address of the register, subtract 0xF0400 (the system register base), then divide by 4, and subtract 1.

So:

1
2
int addr = ...;
int reg = (addr - 0xF0400) / 4 - 1;

Doing this, CONFIG becomes 0b0000, STATUS becomes 0b0001, PC becomes 0b0010, etc.

Note that some gaps exist in the system register region. Instructions reading from or writing to these will trigger undefined behavior.

SYNC, WAND, and MBKPT

The encoding for these instructions is not present in the decode table. They are all 16-bit instructions with no operands or configuration bits and are encoded as 0x01F2, 0x0182, and 0x03C2 respectively.

MOV and BL

The conditional variant of MOV has enough bits for encoding the BL (branch and link) condition code. This makes no sense for this instruction, however, and triggers undefined behavior. Note that e-as will actually reject it.

This begs the question whether BL being a condition code really makes sense, but I digress…

LDR/STR (DISP) (16)

This instruction is listed twice in the decode table. The second listing is actually the 32-bit variant.

FLOAT, FIX, and FABS

These instructions are listed as having Rm operands. They of course don’t, since their operate on a single value and store that to a destination register. Simply ignore the bits that claim to be Rm parts.

Bad Instructions

If an instruction does not correctly decode to anything the processor can recognize, undefined behavior results. The manual does claim that a software exception can be raised for invalid instructions, but this is not actually the case.

Memory Access

The Epiphany is rather fragile when it comes to memory - there are many ways to screw up and throw the core into an unpredictable state. This section tries to document all of these cases.

Invalid Memory Access

When an Epiphany core attempts to access some random, unmapped memory that lies outside of any core’s local memory banks and the external memory segment, undefined behavior occurs. In practice, the core will most likely hang, but in any case, it will not behave as intended.

Note that this goes for LDR/STR, DMA, and everything else that might access arbitrary memory.

Registers and Active Cores

The manual isn’t completely explicit about it, but reading from or writing to system registers is legal while the core is active. This is not true of general-purpose registers, and for those, undefined behavior will result. This holds true for reads/writes coming from both the host and other cores.

Memory Faults

The manual says that a memory fault interrupt can be raised on a “memory protection fault”. Don’t be fooled; this interrupt is only raised when local memory banks protected with MEMPROTECT are accessed - not arbitrary, unmapped memory.

TESTSET Semantics

The TESTSET instruction is only atomic with respect to other Epiphany cores in the system. If the host and an Epiphany core both write to the same location (with the host using its equivalent to TESTSET), it will not happen atomically.

Further, TESTSET on a memory location that isn’t within an Epiphany core’s local memory banks results in undefined behavior, even if it’s in the external memory segment.

Interrupt Handling

This section describes some missing details about interrupt handling in the architecture manual.

Interrupt Actions

The diagram in section 7.8.1 of the manual doesn’t give the whole story as to what an Epiphany core does when an interrupt enters the core.

Let N be the interrupt level.

  1. The PC is saved in IRET.
  2. Bit N in ILAT is cleared.
  3. Bit N in IPEND is set.
  4. The GID bit in STATUS is set.
  5. If bit 25 of CONFIG is on, bit 2 in STATUS is set.
  6. PC is set to N * 4 (an index into the IVT).

RTI Actions

Similarly, the manual doesn’t give the whole story on RTI.

Let N be the interrupt level.

  1. Bit N of IPEND is cleared.
  2. The GID bit in STATUS is cleared.
  3. Bit 2 in STATUS is cleared (unconditionally).
  4. PC is set to IRET.

Intuitively, issuing RTI outside of an ISR would make things blow up as there isn’t a current interrupt level. In this case, IPEND is all zero, so it isn’t changed. The instruction will still not do anything terribly useful (though its behavior is well-defined).

Traps and System Calls

The TRAP instruction is included in the architecture so that Epiphany cores can call up into the host system, where something akin to a kernel can service system calls and the like.

Trap Codes

Some trap codes don’t have clear meanings.

Trap codes 0, 1, 2, and 6 were the old write, read, open, and close system calls, respectively. They are no longer used as such (except for a bug relating to close; see below). The manual calls these reserved, but programs are free to use them for their own purposes. That being said, the official simulator will still interpret them as the aforementioned calls.

The two pass and fail trap codes (4 and 5) are primarily intended for testing. The program can issue them when an assertion passes or fails. Note that the program will immediately stop upon issuing one of these (the manual is not clear about this).

System Calls

The manual doesn’t document system call 19, which is gettimeofday, and system call 21, which is link.

System call 3, which is close, is documented but is not actually used by the Epiphany port of Newlib. This is a bug in the port.

ABI Rules

The manual isn’t too clear on the exact ABI surrounding TRAP.

For the deprecated trap codes 0 and 1, r0 was the file descriptor, r1 was the buffer address, and r2 was the length. The deprecated trap code 2 used r0 as file name pointer and r1 as open mode. The deprecated trap code 6 used r0 as file descriptor. Trap code 3 uses r0 as the status indicator, i.e. 0 for success, 1 for error, etc.

Trap codes 0, 1, 2, and 6, as well as all system calls use r3 as the errno register, and r0 as the result register. All other trap codes do not set any result register(s).