PPE and SPE memory details

Recently  I have found the clear details on CellBE’s (PPE and SPE) programming limitations in terms memory. So, while programming it is very essential to keep these points in mind in order to avoid future confusions.

• SPE’s local store limit is 256KB.
• Size of each DMA transfer can be at most 16KB. (Means the common data shared between PPE and SPE).
• DMA transfer supports sizes that are 1, 2, 4, 8 or multiples of 16 bytes long.
• The Elemental Interconnect Bus (EIB)’s, which is the I/O between PPE and SPE as well as for main memory, internal bandwidth is 96 bytes per cycle, thus it arrives peak bandwidth of 204GB/s.
• EIB can support more than 100 outstanding DMA memory requests between main memory and the SPEs.

No more Bits: Only Bytes!!!

One way of sharing data between PPE program and SPE program is by using a header file defining a common data structure. The era of counting for bits of memory is comes to end by Cell BE’s byte level memory processing. The data passed between PPE and SPE programs should be in multiples of 16 bytes. And there is also a limit on the variables declared in a SPE program. So, in header file the size of total variables declared should in multiple of 16 bytes. If it is not; we have to follow a work around of adding few dummy variables in order to meet the 16bytes criteria.

For example following is the content of datadef.h header file:

typedef struct
{
/*Array to store the name of the array*/
char genome_file[32];
char output_file[32];

/*Variable to store the total number of input queries*/
unsigned int num_queries;
float aa_ga_weighting[20];
unsigned int dummy[3];

} filexchg_data

In the above sample at the end one dummy integer array is declared inorder to make the total size divisible by 16.

Here is the calculation:

32 + 32 + 4 + (4×20) + 12 = 160 / 16 = 10

Make File Content

Assuming the C program name in the PPU folder of project is knn_ppu.c and SPU folder of the project is knn_spu.c. Following are contents of make file for each programs:

Make File for knn_ppu.c contains the following content:

########################################################################
#            Subdirectories
########################################################################
DIRS        :=     spu
########################################################################
#                       Target
########################################################################
PROGRAM_ppu    :=     knn_ppu
########################################################################
#                       Local Defines
########################################################################
IMPORTS         = ../spu/lib_knn_spu.a -lspe2 -lpthread
INSTALL_DIR    = $(EXP_SDKBIN)/tutorial
INSTALL_FILES    = $(PROGRAM_ppu)
########################################################################
#            buildutils/make.footer
########################################################################
include /opt/cell/sdk/buildutils/make.footer

Make File for knn_spu.c contains the following content:

########################################################################
#            Subdirectories
########################################################################

########################################################################
#                       Target
########################################################################
PROGRAM_spu    :=     knn_spu
LIBRARY_embed     :=    lib_knn_spu.a
########################################################################
#                       Local Defines
########################################################################
IMPORTS        :=     -lm
########################################################################
#            buildutils/make.footer
########################################################################
include /opt/cell/sdk/buildutils/make.footer

Here IMPORTS is the part where we can give the execution options like “-lm” which is used for math functions.

Steps to execute a C program in Cell BE SDK

1. In order to write C projects; first create a folder in the root directory. This folder will work as repository for all the projects.
   • # mkdir CellBE
     
     
2. Then change to CellBE folder [ #cd CellBE ] Then create a folder to represent the project name . For example #mkdir knn_parallel
   
   
3. Then within the new project folder (knn_parallel à #cd knn_parallel) create two more for folders. One folder for ppu (#mkdir ppu) and another for spu (#mkdir spu).
   
   
4. Then write the C program with appropriate make file in ppu and spu folders.
   
   
5. A make file is required to compile each PPU and SPU programs.
   
   
6. Then go to the installed folder of CellBE simulator #cd /opt/ibm/systemsim-cell/bin
   
   
7. Type the following command:
   #export SYSTEMSIM_TOP=/opt/ibm/systemsim-cell
   
   
8. Start the system simulator by typing the following command: #./systemsim –g
   
   
9. In the System simulator window choose the mode of operation to “Fast mode”.
   
   
10. Then compile the C programs using the make file. First compile the SPU C program then the PPU program.
    
    
11. Go to the terminal emulator of Cell BE. Assuming the name of the C program in PPU is hello.c type the following command to execute hello.c:
    • callthru source /root/test/hellp/ppu/hello > hello && chmod +x hello && ./hello

Installing CBE SDK 3 on Fedora 7

Cell Broadband Engine SDK 3 requires Red Hat 5.5 Enterprise edition or Fedora 7. It is better to go with Fedora 7 for educational purpose projects.

Following are the RPMs required (need to be downloaded separately if possible) before starting the installation.
cell-install-3.0.0-1.0.noarch.rpm
CellSDK-Devel-Fedora_3.0.0.1.0.iso
CellSDK-Extras-Fedora_3.0.0.1.0.iso
systemsim-cell-2.1-10.i386.rpm
sysroot_image-3.0-7.noarch.rpm
cross-spu-gcc-static-4.3.0-27.i586.iso

Following are the steps to install Cell Broadband Engine SDK 3:

1. Create a cellsdkiso folder in /tmp folder
   • # mkdir -p /tmp/cellsdkiso
2. Change the current folder to cellsdkiso
   • # cd /tmp/cellsdkiso
3. Download the cell-install RPM files and ISO images from developerWorks or Passport Advantage Web site and place it into the /tmp/cellsdkiso directory that you created in the previous step. Following packages should be needed for Fedora 7
   • cell-install-3.0.0-1.0.noarch.rpm
   • CellSDK-Devel-Fedora_3.0.0.1.0.rpm
   • CellSDK-Extras-Fedora_3.0.0.1.0.rpm
   • cross-spu-gcc-static-4.3.0-27.i586.rpm
   • systemsim-cell-2.1-10.i386.rpm
   • sysroot_image-3.0-7.noarch.rpm
4. Then stop the YUM updater daemon.
   • The status of the YUM updater running can be known using the following command. # /etc/init.d/yum-updatesd status
   • There are few Fedora 9 (for SDK 3.1) distros missing yum-updatesd package by default. So install it manually (yum install yum-updatesd).
   • To stop the YUM updater give the following command: # /etc/init.d/yum-updatesd stop
5. The SDK Installer requires the rsync, sed, tcl, and wget packages. If they are not installed on your system, type the following command: # yum install rsync sed tcl wget
6. Then install the cell-install-3.0.0.1.0.noarch.rpm using the following command.
   • # rpm -ivh cell-install-3.0.0.1.0.noarch.rpm
7. To install from the ISO image type the following commands.
   • # cd /opt/cell
   • # ./cellsdk –iso /tmp/cellsdkiso install

8. The above installation will also download some RPMs from internet. So, setting up internet connection with reasonable speed will be preferred. If the internet connection is slow there is a possibility of ftp time out errors to occur. This is the final stage of installation.
   

Basics of Cell Broadband Engine

Basic information on Cell Broadband Engine is available at the reach of googling it or checking in the wikipedia. Cell BE is an awesome technology created by Sony Entertainment, Toshiba & IBM.

Following links provide the direct access to the information :
http://en.wikipedia.org/wiki/Cell_microprocessor

www.research.ibm.com/cell/

Following is the link giving SDK 3 site

http://www-128.ibm.com/developerworks/power/cell/downloads.html?S_TACT=105AGX16&S_CMP=LP
In this Developer pack & extras package should be downloaded for non-commercial applications

Why is this blog?

This blog is an outcome of my trial and error efforts in Cell Broadband Engine SDK 3 programming. The main problem i have faced in Cell BE programming is “it is not spread widely”. So, there is less number of programming helps available online. This blog is an effort towards solving this problem. And in overall its a Layman’s perspective for a Layman. Comments are welcome.