Porting LinuxBIOS to the AMD SC520: A Follow-up Report

Getting the board Flashed led to some interesting detective work for the LANL team.

As of July 15, 2005, we have moved the arch repository to Subversion. Arch checkouts will continue to work, but any new changes will be available only in Subversion.

It's quiet. Too quiet.

Well, it was too easy. Things were going well with our project to port LinuxBIOS, until we tried to Flash the Flash part. Then we started to hit some problems with the board, the board design and the AMD SC520.

What went wrong? Put simply, when we tried to use the flash_rom program to Flash the part, it failed even to discover the type of part we had on the board. From there, it got worse. We wrote a small program to dump the Flash part, shown here:


#include <errno.h> 

#include <fcntl.h> 

#include <sys/mman.h> 

#include <unistd.h> 

#include <stdio.h> 

#include <string.h> #

include <stdlib.h> 

#include <ctype.h>

main(int argc, char *argv[]) { 

int fd_mem; 

volatile char *bios; 

unsigned long size = 512 * 1024; 

int i;

if ((fd_mem = open("/dev/mem", O_SYNC|O_RDWR)) < 0) { 

  perror("Can not open /dev/mem"); 

exit(1); 

}

bios = mmap(0, size, PROT_READ, MAP_SHARED, fd_mem, 
off_t) (0xffffffff - size + 1)); 

if (bios == MAP_FAILED) { 

perror("Error MMAP /dev/mem"); 

exit(1); 

} 

write(1, bios, 512*1024); 

}
 

When we ran this program, we couldn't get sensible results. This program runs and runs well on everything else we own--several thousand K8 nodes, our laptop, 1,500 Xeon nodes--so it is not the program. What's going on?

As mentioned, we found a problem with the design of the MSM586SEG and the other SC520-based boards from Advanced Digital Logic. The problem, put simply, is the full Flash part cannot be accessed from the CPU; only the top 128KB of the part can be accessed. This limitation requires us to modify all of the tools that we support for Flash access, so they are aware that although the nominal size of the Flash is 256 or 512KB, only 128KB of that space is available.

Making that change, however, still did not help. When we dumped the Flash part, we got not garbage but nonsense. We saw strings that read CCCCoooo and so on. This nonsense led us to think that the Flash space was being cached somehow. In addition, we believed the hardware design had a problem such that burst reads from the Flash part--which would happen if the cache were enabled in the range of memory--were returning the same byte four times, not four consecutive bytes.

Then, we hit some other problems. We had two MSM586SEG boards, and the IDE interface on both of them stopped working. It turns out that the MSM586SEG has an FPGA controlling many functions, and we suspect that this FPGA has some teething problems. We decided to try out the older design, the MSM586SEV, which has no FPGA.

The MSM586SEV resolved all our problems save one: we still got nonsense when we tried to read the Flash. It now was time for some deep-diving into the SC520 architecture. We learned that a set of 16 registers, called the PAR registers, need to be managed in order to enable Flashing the part.

What are the PAR registers? They are used to steer memory and I/O access issued by the CPU. Almost all processors today have a special set of registers in the memory and I/O address generation path to modify the manner in which such addresses are handled.

Why is this type of register needed? With multiple busses capable of supporting memory and I/O access, the processor has no idea where to send the access unless it is told. That is the function of the PAR registers. Consider the block diagram of the SC520 shown below.

Figure 1. SC520 Block Diagram

A given I/O access can go to the PCI bus or to the GP devices shown at right. A memory access can go to SDRAM, the Flash part or the PCI bus. The PAR registers allow the BIOS to specify, for a given range of I/O or memory, which bus it goes to, whether it is writable or read-only and whether it is cached.

We found that for the BIOS range of memory, 0xe0000-0xfffff, the PAR register was set to SDRAM. This setting is not surprising: for performance, the BIOS typically copies the BIOS image to SDRAM and then makes sure all BIOS code fetches go to the SDRAM holding the BIOS. This operation is commonly called "shadowing the BIOS".

Because Linux doesn't use the BIOS at all, we can ignore this setting. What we do is set the PAR register for the BIOS region, PAR register 15, back to the original BIOS. This is a simple matter of mapping in the registers, and then setting the register. Here is a code fragment to do so:


if ((fd_mem = open("/dev/mem", O_SYNC|O_RDWR)) < 0) 

{ perror("Can not open /dev/mem"); exit(1); }

mmcr = mmap(0, 4096, PROT_WRITE|PROT_READ, MAP_SHARED, 
fd_mem, (off_t) 0xfffef000); 

if (mmcr == MAP_FAILED) 

{ perror("Error MMAP /dev/mem"); exit(1); }

p = mmcr + 15; l = *p; printf("l is 0x%lx\n", l);

/* clear cache bits */ 

l |= (1<<27);

/* enable writeable bit */ l

&= ~(1<<26);

/* set type to flash, not sdram */ 

l &= ~(7<<29); l |= (4<<29); 

/* 64k pages */ l

|= (1<<25); 

/* blow away base and size stuff. */ 

l &= ~(0x1fff | (0x7ff<<14)); 

printf("l is now 0x%lx\n", l); 

l |= (8 << 14) | (0x2000000>>16); 

printf("l is now 0x%lx\n", l); 

*p = l; 

Once we had this done, we still had troubles. The further problem was the design of the PAR registers. They live in memory at 0xfffef000; in other words, they are placed right in the middle of the top 2MB of the 4GB of memory space. This space is, by convention, reserved for BIOS Flash, but the SC520 breaks that convention. So, although we had worked around the board problems, we now were faced with an architectural problem.

A light bulb went off at this point, though, relating to comments we had seen in sample code from AMD. The AMD code always was careful to program the PAR registers to place the Flash part above the top of DRAM, that is, at 32MB or hex 0x2000000. We modified our parbios program slightly, and voilà--all 512KB of Flash now was available, starting at 0x2000000.

Th effects of this change are far-reaching. We had to modify our flash_rom program to enable the Flash on the SC520 and place the Flash at an odd location in memory. Nevertheless, at least we can program it now. This change also affects LinuxBIOS itself. If we want to use all of the Flash part, or simply more then 64KB, we're going to have to make a lot of changes to how LinuxBIOS addresses Flash. We've never seen a machine to date that could not address Flash directly at the top of memory.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

MMCR can be moved...

AndrewD's picture

Quote:
"Probably the single biggest problem is the location of the configuration registers, placed right in the middle of the top 2MB of memory"

You can map the MMCR to any 4k offset in the lower 1G by writing to the CBAR register. 0xFFFEF000 is just the default location, so I would not see that as a flaw.

I ended up using rolo instead of Linuxbios on a SC520 based product I designed a few years ago. U-boot is another very powerful option, but they are both more targeted to embedded products.

MMCR can be moved ... but not removed.

Stefan Reinauer's picture

The problem in this case is that even if you "move" the MMCR, it is still sitting at its old position 0xFFFEF000, in addition to the new one.

Code sample

Craig Ringer's picture

The code snippet is broken - presumably due to mangling by a CMS. Given the extreme difficulty I'm having in getting this godawful commenting system to not mangle these code snippets, that's my working theory. LJ staff, please fix your CMS so it doesn's strip spaces in >code<>/code< blocks! I had to use   in a >code<>/code<block, which is (a) gross and (b) should probably not be interpreted as an entity, but rather as a literal.

In addition to the total loss of indenting, I found two other problems. First:


bios = mmap(0, size, PROT_READ, MAP_SHARED, fd_mem,
off_t) (0xffffffff - size + 1));

looks like it should be:


bios = mmap(0, size, PROT_READ, MAP_SHARED, fd_mem,
           (off_t)(0xffffffff - size + 1) );

and:


#include #
include

should evidently be:


#include
#include

With those changes it builds OK here. The dump doesn't look too interesting on this system - probably not a flash image, anyway - but it's an AMD64 box so it's quite likely the flash is mapped to somewhere different. Any idea what address it might be at?

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix