The Right Bus In The Right Place: a Tutorial (Part 2)

A lot of parallel busses are in use today. Some of them are competitive, others are complementary. The misuse of a bus may have dramatic results in a project. This paper explains different profiles of bus use. A system architect should always take into account the basic rules and reduce the risk of bad system behaviour.

This paper is a continuation of "The Right Bus In The Right Place - A Tutorial Part I", published in the Bus Interfacing Issue of Real-Time Magazine, and further explains different profiles of bus use. As explained in the previous article the paper subdivides bus use in different profiles or types of system architectures. To recapitulate: Profile 1 is a simple single processor bus. Profile 2 is a cache supporting bus in a tightly coupled multiprocessor system. Profile 3 is a loosely coupled multiprocessor system with a master - slave Operating System structure. Profile 4 is identical to profile 3 except for a master - master Operating System architecture. Profile 1 and 2 were discussed in the previous issue we will now continue with the other bus profiles. After this some bus profile combinations do's and don'ts are explained.

BUS PROFILE 3: INTELLIGENT I/O BUS

Introduction

A second way of reducing bus load in multiprocessor architectures is to provide for sufficient memory on each board. The code to be executed can reside locally on it. No more instruction fetch bus accesses are needed and most of the data access can be done locally. Only data shared by more than one processor occasions data transfer cycles. (Figure 16)

Figure 16: Profile 3 and 4: execute code on board

Today it is no problem to put enough memory on one single board. One who needs more memory can use a subsystem approach. These can be built with mezzanine technology or with a multiple board approach on a secondary bus (example VSB). The subsystem bus is a profile 1 bus.
Such an architecture is a multiprocessor or better a multi-master system. Depending on the software architecture we have to make a difference between profile 3 and 4.

Bus Definition

Profile 3 bus systems are multiprocessor systems. One subsystem has the main OS. The other subsystems are intelligent I/O subsystems. They may or may not have an OS on board.
If there is one, it is a Real-Time OS (RTOS) chosen by the board manufacturer. A very popular OS for such applications is VRTX, but any other RTOS will do. All depends on the size and the complexity of the I/O problem.
We call it here "Main OS System" and not Master Board or Supervisory OS board. (Figure 17) The VMEbus specifications are using words like master, slave, supervisory and this may lead to confusion. (Figure 18).

Figure 17: Profile 3: OS technology wording

Board and System Functionality

To understand the functionality we need on the board we have to study the typical traffic on a profile 3 bus.
The Master OS board will have to send a command to the intelligent boards. Writing to a dual ported memory on the intelligent I/O board (VMEbus example) or sending a message (Multibus 2) is a solution. The processor of the last board interprets the command, executes it and drops the result in the memory of the Main OS board via dual ported memory (VMEbus) or via an ensemble of data messages (Multibus 2).
To achieve this the intelligent I/O board has to become bus master. Once the result is in place, the intelligent I/O board has to signal the job termination to the Master OS board.

Figure 18: Profile 3: Martin's wording

Board manufacturers provide different ways to do this:

Solution 1: Interrupts are sent from the intelligent I/O board to the main OS board (VMEbus) or a special interrupt message is send (Multibus 2)
Solution 2: The Master OS board uses a mailbox interrupt. The access to this mailbox-interrupt-location from the bus generates an on board interrupt.
Solution 3: Use of (VMEbus) Location Monitors as mailbox interrupts.

We enumerated three essential board functionalities: dual-ported memory, mailbox interrupts or something similar, and the possibility for an intelligent I/O board to launch such an interrupt (be interrupter).
The major problem for a lot of users is the fact that these board functionalities are essential to make such a profile 3 system work correctly but they are not part of the bus specification!
Most people limit themselves to define what I call computer functionality (I need a certain amount of memory, some serial ports, an Ethernet interface, a SCSI interface, etc.). They do not care about the functionality they need to implement correct inter-processor communication protocols in such a particular architecture. This may lead to wrong designs.
A profile 3 bus is a single interrupt handler bus (all interrupts are going from the intelligent I/O boards towards the Main OS board).

Commercial products

As Futurebus+ has been abandoned, the VMEbus is nowadays the most popular solution to provide for a profile 3 bus. VMEbus has 7 hardware interrupt levels.

BUS PROFILE 3BIS: SINGLE MASTER BUS

Introduction

Multiprocessor systems can be built with the intelligent I/O boards which are not capable of becoming bus masters.
Reason 1: The bus has no multiple master capabilities. An example of this is the PC bus (XT/AT). Here all communication between the intelligent I/O channels and the master OS board (Motherboard in PC systems) goes via dual-ported RAM.
Reason 2: One wants to design a system in a profile 3b way to avoid bus collisions. Indeed, if there is only one bus master in the system, this one has to organise the bus traffic. No collisions can occur.

Bus Definition

Profile 3 bis bus systems are profile 3 systems where the main OS board is the only master in the system. The other subsystems are intelligent slave I/O subsystems.

Board and system functionality

The boards must all have dual-ported RAM. An intelligent I/O board must be capable of generating an interrupt. The Main OS board is a single (the only) interrupt handler.

Commercial products

You can use all available busses on the marketplace in this way.

BUS PROFILE 4: LAN IN A BOX BUS

Bus Definition

Profile 4 bus systems are multiprocessor systems in which all subsystems have a main OS. (Figure 19)
This system is comparable with a LAN. However instead of connecting systems (boxes) via (for example) an ethernet, we connect them via a parallel bus. Intel (Multibus 2) introduced the name LAN in a box for this kind of architecture.

Board and system functionality

In such a system, where all communications between the subsystems may go in all directions, it makes no sense to use interrupts. Indeed, in the case you want to use interrupts, each Main OS board should be interrupt handler and each board should be capable of sending interrupts to signal the end of a packet transaction. This is theoretically possible with VMEbus but limited to 7 interrupt handlers. The problem is that you are mixing very short interrupt traffic with packet bulk traffic. This is not a good solution because the interrupt responds largely depends on the arbitration mechanisms used. (VMEbus: 4 arbitration levels (PRI, RRS, PRR, SGL) number of release options (ROR, RWD, FAIR, etc.))

Figure 19: Profile 4 use

Multibus 2, especially designed for such an architecture, has no hardware interrupt scheme. An interrupt message structure replaces it.
On the software side, it is easy to communicate on a parallel bus by just considering the OSI layered structure. Replace the 2 lower layers by the parallel bus. (Figure 20). All applications using TCP/IP or another protocol do not need any modification!

Figure 20: Profile 4 and OSI layers

Commercial products

VMEbus, Multibus2 and Futurebus+ are possible candidates for such an architecture.

SPECIAL FEATURES

Fail Safe - Definition

A bus has fail safe characteristics if there is a provision to continue working with a subset of the available elements in the system. This feature provides for graceful degradation.

Fail Safe: Board and System Functionality

A major element here is redundancy. A system must be built with each board or each function being available at least twice in the system, including a system to decide which element to use in case of failure.
To say this is easier than to implement it. Here we will limit ourselves to some examples.
Example 1: Redundant memory boards. When the same memory board is twice present in the system, both have to load the same data during a write cycle. In synchronous busses this is easy to achieve. In asynchronous busses, a question rises: Which board is going to do the ACK(knowledge). In VMEbus for example there is no solution. VSB provides a special protocol of "responding" and "participating" slaves to achieve this.
Example 2: In multi-master systems, we need a bus arbitration function. If there is only one bus arbiter in the system (like in VMEbus), the failure of this function will result in a failure of the complete system. VSB, Futurebus+, SCSI and IEEE488 use another arbitration scheme: there is no centralised arbiter (every board has an arbiter capable of arbitrating the next bus cycle).
Example 3: The backplane may fail. Is there another bus available? This leads to the construction of double bus systems. This means that the whole bus definition must fit in ONE single connector. This is for example not the case for the XT/AT bus and the VMEbus.

Life Insertion - Definition

A bus has live insertion capabilities if it is possible to insert a board while a bus cycle is going on. This has nothing to do with the insertion of a board with the bus powered on. Most busses will accept this but will corrupt the current bus cycle.

Life Insertion: Board and System Functionality

This functionality must allow maintenance during normal system operation. Therefore, the system will probably have fail-safe elements if live insertion is a necessity. Only Futurebus+ provides the features necessary for this. But as Futurebus+ is not very popular there is considerable effort done to provide such functionality in existing busses like VMEbus.

BUS PROFILE MIXING

Multiple Profile Support

Some busses are so open that they accept the use of it in all kind of architectures.
An example is VMEbus: It can be used in profile 1, 3, 3bis and 4. Most bus specifications do not provide descriptions of possible architectures. It is theoretically possible to mix everything. However, it is not a good practice to mix different types of bus traffic on the same bus. A good design rule is: One bus for each type of traffic. (Figure 21 and Figure 22)
The following advice is applicable:
Profile 1 and 2 used alone or as subsystem bus.
Profile 3 and 4 are mixable but in this case avoid interrupt traffic.
If you mix 1 or 2 with 3 and 4 you have to know what you are doing and have a clear view on the bus load percentage each traffic will produce. A bus analyser tool is needed here.[2]
As stated there is already one reason to have multiple busses in a system: One bus per type of traffic or profile. Another reason may be bus bandwidth and traffic planning.

Bandwidth and Traffic planning

If bus traffic is high, then only multiple data paths can offer a solution. In such a configuration, you should use more than one bus. This is applicable in a crate and also in a subsystem (even if this subsystem is only a single board).
You need careful planning during the design: when distributing the functions or tasks over the boards, consider the induced bus traffic. Rearranging the functions over the boards may considerably reduce the bus traffic. Make a sort of bus flow diagram. You can compute the dataflow on a bus starting from the data rate of incoming signals, taking into account the compression or data treatment that happens on a particular board.

Figure 21: Profile combinations (1)

Always take into account to manipulate and buffer a steady data rate of an incoming channel to cope with the non steady traffic rate on a system bus (profile 3 and 4 bus).
If you want to use the bus in a Real-Time System environment you can imagine to apply Rate-Monotonic Scheduling theory to foresee for predictable bus behaviour. [7] (Bus arbitration and scheduling are similar problems). Software like PERTS can help you here. [1]

CONCLUSION

This article tried to give an overview of how busses can be complementary. Only simple systems will end up having one single bus. Multiple bus use is a characteristic of modern (complex) systems.
Choosing the right board should not only take into account the classical computer functionality but also the bus architecture functionality.
Designing predictable Real-Time Systems should take care about bus behaviour in a rigorous way.

Figure 22: Profile Combinations (2)

Dr. ir. Martin Timmerman is graduated in Tele-communications Engineering at the Royal Military Academy (RMA) Brussels and is Doctor in Applied Science at the Gent State University (1982). He is giving general courses on Computer Platforms and more specific courses on System Development Methodologies at the Royal Military Academy Brussels. Outside RMA, Martin is known for his audits, reviews and workshops on system architectures, Real-Time Operating Systems and software engineering for Real-Time Systems.

REFERENCES

[1] P. KORTMANN and G. MENDAL, PERTS Prototyping Environment for Real-Time Systems Real-Time Magazine 1Q96 (ISSN 1018-0303)
[2] T. NYGAARD, Using bus analysers to find and debug VMEbus problems. Real-Time Magazine 3Q94
[3] M. TIMMERMAN, The Right Bus on the Right Place, Real-Time Magazine, 1Q 1993
[4] M. TIMMERMAN, Bus error flaws in VMEbus Systems, Real-Time Magazine, 2Q 1992
[5] M. TIMMERMAN, Workshops on RTOS - info at Real-Time Consult
[6] J. ZALEWSKY, Advanced Multimicroprocessor Bus Architectures, IEEE Computer Society Press ISBN 0-8186-6327-8
[7] J. ZALEWSKY, What every engineer should know about Rate-Monotonic Scheduling: A tutorial, in [6] and in Real-Time Magazine 1Q 95

Go to the previous part of the article "The Right Bus In The Right Place"
Go back to index Real-Time Magazine 97Q1