Packet representation

Packet representation

The goal of this proposal is mainly to speed up the packet handling and to provide the developers a simpler and more Object Oriented packet representation. It aims to remove the current SocketBuffer and replace it with Packets.

Currently the packets are represented with the SocketBuffer class. The SocketBuffer is something like a dynamic buffer (to be more accurate it is more like a list of dynamic buffers). When a program wants to send a network packet, it creates the protocol headers, and then the headers are inserted in this buffer list with the packet payload (if there is a payload). Finally the NIC driver copies all the bytes of the packet into another fixed array to be used from the driver.
When we send a packet we move all the packet data 2 times (from the headers to the SocketBuffer and from it to the final fixed array). When we receive a packet, the SocketBuffer acts like a fixed array, which provides us with some useful methods to read data from him.

What I suggest is to represent the packets as…Packets. All the protocol packets are the same thing. They have a header and a payload (some of them have a tail too). Every packet is able to know its size and how to handle its data (including the payload).

So let’s say we have the interface Packet.

What we need from the Packet implementers:
- Set and Get methods for the data of the packets (class specific and wont be in the Packet interface)
- A way to store the data in an object and to have a method that will return the entire packet as a byte[] (when we send the packet).
- Methods that will parse a byte array to create the Packet object (when we receive a packet).

Any packet is represented with a class that implements the Packet interface. For example IPv4Packet, ICMPPacket, TCPPacket etc. Every component of the networking can access the data of this packet with the appropriate set and get methods (If a program uses a packet knows it’s specific assessors).

To accomplish the second and the third we need the following methods for the interface Packet:

public void getData(byte[] b);
public void getData(byte[] b, int offset);
public void getSize();

The getData(byte[] b) writes the packet in the given byte array. If the length of this array is not enough we can throw an exception. The getData(byte[] b, int offset) writes a packet to the array b, starting from the offset position. The getSize method will return the length of this packet, including the length of its payload.

These methods will be called mainly from the network drivers. This is the point where the packet is converted from object to a memory buffer and vice versa.

When the Ethernet protocol sends a packet to the driver the driver will call the getSize() of the Ethernet packet to determine how big will be the array that will store the entire packet. The Ethernet packet getSize() method will return for example 14+payload.getSize(). Remember that the payload is also a packet, let’s say an IP packet that may return for example 20+payload.getSize(). As the driver has determine the length of the entire packet, it will create the memory buffer b and it will call the getData(byte[] b) to the Ethernet packet, which will write the first 14 bytes and also it will internally call payload.getData(b, 14). This way we move the data only once, from the objects to the final byte array.

A common class that implements the Packet interface is the RawPacket, which is a packet that maps a byte array, or a portion of this array as packet.

The Raw packet will be mainly used:
- To store the data payload of a packet (for example the data payload of a TCP packet)
- To map a received packet before it will be parsed from the networking components.

A practical example for the second:

When a packet is received from a NIC, the driver will create a RawPacket with the array that stores the currently received frame. Later this RawPacket will be send for example to the Ethernet protocol, which will parse the first 14 bytes to its attributes and create another (or modify the same) RawPacket that will map the same byte[] from the position 15 to the end. This RawPacket will be the payload of this Ethernet packet, that later will be send to the IPv4 stack for example and so on.

Pavlos Georgiadis


May I suggest that one works with ByteBuffers or at least the Buffer interface instead of byte arrays. This will allow for optimisations in the future (particularly when nio gets implemented on top of the packet stuff).

What do you mean about

What do you mean about optimisation in the future? You say "particularly when nio gets implemented on top of the packet stuff".

Possible optimisations

The implementation of a ByteBuffer can be optimised, for instance DirectByteBuffers provide direct memory access, and you could have a PCIByteBuffer which provides direct access to the memory on PCI cards. With byte arrays you have no freedom over memory allocation (always on the heap). Also, if the underlying packet framework uses ByteBuffers then these can be directly exposed to nio, since that works entirely in terms of ByteBuffers. Otherwise, you face the overhead of wrapping byte arrays with ByteBuffers, which kind of misses the whole point of nio. NIO was designed to take advantage of low-level direct memory access.

If all the packet stuff uses byte arrays, I think you will end up with a crippled nio implementation. In summary, ByteBuffers are lower-level than byte arrays, and so should be used (aka java primitive arrays not equal to machine primitive arrays, but ByteBuffers can encapsulate them). I think it is also possible to provide JVM hooks for particularly ByteBuffer subclasses which gives further scope for optimisations.

Yes i agree with you. In this

Yes i agree with you. In this case, the only thing we need to change is the RawPacket implementation, to use ByteBuffers instead of byte arrays.

That's why the packet representation is designed this way in this proposal.

And the rest

Don't forget, ofcourse, you need to change the high-level Packet interface. The getData(), etc methods need to be defined in terms of ByteBuffers not byte[]. Otherwise, you will lose the advantage. In fact, just simply ban yourselves from writing "byte[]" in any source.

Hmmm you are right

Hmmm you are right. I have not think it this way.

There must be a a wraper class for the byte[] and byte buffers or anything else. This would be the return type of getData() of all the packets.

This wraper class would be usefull for the entire operating system, in order to represent in-memory buffers. Do we have something like that in JNode yet?

ByteBuffer is the wrapper class

ByteBuffer is that wrapper class. It even has a static method ByteBuffer.wrap(byte[]) specially to wrap byte[]. It is designed to represent in-memory buffers, why invent yet another wheel? And, yes, i agree the entire JNode OS should use ByteBuffer subclasses to represent memory. You then have a nice level of transparency. Hopefully the jnode fs guys are using ByteBuffers for their low-level I/O routines, then it'll be simple to do things like a direct-write from network to disk, as well as memory mapped files. Ofcourse, it is just these types of operations that NIO was designed to make possible....

....Just looked at the fs stuff. As I feared, they are using byte[] not ByteBuffers in FSFile....

....Looking at some of the core/gui stuff, I think some kind of MemoryByteBuffer extends ByteBuffer implements MemoryResource is needed as well.

I should probably post something in the forums about this...

Yes, byte[] is used for fs stuff and not ByteBuffer

I can confirm byte[] is used for fs stuff and not ByteBuffer and,
because I am working on the cache for any block device (that is used by fs stuff), I have thought of such a class ... and realize it already exists !

To avoid this kind of error later, what do you think about having something like a javadoc of JNode sources and maybe a search engine for it ?


I share your idea of mak

I share your idea of make the current design with the SocketBuffer more a OO minded design.

Only thing is that it should be made in a branch, so that we don't break anything for the jnode development process as this will involve changes to all the network drivers and more. When it works it should be merged into head. We should start of with this proposal and there after start looking on the other proposal you made.

The branch name for this proposal should be something like "JNODE_net_packet_representation-branch".