JNode System-file Tree Structure

It is time for us to start thinking about how JNode should run from a normal harddisk based system, how it should be installed and maintained.

An essential part of this, is how and where system files should be stored, and what system files are. This page answers the question of what system files are and proposes a tree structure for system files to be stored.

What are system files

In JNode there are a few different types of system files. These are:

  1. Kernel image, (jnodesys.gz) which contain the nano-kernel, the system plugins and the system classes.
  2. Initial jars, (e.g. default.jgz) which contain the plugins needed to start the system.
  3. Plugins, which contain services (drivers, apps, ...) and are loaded on demand.
  4. Configuration data, which contain configuration data specific to the device it is installed on.
  5. Bootloader files, (stage2, menu.lst) used by Grub to boot the system.

Tree structure

In JNode system files should be placed in a container in a structure that is "well known".

A container can be a filesystem, but can also be some kind of database. See the next paragraph for a discussion on this.

The proposed tree is as follows:

/jnode The root of the container
/jnode/system Contains kernel and the initial jars
/jnode/plugins Contains the plugins
/jnode/boot Contains the bootloader files
/jnode/config Contains the configuration data

System file container

Traditionally the system files are stored in a directory on a "root" filesystem. This filesystem is identified by a parameter, or some flag in a partition table.

This method is easy, because all normal filesystem tools can be used on them, but makes it harder to protect these files against virusses, ignorent users etc. Also this method limits the system files to being harddisk based. (E.g. look at the trouble Linux had to have an NFS root filesystem).

For these reasons, I propose a more generic method, that of an abstract container interface for system file access. This interface can have implementation ranging from a single file based container to a network loader. The essential part is that the actual implementation is hidden from the part of JNode that uses it (either to load files, or to install them).

This is all for now, please comment on this proposal.

Ewout

Gobo done right way

According to the message by Daniel Noll :

 
It would probably be a good idea to avoid the UNIX directory layout. It's full of design decisions 
which no longer make sense in the modern world. 

e.g. separating bin and sbin when you could use the +x flag to distinguish the two types of 
executable, separating bin into  /bin, /usr/bin, etc. to support boot-time repair when live 
CDs are a better solution. About the only principle which still applies is that user data 
should be on a different partition to system files as the system files are more easily created from 
scratch. And that's only when a user actually cares about data integrity, the vast majority of 
users can make do with a single partition these days (not counting swap of course.)

The only Linux distro I know of without this problem is GoboLinux (there is probably at 
least one more.) 

Having the commands similar is fair enough I think, although you'll get a fair number of 
people who say they're used to dir and not ls.

This is the right position of this opinion.

Stupid newbie rambling

Background bits:
a fundamental design decision is the guideline that "everything" should be a "file", manipulated by a single simple API (open, read, write, close) that allows extension (ioctl)

physical storage, network filesystems, and in-memory "pseudo" filesystems are abstracted together in a single "tree" via mount, rather than using "drive letters" (using a vnode tree in Solaris, for example).

In a traditional inode-based filesystem, all files in a physical filesystem are directly accessible by their inode number.

a "directory" is a metadata file that contains name-inode pairs (conceptually) rooted at directory '/' (pointed at by the mountpoint inode)

code and data are simply datablocks referenced through inode maps, treated differently based on permission bits in the inode.

So, that's a basis what I know for filesystems. If you want the detailed version, see Solaris Internals by Jim Mauro and Richard McDougall, page 470 (hehe) and chapters 11-15

So, the 'abstract container interface' sounds to me exactly like the Solaris virtual filesystem nodes (vnodes). Traditionally, the mountpoint inode of the disk filesystem that holds "root/boot" is mounted as the root of the tree, but there's no reason you have to do that. During jumpstart, the kernel mounts an NFS path (one example) to the in-core root vnode.

=== Ondisk filesystem ===
Since I hate defragging, let me make my plea for EXT2 or ZFS to be the default on-disk filesystem, with JFAT as a selectable option, rather than default. Also, there's no patent question involved, which FAT still has hanging over it.

=== Tree Layout ===
A tree is simply a namespace.
Java already implements a namespace layout in its design and operation, in the form of the classloader's package layout/loading scheme. Why not base your initial tree design around that?

So, why not just anchor the package paths at / for classloader?

When a user/permission scheme for multi-user use comes into play, it could still protect those from accidental user error.

--- System/Plugins ---
org.jnode.etc ... why not just use their package paths as-is?
Package org.jnode.etc becomes /org/jnode/etc/
com.exileinparadise.insanity becomes /com/exileinparadise/insanity.

Extensions by the local user(s) and incorporation of add-ons becomes a no-brainer. Tie in a subversion client and you have a ready made layout for network updates too.

--- Container root /jnode ---
As far as the namespace structure goes, is there any technical reason that /jnode/ has to prefix every path in the tree, or is it simply everyone's preference?

--- Namespace Personalities ---
Why not allow each user to have their own file namespace?
A flexible filetree/namespace layout scheme might make it easier for customized distributions later too, for people who like the Unix, Linux, Mac, Windows, or other layout.

Jnode could ship with a "default" filetree configuration, but have that be controlled or built from a properties file that the system owner can customize.

The namespace/filetree mapper reads a configuration that maps Jnode objects like classes, Java sources, Javadocs, plugins, metadata, pseudo-filesystems and more to some set of tree paths specified by the system builder/owner.

A user could then have their own custom file to override that.
If I am a Linux user on a system that the owner setup to look like Windows, then I could simply shuffle the view of the standard system objects around to suit me.

Fundamentally, this is just an extension of the existing vfstab/fstab concept which says where to make disk device content into the in-memory tree. Extend that to cover all Jnode objects, not just disk content of specific filesystems?

A namespace/filetree is simply a "renderer" that reads a set of data structures and presents a view of them, so why fix that in stone?

Jnode boots into core memory and can run exclusively from there, now.
A disk is not even required, and that is a powerful thing to leverage fully.

However, if we are simply talking about nailing down the default tree, my thoughts are something like:

/{com,org,net,us,in,...} DNS domains/ISO3166 country codes/package classloader paths, mapped as is with slashes instead of the dots, the way Java intended. Store the sources here too. What better way to learn the system?

/org/jnode/ - the jnode.gz and initjars go here, for build.sh too
It's the base for Jnode's package path anyway, just build into it, and let GRUB load straight from it.

/boot - a GRUB favorite

/jifs - Jnodes pseudofs needs a home, right? devices/ belongs here too, not as /devices, IMO

/
If the system will host multiple users, then /home, or whatever.

Okay, so much for thinking out loud.
Feel free to flame or trash this rambling as you see fit.

--
Robert "Exile In Paradise" Murphey

Some comments

Hi. Imho good you brought that topic up again! As you probably know, I personaly use vmplayer+iso to work with JNode. But since there has always been talk again about an installer (and I think there are already parts of it) we need some layout.

I do not have a real opinion on that (yet), but I'd still like to comment to collect ideas.

> As far as the namespace structure goes, is there any technical
> reason that /jnode/ has to prefix every path in the tree, or is it
> simply everyone's preference?

This prefix has a "technical" background. JNode only allows to mount filesystems to directories in the VFS. So if partition 1 is mounted on /foo/ you may not mount partition 2 on /foo/bar and you can never mount a partition to / (then you wouldn't be able to mount anything else anyway Eye-wink)

Afair this decision was made so that "your grandmother understands what happens", and mount to a mounted filesystem,.. is too "complicated".

Anyway I think this is not so important. You made a better point: There are two or more types of "view" to the filesystem. One is the system layout which can be seen from system plugins (which is present on the filesystem that way). It is the VFS of JNode. The others are "user views". We might give each plugin or each user or whatever it's own "view". So the Files an application can see and how they will be presented to the application could be totaly customizable. E.g. an untrusted application would only see one root with a temp directory in it, the shell would see either the complete file structure or only the user's home directory,...

About the file structure at the fs level I have no favorite. I just want to have one point considered: The structure should be build so that directories with very very many files are avoided. E.g. no /lib/ directory with thousands of .jar files.

Some comments on comments

Thanks for the info.
I can see the limit of "no mounts to filesystems" being a real problem for Jnode adoption in the distant future, because it is a limit that someone will hit, especially if you were ever looking for enterprise uptake.

Mounting filesystems to filesystems is complicated, true.
That's why I pointed out the Solaris internals book.
It has a whole "filesytem framework" chapter that talks about virtual filesystem vnodes, and path traversal.
Mounts on mounts is a path traversal problem, made much easier by using a virtual filesystem in-memory for all internal lookups, and let the path traversal handle converting the "all vfs" in-kernel paths used into the sequence of filesystem-by-filesystem calls.

I will keep this in mind as something to build/rebuild/change when I get to looking into the FS source.
--
Robert "Exile In Paradise" Murphey

Keep it for the PRIMARY level of 0.3Version

hai,
We should keep this simplest Form of the SystemFileSystem proposed by Ewout for this primary stage.Because this matter will always create conflictions between all to make a Exact decision, as the viewpoint of the all Users+developers are always different.
So, it will be better primarily keeping it.And if possible to give freedom the users for making their SystemFileSytem as they want.
Thanks tango

directory tree

I propose the following:

/jnode The root of the container
/jnode/system/kernel Contains kernel and the initial jars
/jnode/system/plugins Contains the plugins
/jnode/system/boot Contains the bootloader files
/jnode/system/config Contains the configuration data
/jnode/user All user installed stuff
(/jnode/user/programs)
(/jnode/user/home)
(/jnode/user/home/me)

I think this way, if you hide som folders from view, the filethree becomes less scary. Example: /bin /usr/bin /usr/local/bin. For a programmer folders are easy. I have been teaching for some years, and the absolut nightmare for beginners are lots of folders.
A consistent and clean file system in the beginning is an relatively easy task to acomplish and gives much value for the work. If some folders are going to be scary hide them behind a nice named one. Folders is also great for explaining what they contain. You say to the users: Do not touch anything in /jnode/system (C:\winnt C:\windows).
If something scary happens in system, the end user never sees any new folders or anything different. Some examples:

ls /jnode

system
users

tar cvfz programs.tgz /jnode/user/programs
tar cvfz everything_installed_so_fartgz /jnode/user
tar cvfz system_backup_02.06.2005.tgz /jnode/system

If there is going to be stored som good old config files, xml seems to me the most consisten choice, I hate linux's 10-100 differtent config scripts. It's working, but you get so dead tierd of everything you have to learn. Think of a linux with a standard dtd for every system xml file! And how easy this makes creating a gui for configurating the system becomes. Not only system configuration, but also programs, logfiles etc. One program to edit and view the entire system and all program's config files.

any desicion yet?

Hi,

is someone working on that topic ? (Or is the desicion not yet made?)

Trickkiste

I'm working on it

It's not going really fast, but i'm working on it.

Ewout

in which direction?

Hi,
are you working on the mentioned interface?
And / Or on one of the mentioned ways (maybe to store it on harddisk) to store the data?

Trickkiste

Re: Direction

A plugin can store persisent configuration data in a self defined object per plugin. This object is made persistent by the plugin manager onto a harddisk, or any other medium.

So the need to config-files is then gone, except (maybe) for the configuration of the persistency-manager itself. This i hope to pass on via the Grub command line, or just figure it out in the software itself.

Ewout

different filesystem views

What about a db based filesystem like the new filesystem in longhorn?
You can have "views" like an audio view.
In the audio view you can select "sort by year" and in the audio view you see directories like 1970 and 1980 which contains audio files from that year. If you select "sort by genre" you see directories like "pop music" and "clasic music".

is this kewl ?

Format of configuration data

To avoid a lot of different types of formats when we are to configurate the system it would be a very good idea to have a common jnode standard for config files. Any good ideas?

Regards,
Martin

Configuration

It is more object oriented to configure a live object via methods and have that object store its state persistently (in my oodb Smiling. I understand, of course that it is not easily doable for low level systems. At a the lowest level config files are needed. But lets not use them in places where we could use persistence.

Thanks.

Alex

storing its state persistently ?

When you say that object store its state persistently, do you think of using your oodb as an object oriented registry.
As opposite to windows registry that is 'primitive types' oriented (DWORD, string, WORD, ...), your registry/oodb is 'object oriented' and maybe use the java serialization mechanism (or a custom persistence).
Is this what you thought ?

With object oriented configs, we can have the configs that follow the class hierarchy (a config 'extends' another config).

Fabien

Configuration

Not exactly. I am against a registry in any form. I merely meant that services that need to maintain state between lifetimes should not be configured using a config file. That is not object oriented, and can be quite confusing.

As an alternative to using config files I suggested using a (my) oodb to store state-data. A email service could use the db to store server names, username/password, security settings, and anything else that needed to be saved.
These properties should be editable only through invoking methods on the service object. Any change in state for the live object should be reflected in the stored object (something my oodb does very well).

However, in the absence of a registry, it will become neccessary to have a central location to alter system state(settings). I have forseen this need, and have an idea to address it.

I propose a Configuration Tool, that will serve as a central administration console. The tool will display all of the objects registered to be configured. Once an object is selected, the bean properties, and no-arg methods (opperations) will be displayed for invoking. This is similar to JMX, though it may need some adjustments because we are making an os, not a destributed business system.

In short this tool will present an easy interface for configuring services in an object oriented maner.

The Configuration Tool is on my list-of-things-to-build. I am close to finishing the first version of my new component system. After I have sent it to Ewout, and any other interested parties (any takers?) I will begin work on the Configuration Tool.

What do you think.

Alex

Versions of a configuration

One thing to remember when designing a store for configuration data is that it must be able to handle upgrades to newer versions of the software gracefully. When moving form one version to the next there are bound to be new fields in the configuation that has to be saved. So the upgrade process must handle this gracefully.

So if we use a object-based database then we must handle multiple versions of the configuration objects as well; this could also mean both the old and the new class(-file). If the upgrade has skipped versions then we must handle multiple versions of old configuration classes.

Using XML or something structured in ruffly the same way that is not directly mapped the internally used objects are probably a better strategy.

Loose coupling is key!

How about access for other ap

How about access for other api developers? We are using dnsjava for then resolver and the api looks for system files like ex: /etc/resolv.conf under unix/linux and on windowsNT/2000/XP it makes a

p = Runtime.getRuntime().exec("ipconfig /all");
findWin(p.getInputStream());
p.destroy();

I guess we need some kind of easy access(just for read-only) for outside jnode apis.

Regards,
Martin

Sorry

Sorry, I have been slow in responding, but I do not realy understand your question. Do you want to know if my peristor is publicly accessable? If so, yes it is.

Any party can use my system. In fact, as of now there are no security checks of any kind. Any party has access to any object that is in the store. There is no diferenciation in read/write access at the api level. That type of controle can be gained by using read-only bean patters in the stored object (get, but no set).

Ewout informs me that the java permision protocal could be added to my system to implement certian security controles. This is not my forte so I have made no attempt to do so. Shortly I will sent my code to interested parties, and they may jugde where or how security may be implemented.

Once again, sorry for being slow. Let me know if I have not answered you question.

Alex

You may count me in as an int

You may count me in as an interested parties. I would be glad to see your work.

Regrads,
Martin

Contact

Contact me at [email protected]

a full oodb or a specific oodb

When you talk about an oodb, do you think of a full oodb with lot of functionalities for general purpose or a specific oodb with only what we need ?

About your Configuration tool, have you thought about a hierarchical view (made from package hierarchy) ? What kind of GUI ? charva and later swing and/or awt ?

When I first saw the integration of charva in JNode, I thought about the utility of writing an application for charva, and after for swing, and after for SWT ...

To avoid rewriting many GUI for the same application, what about creating a generic framework to build a GUI : many factories that are able to build labels, tree, table, textarea ...
Thus, we have a factory for each of these : charva, swing, (swt ?), (awt?), and maybe html. Perhaps we can use JSF (Java Server Faces) or XUL (that convert XML->GUI and is used by mozilla).

However, I am interested in seeing your first version. Can you send it to me by mail when it will be finished ?

Fabien

Details

I do not see the need for a complete oodb. The system that I have made is more rightly called a persistence mechanism. Its is capable of:
Storing almost any serializable object (there are a few extra rules).
Retrieving stored objects based on objective state. (see queries)
Maintaining perfect synchronization between live object state and persistent state (even against crashes) and,
Efficiently managing live object lifetime (performance enhancement)

This is all acomplished via objective proxy (not interface proxy) which means that the persistence mechanism is transparent to the client.

Queries are handled by custom implementation. For example:
new PersistenceQuery()
{
public Class getType()
{
return Document.class;
}
public boolean fitsConstraints(Object o)
{
Document d = (Document)o; //don't need isInstance()
int i = d.getText(0,d.getLength()).indexOf("War and Peace");
return (i != -1) ? true : false;
}
};

This PersistneceQuery will yeild any object of type Document that contains the String "War and Peace". Thus you can see that java is used to select java objects.

An os must mantain many small, unrelated bits of data, not a few very large tables as is seen in business. My system seams well suited for our purpose.

When I do write the configuration tool, I will start with swing. But the charva contract says that I can just replace java with charva. So that will be ok. (By the way what is SWT? SwingWT?)

I do not know about the fesability of crating a UI factory to abstract the text/graphic complex. I would be very impressive. But the primary question is, Do we intend to mantain long term support for textural graphics?

I will gladly send my new component sytem to all who would like it. Contact me at [email protected] and I will mail it when I am done.

Thanks for you support,

Alex

SWT are GUI components used by Eclipse

I moved the discussion about GUI components here.

Fabien

Isolating Read-Only Parts of FileSystem

Hi,
wouldn't it be possible to isolate some dirs of the OS, to mount them in ReadOnly ? For ex. in Read-Only all the /boot , and all of the system, but WITHOUT the variant part (config, logs, etc) in Read-Write ... like /var in Linux
Having the ability for the System Admin (via a User Role), or the JNode Installer to change the mount mode as it will be necessary.

For all the (Non-OS Packages) we could use a similar approach (another partition):
/prg for the read-only part, and (/var ? )/prg.config for the variant parts.
Then, the read/write here could be handled by the JNode Installer and the JNode Security Manager.

For this the best could be having (in the future) the LVM Like that in Linux, to allocate logical partitions as they are needed.

Comments ?

Bye,
Sandro

potential container implementations

If I understand well, System files can be installed on :
- its own new partition (harddrive, removable media, ...)
- an existing partition (single file based container)
- a ROM (maybe it can replace the BIOS)
- an USB key (cool, but need a compatible BIOS)
- a network loader (a 'network computer' without harddrive I think)
- other things I don't think about ...

For network loader, maybe JNode can be loaded from some radio technology (WiFi). I don't know if it's possible to do it nowadays (booting from WiFi).

Correct

Ewout