Intrepid Blog

A week ago my windows was agonizing slow. Starting up would take ages, and getting those ‘windows is out of virtual memory’ messages was common, so I decided to reinstall windows.
When reinstalled windows was using 120 mb without anything special installed.. so I started to install my usual application: apache, mysql, mssql.net express, php.. etc..
At the point I got those installed windows was using 250 mb instead of 120mb! msSQL server uses 50mb`s (although that number isn’t displayed in the task manager properly (i guess due to modules). MySQL and Apache both 25mb`s…
So I put the apache/sql servers services off by default and made a little bat script to start those when I want and put that one on my desktop. I also made a backup of windows so I didn’t need that horribly slow reinstall..
I installed some basic drivers via the provided driver cd’s loading a lot of junk like tray icons on my computer. They did not appear to use a lot of memory in the taskmanager but when quiting them it saved 100 mb :-/. 100mb used by selfish driver control panels thinking they are the single-most-used application and the only one sucking that amount of memory…
So instead of using those horrible flashy flash autorun installers I let windows find the correct driver .inf files and install it.
This really saves a lot of performance.. (I’ve cut down my startup time by 4 times).

So, better let windows find the drivers you require on your driver cd’s instead of using those selfish flashy traybar spamming driver control panel installers.

I made some updates on my command line parser for .net .

It now supports aliases and you can set whether the parameter should accept a value or not so you can use -foo bar where foo doesn’t accept a value and therefore bar is regognized as a position dependand parameter.

An example parameter definition:

[Parameter(Position = 0, IsMandatory = true,
AcceptsNoValue = false, Name = "folder",
Description = "The folder",
Aliases = new string[]{"target"})]
public string FolderParameter
{
    get { return _FolderParameter; }
    set { _FolderParameter = value; }
}

(Ignore the slashes placed before the quotes, wordpress seems to add them automaticly) I also have added the Intrepid.Automation.CommandLine.OutputHelp function, which outputs help for the parameters of an object to a stream. Like this:

SUMMARY
Creates playlists in a folder and its subfolders

PARAMETERS
-target

[String excludefilter = NULL] (efilter)
Only files not matching this regex are included

[String includefilter = NULL] (ifilter)
Only files matching this regex are included

String folder (target)
The folder

[Boolean expand]
Whether to expand referenced m3u’s

REMARKS
m3u playlists are automaticly excluded; use -excludefilter to add additional excludes.

You can still download the Intrepid.Clorelib assembly here and you still may only use it for non-commercial open-source usage and may not change/reverse engineer/etc it in any way.

Hope it will prove usefull :-).

I just wrote a command line parser for .net which parses the command line similar to the way Monad’s cmdlets do it, with the key difference that this still is for the normal .exe executables.

It’s very simple, first get yourself a program class, which contains some properties you want the user able to set via the commandline and apply the Parameter attribute:

class CopyProgram
{
    private string _Source;
    [Parameter(Position = 0, IsMandatory = true)]
    public string Source
    {
        get { return _Source;}
        set { _Source = value; }
    }
    private string _Target;

    [Parameter(Position = 1, IsMandatory = true)]
    public string Target
    {
        get { return _Target;}
        set { _Target= value; }
    }

    private bool _WalkRecursivly;
    [Parameter(Name = "recursivly")]
    public bool WalkRecursivly
    {
        get { return _WalkRecursivly;}
        set { _WalkRecursivly= value; }
    }   

    static void Main(string[] args)
    {
        new Program().Run(args);
    }

    void Run(string[] args)
    {
        CommandLine.ParseCommandLine(args, this);

        // Do stuff here
    }
}

The CommandLine.ParseCommandLine function takes care of parsing the command line string array and filling the required properties of the class provided.

You can call this program in a few different ways:

copy.exe c:\source.txt c:\target.txt
copy.exe -s c:\source.txt -target c:\target.txt
copy.exe -targ c:\target.txt c:\source.txt  -r

Although it only required around 250 lines of code it certainly is an enourmous time safer and makes stuff a lot easier for the user.

At the moment I’m programming a few small utilities and will change the code a bit and fix some bugs if any.

For those interesting in testing it themselves (I encourage it), just download the Intrepid.Corelib .net 2 assembly*. The classes required are in the Intrepid.Automation namespace. Please send any bug reports to: bas.westerbaan@gmail.com.

(* You may use everything in the Intrepid.Corelib assembly freely for non-commercial open-source usage. Do not modify or reverse engineer the assembly in any way.)

Update:
Got rid of some bugs and added the RequiresValue property to ParameterAttribute. Will replace lateron with Type, this allows you to use -e bleh where -e takes no value and bleh is interpreted as a nameless parameter. Will add alliases too.

Regular Expressions in .Net (System.Text.RegularExpression) are fast!

When creating a Regex with the compile flag the regex will actually be compiled to very optimized native assembly which makes the .net regular expression library faster than any c# code you could write for one specific case.

This was a pleasant suprise for now I know I can use regular expression for all those small things like parsing input.

Finaly I got grip of a recording of an electric guitar without distortion and applied the gamma wave distortion on it:

Sine gamma effect results and positive linear gamma results

Negative linear gamma effect results

As promised a few pictures of the electric guitar effects we’re working on,

first up, the gamma wave effect.

The gamma wave effect pulls the amplitudo’s either to the 0 line or the -1/1 line, just like the gamma on your monitor does:

The effect basicly makes the wave a lot louder and gets rid of the faint sounds. When on fully applied as in the images above it also creates some distortions due to either making the wave unharmonic-ish or getting rid of the nuances.

Another image taken on a higher oscilation (smaller zoom):

This clearly shows the distortion

(The inversion of peak and valley doesn’t matter for for sound only the transition from a valley to a peak matters)

As I said before I am working with some others on a software based distortion. I finaly got the base running pretty smoothly but the main obstacle is how to create that distortion electric guitars use.

We came up with about 3 different methods:

(When I got my osciloscope control working I’ll take before and after pictures, for now I ain’t sure how the formula’s effect real-world sound yet)

Gamma corrections
This works a bit like the gamma of your monitor. The input for graph for gamma correction is called the epsilon which comes in 2 forms, linear and sine. The first one creates a more distortion like sound than the latter but also makes the wave not harmonic anymore which possible means that most of the distortion like sound is caused by the speakers not handling unharmonic waves very well. The latter one only sounds near the amplitudo distorted.

Sharpening valleys/tops
This method requires some buffering of the current top (or valley) of the wave and sharpenes it by a specific amount. This works a bit like the gamma correction method although it works on every volume making it usefull too for low-volume sounds. The major problem is that it requires tracking a top or a valley, which with a high bitrate requires a really big buffer to analyze, and it has a delay of one top/valley. The big problem is that this isn’t ideal for live playing for which it was designed, a delay of a few extra hundreds of a second would be noticable and it is fairly possible that this algorithm requires too much resources.

Adjusting speed resistance
At a certain point in a wave you can derive the speed and the angle. By registering the original speed and resistance of the wave a derived one can be created which could leed to sharper or softer edges of a top/valley just as the previous method but without being required to analyze the whole top/valley. We’re hoping this method will work best but this is just a concept hoping to work out as it should.

If anyone actually knows how analog distortions work exactly we would be more than happy to learn about it, just comment.

More on this to come…

At the moment I am busy creating a software based distortion for an electric guitar.

The most challenging part is getting effects to work, to explain this in more detail you need to know how a computer handles audio.

Sound itself is nothing more than a vibration in the air. It can be represented by the amount of force the air is pushing or pulling.

A tone of a specific frequency would look like:

A computer stores sound by sampling the amplitude of the air at regular interfals.

It would be easy to write a program to increase the frequency of the sound above, but a normal sound doesn’t look that regular:

Increasing the bass or treble of that wave would require some advanced algorithms, which take time to execute which creates a larger delay. One thing that a distortion shouldn’t do is lag.

More on this when some stuff is working.

There are a lot of negative myths about .net which people tend to use to favor the traditional languages like C++ above .net. I’ve busted the ones I read frequently:

• The GC is really slow
  malloc is way slower! The Garbage Collect of .net actually is faster than any Unmanaged code for it nows whether a value is a reference (pointer) and therefore can move objects in the memory. The GC puts objects of about the same age (generation) close to eachother in the memory. Objects tend to refer and use objects in the same generation. The processor itself doesn’t directly load a value from the memory but loads a whole block of a few KiloBytes in the Cache. When the processor directly caches all the objects which one object uses it just runs a lot faster for working from the cache is a lot faster than recaching different parts of the memory over and over again which happens with unmanaged languages which just put objects where there is free space.
• Interpreting that stupid Intermediate Language is damned slow
  .Net doesn’t interpret its IL, it compiles and optimizes IL runtime
• Compiling runtime is very slow anyways
  (That compiling a C++ is slow doesn’t mean that .Net is slow) It saves a lot of time for compiling at runtime allows great optimalisations like getting rid of unreachable code and inlining depending on the current runtime variables. Also operations can be compiled with processor specific optimalisations from one IL source. Most of the resource intensive compiling is done at the startup of the application, it is done while the program is running too but that really makes it a lot faster instead of slower
• If I write assembly myself it will be way superior to anything .Net can generate
  .Net can’t make all the optimalisations possible for it would take longer to analyse code than the optimalisation would gain. But usualy it creates still very optimised code. The big problem with writing very optimised assembly yourself is that the most optimised code is very processor specific and would be very hard to port, and even worse to maintain. Wanting to add one little extra feature could let you rewrite the whole code again. (Yes I indeed have made programs with assembly). Languages which avoid this a bit like C++ still require you to make a different build for every specific processor when fully optimising. Also it is nearly impossible to debug fully optimised unmanaged code but in .Net it still provides you with at least the functionname in which it has happened with the offset (try to accomplish that with C++ in release mode)
• The runtime is soooo damned big, it sucks
  20 Mb’s isn’t a lot. It only has got to be downloaded once, and the .net framework is in Windows Update so everyone who updates his computer would have it installed by now. Usualy there is room enough on your software installation CD to include .net, it is more than worth those 20 mb. Also languages like C++ require certain runtimes which arent that cooperative. Does ‘DLLHell’ ring a bell?
• The .net library naming SUCKS
  Yeah.. its naming is different than what MFC uses. At least the naming is very consistant which is way more important than ‘nice naming’, although when seeing some C++ API names used I still wonder why someone could prefer that above the clear .Net naming
• The .net library itself sucks
  Really? Like what? What can’t it do?
• You can’t use API calls like CreateFile
  Now I can’t…
  [DllImport("kernel32.dll", SetLastError = true)]public static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess, uint dwShareMode, IntPtr lpSecurityAttributes, uint dwCreationDisposition, uint dwFlagsAndAttributes, IntPtr hTemplateFile);
  … now I can!
• .Net sucks cause it is Microsoft
  Yeah, so what. .Net is a ECMA standard so you are pretty free to use it, and if there is a catch then that one hasn’t been exploited yet for on linux people are happily using mono to run .net stuff

http://www.microsoft.com/downloads/details.aspx?familyid=C8F904E1-B4CA-402B-ACCF-AAA2BD60DA74&displaylang=en

When having Avalon installed it adds a few new project templates to my Visual C# 2005 express, with which avalon applications can be made. Avalon is a Window system which uses xml files to define a form. At the moment I couldn’t find a designer neither reference to it in the help files and assume it isn’t implemented yet, which is rather a pain for creating a form by editing Xml by hand is just a pain: Xml is hard to write and I am just too dependant on the user friendlyness of the form designer.

When compiling Avalon generates classes and serialized data files in your application to replace the Xml files.

So what does it basicly do? Lets you design your forms easily (although there isn’t a proper designer yet) maintaining a good performance by replacing the slow Xml files by generated classes and serialized resources compile-time.
Having a few test applications decompiled and having looked at the sdk it seems that Avalon can do practicly the same as the current Windows.Forms dll.
So now I wonder, in what way would Avalon be better than using the great current form designer and the Windows.Forms dll?

While working on a design of a .pak files, which are nothing more then archives, I noticed most existing .pak files are just very simple. This for they put all files after eachother which makes it really hard to even just add a file, or delete one.

Therefor I started thinking about a way to make an efficient archive file for read and write access.

There are basicly 2 ways to accomplish this:

• Clustering
• Fragmenting

Fragmenting.
The archive consists out of several files, and an index which is counted as a file. The start of every file is pointed to from the index by its offset. When a file is deleted, its record in the index is null-ed, which is lateron filled with a new entry and in the meantime ignored (for the only file that can start at position 0 is the index itself).

A file itself is fragmented, every fragment starts with 4 bytes marking the length of the fragment and after that again 4 bytes marking the offset of the next fragment this time. In case there is no next fragment present the next-cluster-offset is 0. The first fragment of every file contains after the 8 bytes marking the length and offset of the next fragment some information about the file like the name.

For the index itself is a file, it also is fragmented and can grow, and shrink.

The 4 byte offset value could be default set to 8 bytes in case files are going to be bigger than 2 Gig, and the oposite in case you are dealing with small files. Although this can decrease the size of files which are frequently edited it isn’t possible to increase or decrease to 8 or 2 bytes in runtime.

Clustered
The clustered way is practicly the same as the fragmented way except for that every fragment is padded to a multiple of the cluster size. Therefore the ‘next fragment’ value can now be a cluster number instead of an offset. This will both increase the speed of writing due to having always a buffer, also defragmenting will be faster. The files itself will also be able to retain a 2 byte next-cluster pointer longer. The only downside is that files will just be larger.

FragmentedStream
The fragmented stream wraps these methods pretty efficiently at the moment.
I am thinking to implement an inteligent algorithm which checks which streams are often used which are then placed at the end of the file with a buffer. This also could be applied for files which are marked to be expanding.