Igmar's tech talk

Sigasi cli in CICD

Fri, 24 Oct 2025 15:20:09 +0200

If you happen to have a license of SVH Enterprise Edition, or like me, have a trial license, you also can use the CLI features of SVH.

A few things on the CLI. As of the 2025.3 release, the CLI is part of the VSCode extension, but by default, it ends up in a bit unconvenient location :

/.config/Code/User/globalStorage/sigasi.sigasi-visual-hdl/cli/sigasi-cli

for me. You will need to add that to your path (or better, symlink it in $HOME/bin. I prefer a manual CLI installation. On a build agent, you will need the separate installation anyway, else it will require a headless VSCode installation, and that is way harder then a download + unzip of the sigasi-cli installation.

The separate download is available here. To install it, unzip it into a location of preference, and make sure the directory is in your PATH variable, so calling sigasi-cli actually works. The CLI is a licensed feature, so you need to set SIGASI_LM_LICENSE_FILE to the correct location of the license file, and make sure the user calling sigasi-cli has access to both the installed CLI, as well as the specified license file.

First, let me explain the value of the Sigasi CLI : it allows for automation of operations that are otherwise not easy to do outside of VSCode, and it can be used to safeguard project quality in your CICD pipeline. If you are a software developer, the IDE and CICD pipeline safeguard certain aspects of your project. If you happen to code in Go for example, most IDE’s already call go fmt on your Go files.

Code formatting

If you use CICD and work with Go projects, you can use a pre-commit hook to automatically reject commits that contain unformatted code.

Why would you want that in the first place ? Well, different whitespace settings in IDE’s can cause very long diffs in version control. It takes focus away from engineers from things that really matter : Functional changes.

Lets give an example. First, a piece of (badly) formatted VHDL code

library IEEE;
use IEEE.std_logic_1164.all;

entity and_gate is port ( a : in  std_logic; b : in  std_logic; y : out std_logic); end entity and_gate;

architecture rtl of and_gate is begin y <= a and b; end architecture rtl;

As you can see, both the entity, as wel as the architecture declaration are one liners. Syntactically, it is correct, the parser will have no issue with it, and the resulting netlist will be a correct one.

Is it easy to read (image a 1000 line file) : Probably not, at least, not for me.

So, lets try to fix that :

sigasi-cli format --align --keywords=lowercase --spaces-for-tabs --vhdl-version 2008 bad_example.vhd --out=formatted.vhd

which will result in

library IEEE;
use IEEE.std_logic_1164.all;

entity and_gate is
    port(a : in std_logic; b : in std_logic; y : out std_logic);
end entity and_gate;

architecture rtl of and_gate is
begin
    y <= a and b;
end architecture rtl;

Unfortunately, the formatted lacks verilog support, as well as configuration options for the formatted. For verilog, there is verible-verilog-format which works well, as has configuration options for determining how it formats. Downside : Another tool to install in the CICD agent.

So, why does this matter ? If you are a lonely designer like I am, probably not. If you are part of a team, it matter a lot. Looking at the example above, both are valid VHDL files. If one developer uses one version, and another developer uses the other, and it ends up in version control, the diff will look like

--- bad.vhd     2025-10-11 18:55:41.282057461 +0100
+++ good.vhd    2025-10-11 18:55:47.554132357 +0100
@@ -1,6 +1,11 @@
 library IEEE;
 use IEEE.std_logic_1164.all;
 
-entity and_gate is port ( a : in  std_logic; b : in  std_logic; y : out std_logic); end entity and_gate;
+entity and_gate is
+    port(a : in std_logic; b : in std_logic; y : out std_logic);
+end entity and_gate;
 
-architecture rtl of and_gate is begin y <= a and b; end architecture rtl;
+architecture rtl of and_gate is
+begin
+    y <= a and b;
+end architecture rtl;

Are there ways to limit the impact of whitespace ? Sure. Do you want to use it : No. Desigers should focus on the design, not on how the code is formatted, or if code conventions are lived after. That is what we have IDE’s for.

Style validation

Sigasi SVH has options to set requirements for a coding style to be used. Do you want all input ports to start with i_ ? You can configure that in SVH. The same applies to many other options related to verilog / VHDL coding styles. You can flag violation of the requirements on info, warning and error level.

Setting the style validation option in a project level will result in a file being writtin in the .settings directory in the project. Those files can be added in version control.

Doing a sigasi-cli verify . from the CICD pipeline on the checked-out project will result in the validation being run, and that includes style validation.

Setting the input port pattern to ^xx_.* will, and using a name that doesn’t match will result in validation messages like below.

rtl/z80/z80_debug.v:10:16: WARNING: Naming convention violation: input port name should match pattern 'xx_.*'

In this case, its a warning, but you can also set it to ERROR level. Adding this to a pipeline is a next step in making checked in code comply to company guidelines, and reject code that fails to meet the guidelines.

If you have really big projects, this can help in maintaning a consistent style.

Summary

The above options provide some basic guidelines how to use the Sigasi CLI in your CICD pipeline. It can do all kinds of validation, based on settings that developers set in their IDE. Doing so helps to keep a project maintainable.

The tools is agnostic of the actual CICD pipeline used, but since its bound to a flexlm issued license, using a PaaS / SaaS hosted CICD solution will probably not work. Most solutions will allow to run your own agent, and that does allow to use licensed features.

Running Lattice Diamond on Fedora 42

Mon, 06 Oct 2025 10:39:10 +0200

As some of you might know, next to a BsC in computer science, I also have a degree in electronics. I never used to do anything with that long-gone knowledge, but since a few week backs, I revived an old hobby of mine : Retro computing.

I’m especially fond of the MSX, which I also own myself. It was also the first computer I get aquited to.

Going back to that era also mean going back to the time you can create your own hardware, and writing software often meant assembly and in-depth hardware knowledge. There days, most is abstracted away by drivers and high-level interfaces (DirextX, OpenGL), and modern compilers take away most need to learn and use assembly language.

Doing you own hardware there days is more ’easy’ then back in the 80’s due to the arrival of FPGA’s, which provide you with the means to create complex hardware platforms at a cheap price. Boards with an Lattice ECP5 or Gowin FPGA are very affordable these days (< 100 Euro’s>). Developing for FPGA’s is done in either Verilog or VHDL (yes, other HDL’s also exist, I know, but those are far from mainstream), two Hardware Description Languages. Looks like a regular programming language (the Verilog language looks like C), but its a descriptive language, which means you don’t always get what you want. Hardware also involved timing and concurrency by default, which both don’t play a vital role in software development. Being able to do both makes you a better developer, being it in hardware or in software.

With FPGA’s in most cases come… vendor tools. In the software world, most is free. In the hardware world, most is priced involving lots of digits. For homebrew development, most vendors (Lattice, Intel, Gowin) provide ‘free’ versions that are usually limited in features and device support. The paid versions start at approx. 3k USD per year, and that goes up to 100k.

This blog is about Lattice Diamond, tooling that comes from the vendor and does ECP5 FPGA’s. Without a license, you can use the most common devices. For hobby stuff, I own 2 devices : A Lenovo T580 and a Dell XPS13, both running Fedora Core 42. The T580 also runs windows because I play games on steam as well.

So, lets get this thing running.

Download & installation

Your journey starts at the download site for the Diamond software. Pick the Linux RPM version (no, that doesn’t actually deliver an RPM).

Create a new empty dir, and unpack the zip file in that directory. You’ll find a single file, called <version>_Diamond_lin.run. Run that binary (as a user, NOT as root), and install it in a proper place. I place all vendor software in /opt, so the installation ends up in /opt/lscc/diamond/ You need to manually create the menu item, the program it needs to run is <install_dir>/bin/lin64/diamond.

the first issue : shared object version conflicts

If you run it, some things you’ll notice (run it from the termimal to start with) : Tons of errors on missing glibc symbols, and some things crash (floorplan editor being one). This is caused by Diamond using its own set of shared objects, but still depending on some system QT libs. It also relies on the system’s libxml2, and that version delivered in a supported OS (RHEL 7, RHEL8) differs from what is there in FC42.

The embedded SO’s are most likely done to prevent issues on somewhat “older” RHEL versions. For me, this resulted in all kinds of errors because it also uses system provided SO’s. FC 34’s libs are in all cases new then what is embedded in the Diamond distribution.

For the libstdc++ issue, I solved it with a

find <install_dir> -name '*libstdc++*so*' -print0 | xargs -0 rm

the second issue : coredumps

The second issue is that using certain functionality result in a coredump. Opening a project, and selecting Floorplan View results in a crash and a coredump. The backtrace gives :

#0  0x00007f1973cd7388 xmlSAX2Comment (libxml2.so.2 + 0x32388)
#1  0x00007f1958525d7f pr_parse (libbaspr.so + 0x125d7f)
#2  0x00007f19584fe446 _ZN11PR_PREFCNTL15syntax_chk_loadEPKcS1_S1_S1_ (libbaspr.so + 0xfe446)
#3  0x00007f195eeb4d1e _ZN7PrfData8loadDataEv (libcommondata.so.1 + 0xb4d1e)
#4  0x00007f1964d76245 _ZN15PNHeavyDutyTask15loadCommondDataEv (libpnmaindll.so + 0x176245)
#5  0x00007f1964d768e5 _ZN15PNHeavyDutyTask7runTaskEv (libpnmaindll.so + 0x1768e5)
#6  0x00007f1974c8b86c n/a (libQtCore.so.4 + 0x8b86c)
#7  0x00007f197427ff54 start_thread (libc.so.6 + 0x71f54)
#8  0x00007f197430332c __clone3 (libc.so.6 + 0xf532c)

So, it crashes somewhere in libxml.so.2. That SO is however not part of the Diamon distribution, and is taken from the system path. The version that Diamond assumes, and the version that FC42 provides are binary incompatible.

The only way to fix this is to provide an older version of libxml, and making that specifically available to Diamon. If you install this in the normal system location, you will brick things for sure.

The recipe that worked for me :

git clone https://github.com/GNOME/libxml2.git
cd libxml2
git checkout v2.10.4
./autogen.sh
./configure --without-python
make
cd .libs
cp -d ibxml2.so.2 libxml2.so.2.10.4 <install_dir>/bin/lin64/

That should repairs the non-working functionality.

I haven’t used all functionality (yet), so there can be other (hidden) issues. I’ll update this blog when I encounter and fix an issue.

Autoscaling like a Pro on AWS

Wed, 05 Nov 2014 22:18:23 +0000

The Amazon cloud (AWS) is a great place to deploy your application to. It’s flexible, cheap, you pay by-the-hour, and it offers great additional services to make your life easier. AWS also offer a great API, with properly written tools, that can act as a complete replacement for the AWS console. If you can manage something from within the console, you can also do the same thing from within the API. I’ll cover the AWS tools a later on.

When it comes to handling a massive amount of traffic, we need to look at a number of component, they we look how that translates into AWS. A few key pointers to dynamically scale your application (I consider a website to be an application) :

Nodes need to be self-contained and stateless. All nodes should be considered equal, so no master or slave nodes. Nodes can also come and go when they please in an auto scaling setup, so if you store data on it, make sure the node is not the only source of data.
If you manage relational data, split up the reads from the writes. Read-only copies of relational databases on AWS are cheap, and easily setup, and even more important, scale well. All reads go to the slaves (which are read-only), all writes go to the master. As a bonus, RDS slaves can span multiple availability zones (CHECK : cross region ?).

If possible, make the write action a service. Requests to services can be queued, and you don’t have all nodes lurking the master database. It also means you can replace how it works without breaking all the nodes. AWS also offers queues as a service, where queue endpoints can be all over the AWS cloud.

If possible, store data in a NoSQL database. If data is read-only, it’s usually a good candidate. It also means you don’t have to deal with ORM layers, which can be a plus (even seen the queries Hibernate generates ?)
Cache things. All of it. Really. If it can be cached, cache it. Memory is cheap and fast, disk access is far less fast, and network access is probably even less faster. If you can keep things in memory for even a few minutes, it will keep the response times fast. Also, a look up from memory can’t go wrong, which is not something you can say from disk or network access.
Use API’s for everything, and always assume something can fail. If it fails, handle it well. Generating a decent error message, and notifying OPS is also considered “handling well”.

Elastic Beanstalk

One of those service is Elastic Beanstalk, which allows you to deploy applications with the push of a single button on the AWS console. Elastic Beanstalk is basically a top-level front-end for a number of AWS services : Elastic Load Balancing, Auto Scaling, CloudWatch and CloudFront.

It basically gives you a load balancer, an auto scaling policy, monitoring and a chef-based infrastructure deployment tooling all combined into one service.

Since it’s a generic solution, it is limited to a number of platforms :

IIS
Node.js
PHP
Python
Ruby
Tomcat
Docker

You have limited control over the environment you get, customization is possible with the use of .ebextensions. While it gives you some control, it’s not the whole deal, but it will be sufficient for most cases.

Since we use the playframework, the options are limited to either Tomcat or Docker. The Tomcat options doesn’t work with our setup (we need apache in front), so, it’s down to just one option : Docker.

Docker on Elastic Beanstalk is run on an instance, with the application run inside a private subnet. So, you have the instance on a private subnet, and the application in the Docker container is run inside a NAT subnet within that container. That’s not convenient if you need to handle incoming traffic outside the load-balancer.

Rolling code updates

Another show-stopper for me was the fact that it doesn’t support rolling code updates. Ideally, you would deploy a new version, it would boot up instances with the new version, wait until they accept traffic, and when they do, you terminate the old instances with the old software version. That way, you would have zero downtime. Unfortunately, Elastic Beanstalk doesn’t support that. So, when switching to a new version, it removes all instances with the old version,

I haven’t been able to get this in a workable state. You might end up with a different situation, where it is usable. If it isn’t : What’s next ?

Auto scaling

Luckily for us, AWS has auto scaling. In short, you group a number of instances (the auto scaling group), and the number of instances in that group is determined by metrics collected from the instances in that group. This doesn’t mean absolute numbers but the actions taken based on the metrics result in a number of instances.

Sounds.. Well. Vague. Let me give you an example.

Let’s start a group with 3 instances, and a maximum of 10. Those are absolute values : The group can’t drop below 3, and can’t go above 10, no matter what happens. You need to provide triggers (alarms in AWS), and an action. Those two things together are called a scaling policy. To make it usable at all, you need to create a policy for scaling up, and a policy for scaling down. So, you start with 3 instances, it gets busy, and the group scales it up by one instance. Then you have 4. If that’s not enough, it will add another one, etc. You can add multiple triggers, so that you can handle traffic peaks.

Scaling policies

First, you should determine what the trigger is. You can pick CPU utilization, Disk IO Operation (Disk Ops), Disk bytes read/write and network bytes in/out on EC2 instances. I usually pick the CPU utilization, because that gives me a direct relation with how busy a node is. The more CPU utilization, the busier it is, and the other way around.

So, you can pick for example 50% average utilization as a trigger, and when that gets trigger, add 1 instance to the auto scaling group. If CPU utilization drops below 30% average, you remove 1 instance for the group.

Scaling is a delicate operation : You don’t want add instances to fast (it will cost money), or remove instances to fast, which will result in a higher load, which can result in a scaling policy kicking. If you create a bad scaling policy, you will have a constant flow of scaling operations, which is a bad thing. To prevent this, AWS has a cooldown period. If a scaling activity occurs, it will wait the number of seconds assigned to the cooldown period before another scaling activity will begin. Effectively, it will suspend scaling activity during that period. During the cooldown, the new instances have the time to settle down. You need to pick that number carefully, which will I cover later on.

What happens if a scaling activity occurs ? It means launching an instances from a predefined AMI. That AMI, and which instance type, is determined by the Launch configuration. Every scaling group has one launch configuration attached. Normally, when the instance is declared “Healthy”, the auto scaling group will add it to the ELB, and it will start to accept traffic. Good, assuming your instance is directly capable of handling traffic after it has booted. In many cases, it’s not. You can have your application starting up, Tomcat booting, all factors that basically means you can’t handle traffic directly after the instance has booted.

If your instance needs time to “warm up”, you need to tell the ELB this. This means that the ELB will do periodic checks, and when those check succeed, it will add the instance to the load balancer. As long as the check fails, it will not send traffic to that instance. You can tell the auto scaling group to use the ELB’s health check instead of looking at instance healthiness. This will allow your instances to warm up, and accept traffic when they are ready. The auto scaling group will wait a certain amount of time before it starts checking, the “Health check grace period”. After that, the checks begin. There is one catch : If your instance isn’t ready to accept traffic after the grace period, the auto scaling group will consider the instance unhealthy, and replace it.

Let me rephrase that : If you instance isn’t healthy after the grace period + (Unhealthy Threshold * interval), your instance will be axed and replaced. So, pick this value carefully. Very carefully.

So, at this point we have autoscaling. We need to be able to rollout code without downtime

Code rollouts

You usually want to rollout a new version of your application without bringing it down. You also want to control how this is done, instead of an automated system doing this for you. In my case, I’ve decided to use some nice AWS properties to accomplish this :

If you attach an ELB to an autoscaling group, it will direct traffic to the ELB to all healthy instances in the autoscaling group
You can tell the autoscaling group to use the ELB health check, instead of the EC2 health check
You can attach an ELB to multiple autoscaling groups

Combining all the above facts, it roughly functions as following :

Every application has two autoscaling groups: One which hold the current version, with a number of running instance, the other one is a target for the new application (with a new launch configuration), and that has no running instances.

In pseudo code, it ends up with :

check_ami_id $ami_id
check_ami_tags $ami_id
determine_app_version $ami_id
find_as_group_name $appname
check_elb_health $asg_current
copy_current_lc $appname $ami $lc_old
assign_lc_to_asg asg_target $lc_new
suspend_asg $asg_current
scale_asg asg_current $asg_target
wait_until_instance_cnt_reached $asg_target $asg_current
wait_until_healthy asg_target $instance_cnt
resume_asg $asg_current
scale_asg_down $asg_current
wait_until_terminated $asg_current

Hey !! This looks like Bash. Well, it actually is. Actual implementation is done in Bash 4, and with some minor work, should also be able to support Bash 3.

I’ve done this for a number of reasons :

I wanted to upscale my Bash knowledge
It’s installed by default on all AWS instances we run, and also with a recent enought version
No external dependencies to worry about : We use tools that are installed by default

I’ll shortly describe each function :

check_ami_id : this checks if the AMI id is correct. This is just name-only.

check_ami_tags : This checks if the AMI has the correct tag set. It wants a
tag named 'autodeploy', with the application name as it's value. This
prevents the wrong application ending up on production, because someone
picked the wrong AMI by accident.

determine_app_version: This pick the version tag from the AMI, and display
that.

find_as_group_name: This finds the two auto scaling groups. It needs exactly
2, anything else is considered an error. It finds the two groups by looking
at it tags, where it expects to find a tag named autodeploy, with the name
of the application as value. How the groups are actually named isn't
relevant. It then fills asg_current and asg_target by looking at how many
instances are running. If both have running instances, it is considered a
fatal error.

check_elb_health: This check the ELB attached to the auto scaling group.
All instances registered on the ELB need to be healthy. If this is not the
case, this is considered a fatal error. We do this to prevent the number of
instances to be determined wrongly, since that number is needed later on to
see if the new auto scale group has scaled up.

copy_current_lc: This copies the launch configuration of the running auto
scaling group to a new one, with the new AMI attached. You can't edit a
launch configuration, so the only option is to create a new one.

assign_lc_to_asg: Assign the new launch configuration to the target auto
scale group.

suspend_asg: This prevents the auto scaling policies from kicking in. The
scripts depend on the number to be consistent during run time.

scale_asg: Scales up the asg_target to match the asg_current

wait_until_instance_cnt_reached: Waits until the auto scale group reaches a
certain instance count.

wait_until_healthy: Waits until at least instance_cnt in the auto scale group
are healthy (as determined by the ELB)

resume_asg: Resumes the auto scaling policies

scale_asg_down: Scales down the auto scale group to zero

wait_until_terminated: Waits until all instances have terminated

The scripts use the AWS cli exclusively, together with some standard *NIX tools : readlink, wget, basename, grep, awk, sed The AWS cli is an one size fits all tooling : You can use it to control all aspects of AWS. Most commands offer filtering, and the output format can be switched to either JSON, a table or text.

text is a tab delimited format, which is ideal as an input for bash, since it operated on tab delimited input by default. While I know that json can also be handled from within bash, it depends on external tooling, or some really ugly workarounds. Since it gives no advantage in this case, I’ve chosen the text format.

AWS CLI

First, make sure you have it configured. You need to set AWS_ACCESS_KEY and AWS_SECRET_KEY before you can use it. To test, run

aws ec2 describe-regions --region=eu-west-1

If you get a list of regions : OK, if not, you need to check your setup. The output will look like

{
    "Regions": [
        {
            "Endpoint": "ec2.eu-central-1.amazonaws.com",
            "RegionName": "eu-central-1"
        },
        .....
        {
            "Endpoint": "ec2.ap-southeast-1.amazonaws.com",
            "RegionName": "ap-southeast-1"
        }
    ]
}

To use the tab separated format use

aws ec2 describe-regions --region=eu-west-1 --output=text

and it will get you

REGIONS ec2.eu-central-1.amazonaws.com  eu-central-1
....
REGIONS ec2.ap-southeast-1.amazonaws.com    ap-southeast-1

The same data, different format. The AWS tools contain two important features :

Filtering. Limit the output to match the filter specified
Query : Limit the output to selected parts

Let me explain both of them to you. Take for example

aws ec2 describe-tags --region=eu-west-1 --filters
"Name=resource-id,Values=ami-bcad1234" --output text

This will display the AMI ami-bcad1234 if that exists, else the output will be empty. You can have multiple of them :

aws ec2 describe-tags --region=eu-west-1 --filters
"Name=resource-id,Values=ami-bcad1234" "Name=key,Values=autodeploy"
--output text

Will display AMI ami-bcad1234 of they have a tag with a key named autodeploy. Matches will only be displayed if both conditions are met. Some command give you tons of output. In most cases, you’re not interested in all of them. Getting all of the output makes filtering the desired information harder, and if fields get added or removed in the future, your script might break on them.

Assume we want to query the load balancer attached to an autoscaling group. We are not interested in the rest, just the name of the autoscaling group. information. To see what you get by default, run

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names
"name of auto scaling group" --region=eu-west-1

Pick the right region, and the right name. You’ll get a lot of output, but we just want to get the load balancer name from that output. Here is where query comes in. It gives you a way to tell the CLI what you want, and that you are not interested in the rest.

To just get the ELB name, use

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names
"name of auto scaling group" --query
"AutoScalingGroups[0].LoadBalancerNames[0]" --region=eu-west-1

So, what does this do ? It asks for the first member from the LoadBalancerNames array, and for the first LoadBalancer. You can also ask for multiple values :

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names
"name of auto scaling group" --region=eu-west-1 --query
"AutoScalingGroups[*].Instances[*].
{HS:HealthStatus,Id:InstanceId,LS:LifecycleState}" --output text

This gives you for all members in the AutoScalingGroups array, the Instances member, and the HealthStatus, InstanceId and LifecycleState members. The CLI tools requires you to ID each field.

Wrapping it up

Now that we have the autoscaling setup and deploy in place, that leaves us with one thing : The AMI. Or better we need a way to create them. Traditionally, we launch an existing AMI, make the changes we need, and create a new AMI from the modified existing AMI.

While this works, it’s also very time consuming, not to mention error-prone. Luckily, we have a solution for this : packer. It’s a image creation tool on steroids.

You start out with a template (a JSON file), let the builder create an AMI from the template, using a one or more provisioners. I used two : a list of assets that get uploaded to the instance, and a bunch of shell scripts to do the actual work. While it can use fancy stuff like Ansible, Chef, Puppet and Salt, I personally are way more comfortable with bash, and it’s also easier for colleagues to understand.

In general, I advise for two templates : A base one, with for example a JDK and all need tools, and one for the application. That saves us time since we don’t have to install the JDK with every build, and items in the base AMI is pretty much fixed.

The application AMI handles installation of the application, with the mention base AMI as source. This way, you can build you AMI with a short amount of time, and with a consequent result : You can’t forget stuff. It builds it the same time over and over again.

While we have a complete set now, on my personal TODO list is a git hook, so it builds a new AMI with every tag that get’s pushed.

Questions / comments ?

Have any questions / comments ? My contact data is on the bottom of each page. I’m also pretty easy to find using Google. Since I’m not doing posts frequently, I welcome feedback.

Server side mail filtering

Tue, 02 Jul 2013 13:08:31 +0000

When you get a lot of mail, and go on vacation, you usually pull in all that mail over your expensive roaming 3G connection.

I use filtering to make sure certain mails end up in the right folder. I usually don’t want to read all mails that come from mailinglists. You can use filtering from within your mail client and that works well. That has one huge disadvantage : Your mail client needs to pull in at least the headers to see if it matches filtering. Not nice if you’re for example subscribed to the LKML.

So.. How do you handle this, without all kinds of mailclient plugins ? The answer : Service side filtering. I use Maildir as my mailbox format, and if you still use mbox : See this article to migrate. I use maildrop as my mail delivery agent, since it has filtering capabilities.

I assume that you’ve told your MTA (Postfix, Sendmail) to use it : If not, you’ll end up debugging why it isn’t filtered, and at the end, you’ll find out it isn’t even called.

When maildrop works, it’s time to start filtering. Configuring is done using the .mailfilter file in your homedir, or a .mailfilters directory in your homedir. A small part of the one I’m using right now :

It’s pretty simple : It specifies a logfile (relative starting at our homedir). After them, a regexp, and a action. I wrap the actions in an exception statement, so that if the move fails, we don’t bounce to the sender (which in case of a mailinglist, will mos likely result in a unsubscribe). In above examples, it will filter messages from LinkedIn or the OSGi mailinglist.

Look at the headers to see a common header in messages that belong together. You can’t assume things, especially not with mailing lists.

If you also run SpamAssasin, you can filter on for example on the X-Spam-Status header. Example :

Happy filtering !!

Asterisk for the novice, part 2

Sat, 29 Jun 2013 21:15:30 +0000

Let’s start with the dialplan. I’ve put it on github : extensions.lua Yes, it’s a beast. It’s also the first piece of LUA I’ve even written, so yes, a LUA guru can probably shorten it by half or so.

SQLite database

Our user list is in a sqlite db. That’s also what lines 1 and 4 are about : Adding the LUA extension (line 1) and opening the database (line 4). It’s a bit buggy : Sometimes we end up with a locked SQLite database, and all further operations fail. The solution is either a MySQL database, reverting back to textfiles, or fix the locking issue. I haven’t fount the cause yet.

**Context **

A context is a group of extensions. In this dialplan, all SIP phones end up in the context sipphones. The incoming ISDN lines are mapped to the bri-incoming context. That means that you can also do nice things with incoming calls, based on the number of the calling party. This script looks up the number using a REST call, and then displays the number on the display of the phone.

The default context (which is always present), maps to the unauthenticated context (using the include directive), and that just calls a function that drops the call.

**Initialisation and extensions ** We define our outgoing interface on line 10 : TRUNK, which is our ISDN BRI card.

The API key on line 13 is for the interface to our admin interface, which requires a login and cookies, except when using the API, which needs the apikey.

The actual work is done in the extensions table on line 352 : That maps what extension (read : number) is handled in which context. It also determines for example what happens when you dial 911 (or 112) : You want that to work, even when the extension isn’t logged on.

In short : It maps dialed numbers to action the PBX takes. For example : If you dial #741, you ask the PBX to call e_hotdesk_login(), with the context and extension as arguments.

It does checking (is the extension enabled, and not already logged in), asks for the PIN, and if that is correct, writes the SIP phone ID to the database, and puts the user on the right queue.

Logging out is the reverse : Remove from the queue, and remove the SIP phone ID from the database.

Dialplan

The dialplan we now use is sub-optimal : You need to think about the public service, and what mapping the SIP phone uses, and is hardcoded. That might interfere with the dialplan you think up, which get’s you into trouble.

Make sure you get this right : Users get annoyed when the dialplan changes on a regular basis, and you end up with unhappy collegues.

Issues

Issues I’ve ran into :

No real argument checking of the calls into Asterisk from within Lua. Documentation is a bit scarse, so you need to debug, and have a debug console run at maximum level open while writing the Lua code.
The SQLite DB keeps getting locked, especially when the DB is accessed a lot. No idea how to fix this, or what the real cause is. A good solution would be to use Yaml instead, since the file is mostly read-only anyway.
When doing a transfer, the person doing the tranfer is displayed, instead of the caller. The cause is that Asterisk acts as a intermediate gateway, doing an outgoing call, and then a passthrough between the two connections. It’s fixable in theory.

Conslusion : You can do a lot of very nice stuff with Asterisk. I have really touched the real nice goodies yet. Hope I can some day.

If you cant to discuss : contact me.

Asterisk for the novice

Fri, 26 Oct 2012 13:46:59 +0000

We switched to VOIP : Our old PBX was lacking features, and the prices for phones went sky-high. I purchased an Asterisk EuroISDN card, and some Aastra phones.

Some lessons I learnt the hard way :

A GUI

If you installed a GUI, deinstall it. If you have the GUI config and installation in a backup : remove it from the backup also. Then, print the GUI sources, and shred them. Really, do so. If you have a real problem, and a GUI, and no knowledge of Asterisk, you’re totally screwed.

The hardware

First, make sure Asterisk knows your hardware. dahdi_hardware should give some output. If it doesn’t, reinstall the kernel drivers, and make sure that the driver is loaded. I used the .src.rpm for building those, but the instructions at https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+DAHDI also work.

Load the kernel drivers : I needed to load wcb4xxp and dahdi.

Asterisk software

I installed Asterisk 10.6, using a homebrew RPM. No special reason for this version : It was the latest available.

Configurating Asterisk

First, I ran dahdi_genconf, that took care of configuring all 4 spans. My chan_dahdi.conf looks like :

[trunkgroups]

[channels]
; Deprecated, don't use it
[lines]

echocancel = yes
echotraining = yes
signalling = bri_cpe
callerid = asreceived
context = bri-incoming
group = 1
channel = 1,2
channel = 4,5
dahdichan => 1,2,4,5

The SIP phones also need config. This is done in /etc/asterisk/sip.conf :

[jdioffice](!)
type=friend
context=sipphones
host=dynamic
nat=no
dtmfmode=auto
disallow=all
allow=ulaw
allow=g722
allow=g729
qualify=yes
language=nl

[00085D3XXXXX](jdioffice)
secret=mysecret

This gives us two asterisk contexts : sipphones, and bri-incoming. You need these to tell Asterisk what to when a call comes in or goes uit on a certain context.

I use a LUA based dialplan : I know how to code, and that gives me an extensions file that is a whole lot better to read then the ’traditional’ Asterisk dialplan config.

Entrypoints are in a table :

extensions = {
  -- Incoming calls
  ["unauthenticated"] = {
    ["_."] = e_drop_call
  };

  ["sipphones"] = {
    ["_#NXX"] = e_hotdesk_login,
    ["_#000"] = e_hotdesk_logout,
    ["_0X."]  = e_outgoing_sip,
    ["_NXX"]  = e_internal_sip,
    ["112"]   = e_outgoing_emergency,
    ["911"]   = e_outgoing_emergency,
    ["_*."]   = e_dial_custnr,
    ["_**."]  = e_dial_employee
};

  ["bri-incoming"] = {
    ["_X."]   = e_incoming
  };

  ["default"] = {
    include = { "unauthenticated" }
  };
}

You can clearly see the context, and the resulting mapping. All names are actual lua functions, which take a context and extension as a result.

Next post : More on the dialplan, things I’ve run into, and the solutions, if available.

A new PBX aka playing with Asterisk

Tue, 03 Jul 2012 12:32:40 +0000

We started out with 1 employee back in 2001. Now, we have 11. We started out with a Siemens PBX, and one ISDN-2 line. That Siemens has 8 digital ports, and 1 expansion slot. That slot has been taken for a second ISDN-2 port, since customers where getting a busy signal.

Now, people share a phone. Not something we want. The Siemens PBX is also as extensible and flexible as solid concrete. We had 2 options :

Buy a new PBX from Siemens (Siemens phones only work on a Siemens PBX)
Throw everything out, and start from scratch with a VOIP based solution.

After receiving the offer for the Siemens PBX, recovering from falling of my chair, and being resuscitated by one of my collegues, I decided to start from scratch with a VOIP solution. We soon ended up with :

Asterisk as the software platform
A Digium B41oP 4 port Euro ISDN card
Aastra 6757i SIP phones

Hardware was quickly ordered. I’ve upgraded our old fileserver with extra memory, a new storage array (which needed replacement anyway), and the Digium card. The OS has been manually upgraded from CentOS5 to CentOS6, since installer totally screwed up the upgrade.

The’re no packages for CentOS 6, only for CentOS 5. I needed a shole bunch of non-standard libs :

libss7
libopenr2
libpri
dahdi

Downloaded, and packaged them. Asterisk was next : Version 10 wasn’t an option, since OpenPBX refuses to work with that. I ended up with 1.8.13.0, since that’s an LTS release. Welcome if you don’t want to restart you PBX every day or so.

I’m going to publish the SRPMS and i386 binaries in my RPM repo, no x86_64, sorry. If you want modifications to those RPM’s, contact be, and we’ll work something out.

GDB 7.3a and a new elfutils

Tue, 27 Sep 2011 14:14:47 +0000

I just uploaded gdb 7.3a It can be found on my RPM repo. I’ve also added new elfutils packages, since these are required by this version of GDB.

Binutils 2.21.1a on CentOS 5.6

Fri, 23 Sep 2011 20:59:51 +0000

I just uploaded binutils.2.21.1a. It can be found on my RPM repo. GDB will be uploaded begin next week. Happy compiling !!!

GCC 4.6.1 on CentOS 5.6

Fri, 23 Sep 2011 15:14:30 +0000

I’ve just created packages for CentOS 5.6 that replace the default compiler with gcc 4.6.1. That version has some interesting options, so I wanted to use that as my default compiler. It’s also more strict, so it’s a good check for my own code.

Packages can be downloaded from my RPM repo

Later today I’ll upload my binutils and gdb packages.

VIM 7.3 with Python 2.6 on CentOS 5.x

Thu, 22 Sep 2011 13:43:33 +0000

I’ve been a OSX user for some time, but my main C development happens on CentOS. OSX has TextMate as editor, which has snippets : A sort of runtime macro’s on steroids. It saves lots of time with common language constructs such as loops.

Vim has a UltiSnips, but that has some preconditions : It need vim 7.3 with Python >= 2.6 compiled in. CentOS has neither of them. You can’t just upgrade Python on a CentOS machine, since lots of things (yum being one of them) depend on a specific version. So if you overwrite the default version, all hell will brake loose.

I’ve created packages for both Python and Vim (x86 and srpms), available at http://www.palsenberg.com/rpmrepo/

Happy VIMming !!!

Migrating to JIRA 4.4

Wed, 07 Sep 2011 09:28:51 +0000

Migrating to JIRA 4.4 is now complete. It has’t been without hickups however :

JIRA 4.4 en Fisheye 2.6.0-2.6.3 don’t get along when using trusted authentication. You get Seraph errors : Error handling trusted applications authentication attempt:BAD_SIGNATURE
Make sure the user that is authenticating the application link (and managing the JIRA project) has read errors to Fisheye projects. Else you can’t link you repo to the JIRA project.

Make sure you have adequate backups before attempting to upgrade. Without them, it’s a one-way ticket of something goes wrong.

Confluence

Tue, 09 Aug 2011 12:32:12 +0000

I’ve installed an milestone of Confluence. I never liked the 3.x editor : Wiki just isn’t my thing. Fortunately, the 4.x release has a brand new editor, and it’s WYSIWYG. No more fancy tagging, no more thinking on what wikimarkup to use.

A nice addition is that it also allows macro’s to be embedded into that page. I’m going to use that to add some information of my own personal projects, like commit history.

Migrating to Atlassian tools

Tue, 24 May 2011 12:14:58 +0000

So.. Finally found to spare time : I’ve migrated my old setup to include (most) of the Atlassian tools. I’ve got Crowd, JIRA, Confluence, Bamboo and Fisheye up-and-running, with SSO authentication. Next step is to figure out how to make Bamboo build my C / C++ projects (which use make, not Ant or Maven which are supported native in Bamboo). I also need to look at the testsuite, for which Bamboo assumes it generated JUnit XML reports. At this moment, it doesn’t. I’ll be opening Confluence spaces for most of my (old) projects soon.

Working with re2c, lessons learned

Thu, 05 May 2011 13:54:09 +0000

I’ve completed the parser that parses bind9 style config files. The parser itself is based on the lemon parser generator, and the lexer is based on re2c. While testing and developing the parser, I’ve ran into some strange, undocumented issues.

Error checking is practically absent. Bad input results in bad runtime behaviour instead of errors.
Make sure that all input in conditions is handled; Input encountered in states without a matching rule results in the resulting code jumping to a (semi) random rule. Again, this doesn’t result in errors.
RTFM about what regular expressions re2c supports. No error checking what so ever.
Rules with wildcard conditions (<*> <regexp>) are first processed, above all other rules. A condition with a matching rules overrides the wildcard condition. This is important if you have a match-all condition [^], since that will precede the wildcard condition rule.

If you live up to the above undocumented things, re2c make writing lexers as easy as it gets.

Migrating from Xen Citrix to RHEV

Thu, 28 Oct 2010 12:53:13 +0000

Since I’ve got fed-up with all Citrix bugs (operations that het stuck on locks, resumes not working, etc) and the fact that the’re probably going to drop Xen in favor of Microsoft Hyper-V we are going to migrate to RedHat RHEV.

The software installs fine (except for the know issue that you can’t register on RHN), but migrating 35 VM’s is not something I really like doing. Well, virt-v2v comes to the rescue according to the docs.

Uhm.. Well… not exactly. What the docs don’t really mention that it depends on a Xen with libvirt on top of that. And that is exactly what Citrix Xen doesn’t use. So, only errors, and no migration.

So.. I’m stuck with a manual migration, using or friends dd and netcat. Bah :(

PHP and the reasons why we consider switching

Mon, 13 Sep 2010 13:28:59 +0000

PHP is a nice language, with numerous advantages :

Easy to learn
Easy to install and upgrade serverwise
Interpreted, so no compile and deploy worries

The disadvantages are starting to annoy me more and more over time :

Zend engine is huge. It’s a major memory consumer
Not all extensions are stable. When it comes to thread safety, there are major problems
Some language constructs simply suck : __construct(), the 5.3 namespace seperator
No type hinting for build-in types (int, string)
No OO function overloading
Multibyte support is basically non-existent

Our current framework (I-Content) is based on PHP4 code, and therefore not object orientated. There are some improvements, but the PHP limitations don’t go away. Our new framework is on it’s way, and build using JAVA. The above limitations go away, althrough we get some new disadvantages in return. Time will tell if the switch is really going to happen…

Booting a > 2 TB partition using EFI

Fri, 20 Aug 2010 14:43:52 +0000

When our new storage server arrived last week, I wanted to skip use EFI t0 boot this system. It requires en EFI capable system, which this Intel server mobo is capable off.

The CentOS installer doesn’t grasp an GPT partition table (needed to enable > 2TB partition tables), so I installed manually. Well.. That didn’t go to smooth. Manually installing CentOS, and the GPT partition table was, looking at it afterwards, the least of my problems.

Booting the system was. The system was throwing me into an EFI shell, instead of booting the system. After doing some RTFM on EFI the issue became clear : The EFI BIOS wasn’t seeing the controller, because the BIOS on the Areca RAID card isn’t EFI capable, but only supports plain old BIOS.

End of story. Areca isn’t going to provide an EFI BIOS, because no OS supports it. No OS supports it because no EFI BIOS is avaiable. You get the point… So, back to a plain old MS-DOS partition table.

Migrating to SSO

Fri, 23 Jul 2010 08:15:45 +0000

We at JDI are moving to a project based working method. We decided to use the tools from Atlassian, since those tools seem to do what we require. We also run a number of internal tools, and all those together mean remembering about 6 different usernames and password.

We decided to give Crowd a try : SSO, with an option to integrate with custom applications. It uses SOAP internally, and SSO is managed with a domain cookie. About a day work, but now all Atlassian tools we use, our administration system and subversion are authenticating against crowd :-)

We're at Maildir now

Sun, 11 Apr 2010 20:59:59 +0000

As noted in this posting, we migrated to Maildir. It went relatively smooth, and faster then anticipated. I’ll be posting the code to the actual migration script (41 lines of PHP code), and the code code to the milter that temporary locks the mailbox (aboyt 150 lines of C actually written by me). This way, it reduced downtime to the actual time needed to do the conversion, instead of shutting down the whole MTA.

The source for the PHP script and milter are here.

Moving from mbox to maildir

Fri, 12 Mar 2010 07:39:54 +0000

I’ve decided that it’s time to move from mbox to Maildir. Mailboxes are getting larger every year, and so serverload continues to increase. During scheduled maintenance the end of March, I’m converting all mailboxes to Maildir format.

I’ve run into a number of issues during some preliminary tests :

The conversion can’t hand the situation that mail get’s delivered during the conversion
IMAP / POP mutations on the mailbox while converting screws up the conversion process.

The IMAP / POP3 part can be handled the simple way: Lock the user during the conversion, then re-enable user after the conversion is done. The incoming part is a more challenging issue : You don’t want to shutdown the mailserver during the while conversion. You can’t tell sendmail to delay mail during a certain time by default.

To solve this, I’ve written a milter (see www.milter.org), so that during the RCPT TO phase, it checks the user’s homedir for a .maildisable file, and echo’s the first line of that file if it exists with a 4xx error. That solves the problem :-)

Moved to VPS

Wed, 24 Feb 2010 13:33:36 +0000

Today I’ve move all my personal stuff to a VPS. It let’s me run my own software (although I manage our company’s web server, I can now run several services I’m not otherwise able to use).

Email wil follow today, and so will be a restoration of all the personal project pages. I’ve lost some source due to a disk crash (and the stupid assumption that I’ve had backups), but I’ll put that info on the projectpages themselves.

Finding a new editor

Thu, 25 Jun 2009 09:36:49 +0000

We, at JDI, are used to working in vi. When we started 11 years ago, most IDE’s wheren’t existant, and with 3 people with their own OS and configuration developing locally wasn’t an option. So we just shelled in, vim’ed all of our files, and created nice things that way.

Today, when the compay has grown, this isn’t an option anymore. People want an IDE, mainly for code completion and debugging. We run multiple OS’ses (Windows, Linux, Mac OS), so everything windows-only isn’t an option.

I’ve looked at three options :

Zend Studio for Eclipse Eclipse Netbeans

Zend studio is buggy, especially when it comes to Remote Systems. Not to mentions it’s slow as hell when it comes to importing a remote project. Eclipse suffers from the same problems, and NetBeans is missing remote SFTP support.

I’m trying Netbeans with MacFusion at the moment

About Igmar

Tue, 01 Jan 2008 08:00:00 +0000

I’ a C, C++, Rust, Golang and Java, en work at ING as a advisory solutions architect.

I have a broad knowledge on today’s IT world, and specialise in giving teams advise on the technical direction they need to take in order to use available infrastructure and technology at best.

In my spare time, I do retro computing, mostly in Rust and the hardware designs in Verilog. PCB’s are done in KiCad.

Open Source

Tue, 01 Jan 2008 08:00:00 +0000

A few of the things I’ve coded in the past years :

Apache mod_suid : An apache module to un all scripts under a seperate user ID. Available on github.

etoken_key : Small module that spits out a key when a Aladdin Etoken key is inserted. Handy in case you need a way to decrypt your filesystem.