Posted on

Catenating mp3 files correctly under Linux

I sometimes download lots of mp3 files from the web (bit-torrent or music) and need to catenate them. This is usually the case when there are just too many files lying around or when they are too short like 3-5 minutes long and I want them to be 30 minutes long or more. This is very useful for educational material. In this post I will review the different methods to achieve this and offer my own solution.

Solution 1: catenate the files simply. cat *.mp3 > out.mp3. There are a few problems with this approach. First, an mp3 file is a file with a header and just concatenating the files would leave multiple headers in the output file. The output file will therefor have the headers of the first file. Not nice. Second, the output file could only be played by certain mp3 players which are forgiving about the file format so it may play well on your Linux system (mp3 players are very forgiving there) but not on a mobile device. Third, the output file has bad length and you will have problems moving around in it using most players.

Solution 2: use mp3wrap. This utility is available in most Linux distributions repositories (in Ubuntu it’s named simply mp3wrap). It offers an easy command line and manual page but it does not create a single mp3 file but rather an mp3 file that later can be split to create the original mp3 files. This wrapped file can be played by various Linux players which are forgiving but has lots of the issues discussed in the first solution.

Solution 3: use ffmpeg. This is idea is based on the following post. The idea is to run: ffmpeg -i "concat:file1.mp3|file2.mp3" -acodec copy output.mp3. This is a much better solution as the resulting mp3 file is a single file with correct length and the tags of the first file concatenated.

Solution 4: use a script since the previous solution is hard to use for many files.

#!/usr/bin/python
 
"""
This script will catenate mp3 files correctly using ffmpeg.
see: http://superuser.com/questions/314239/how-to-join-merge-many-mp3-files
"""
 
import subprocess
import sys
 
if len(sys.argv)<3:
        raise ValueError('usage: outfile.mp3 [infile1.mp3] [infile2.mp3] ...')
 
args=['ffmpeg','-i','concat:'+'|'.join(sys.argv[2:]),'-acodec','copy',sys.argv[1]]
#print args
subprocess.check_call(args)
Posted on

Using gpg-agent to write authenticating scripts

Sometimes you want to write a shell or other script, and that script is going to have to run under sudo. Under such conditions if the script does anything that requires authentication it will not act as expected. In plain terms it means that the regular popup for authentication will not appear. The tool maybe written in a way which deals with the problem and falls back on other authentication methods, and yet it may not. In any case what you really want is for your own authentication agent (the little program called gpg-agent which is running on almost every Linux distribution from the time you log in till the time you log out) will do the authentication. This saves you lots of clicking. Imagine that the script has to do something which requires authentication X number of times. If the script does not use an agent it will not be able to cache the pass-phrases and so you will have to retype the pass-phrase several times. It can also be the case that your authenticating agent already has your pass-phrase in it’s cache and you can save typing it yet another time.

Ok. So how do you do it? Well, in your original environment you have a variable called GPG_AGENT_INFO. This variable holds the details of how to connect to your authenticating agent. If you are running regular scripts then this variable, which is an environment variable, is automatically available to them. But if you run your scripts via ssh or sudo then it is not. Just make the variable available to those scripts. Obviously the users that these scripts will be running under will have to have the right level of permission to talk to your gpg agent. How do you make them available? One way is to pass this variable over the command line and turn it into an environment variable as soon as the script starts.

Posted on

Producing MySQL dates from Perl

Ever written the occasional Perl script and wanted to insert the current date and time into a MySQL database? Here is the function to do it. This works for a column of type ‘datetime’.

# function to return the current time in mysql format
sub mysql_now() {
        my($sec,$min,$hour,$mday,$mon,$year,$wday, $yday,$isdst)=localtime(time);
        my($result)=sprintf("%4d-%02d-%02d %02d:%02d:%02d",$year+1900,$mon+1,$mday,$hour,$min,$sec);
        return $result;
}
Posted on

Test procedures for new memory installations

When you buy a new computer or get one and you are not sure of the quality of memory that it has, or when you buy, upgrade or add new memory you should test it before going on to simply use it. The reason is quite intricate. In most probability the memory with will either work or won’t and if the machine works it will be a good indication that the memory works fine. But in a few of the cases your machine may exhibit very strange behavior indeed. Various programs crashing, machine freezes, kernel crashes and the like. In that case, which may happen some time after the upgrade you may fail to connect the symptoms with hardware memory issues and attribute them to other factors like OS upgrades, driver installations or other peripheral failures. This may lead you, as it has led me, on wild goose chases after non issues which will certainly drive you insane or into writing blog posts at 4 AM. So what do I suggest? A simple and short 2 step procedure to be executed when using new memory in order to be sure that your memory is functional and well configured. This can also save you money since from my experience the probability of buying faulty memory is very high (at least 15% from my statistics).

First phase is to run the ubiqitous memtest86+. This is available via the boot menu of most current linux distros. This test runs for some time and long years of using it have led me to a solid statistic according to which if memtest does not find a problem with your memory in the first 30 seconds it will not find any problems in the next 30 hours. But, then again, this is just a statistic, feel free to run this for as long as you wish. If memtest fails return the chips to the manufacturer and get new ones (if you feel that it is the chips fauls – see the note below). If it succeeds then you need to go on to the second phase of configuring the memory properly.

Once the memory is installed open your bios configuration and see how is it configured. How are its parameters (speed and 4 more numbers) set. Is it on automatic or is it on manual? Do you have heterogenous memory banks? If so what is the speed of each and what is the overall speed of the entire memory subsystem? Why should you know all of this info, you rightly ask. Well, in a perfect world you would just buy memory, plug it in and the BIOS would configure and use it properly. Alas, this is not the world we live in. In reality you usualy buy the motherboard at date X and buy the upgrade or new memory at date Y. Y is a couple of years following X. This means that the memory you are buying is too fast for your motherboard. Shouldn’t your BIOS be able to handle this? Well, yes and no. In lots of cases it does manage to handle it but in some it doesn’t and believe me, you don’t want to get stuck in the latter.

In my case I installed a DDR2 800 MHz memory on a standard Intel board which did not complain and the BIOS ran that memory at the auto-selected optimal speed of 800 MHz. There was no problem with the memory and so memtest ran smoothly. It’s just that when the 2 cores were accessing it together with high speed then put more pressure on it than memtest did and memory faults started happening.

Second test is to just see if the memory is working properly with multi core. This phase can also be used to “overclock” your RAM and to make sure that you will not be experiencing any weird side effects from this overclocking. In this phase we will test the memory in practice using the N cores. I found that the best way to achieve this is to just compile the Linux kernel on the machine using make -j N where N is the number of your cores. Whenever I had memory problems this compilation would crash in some spectacular way and in random places and so served as a clear indication of RAM issues.

If you want to learn more about memtest and dual core checkout this and this in the memtest86+ discussion board. It seems that memtest86 (as opposed to memtest86+) does have multi core support. Cool. The problem is that on Linux systems usually memtest86+ is the only one installed…

If you want to know how to compile a Linux kernel learn more at this URL.

memtester: There is a package called memtester in Linux which will test memory from user space. In Ubuntu this package is simply called memtester. It is developed here. I have tried it out and it is a fine piece of code but does not do multi-threaded testing with CPU affinity. You have to do that on your own at the command line by running two instances of memtester and assigning them to different CPUs via taskset. Another problem with memtester is that you need to let it know how much ram to test which is very hard to do since you want to test as much as possible. This means that you need to calculate the size to test which is roughly total_ram_size-(size_of_os+size_of_all_currently_running_programs) which is a hard to calculate and if you miscalculate the program may fail since it locks the memory that it gets using mlock, which you need to have permission to perform. It may also throw other programs that you are running at the time into swap (since they are not locked into memory).

The kernel compilation mentioned above is better in my opinion due to the following reasons: it uses all of your CPUs and it also uses every last bit of RAM you have since the kernel is big and during the compilation stage will use up all of your Linux cache which means all of your spare memory.

Note: as mentioned in the memtester documentation, if you do find any problems with your memory it may not be the fault of your memory chips at all. It may be the fault of your motherboard not supplying enough power for the chips or the CPU, it may be an overheating CPU, a mis-configured BIOS or other reasons.

Please leave comments if you think that I am wrong in any of the above and I promise to improve the post if you convince me that I could do better…

Posted on

Configuring ssh server for pubkey + password authentication

In a struggle to secure my home computer I did battle with the ssh server once again to configure it “just the way I want it” ™. I prefer pubkey + password since this ensures that if I lose the laptop/phone/whatever then the lucky finder will not find his/her way into my home computer.

So, without further fanfare here are various bits that need to be done.

Configuring the ssh server
edit /etc/ssh/sshd_config and use the following entries:
Protocol 2 # protocol 1 is outdated
PubkeyAuthentication yes # I want public key to be used for authentication (and possibly to be combined with a pass phrase)

And of course there a bunch of authentication protocols that are not needed:
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
PasswordAuthentication no
UsePAM no

Creating the keys
Still on the server in the home folder of the user you want to login remotely with, create the private/public pair using ssh-keygen -t dsa in ~/.ssh (the default location for ssh-keygen). You get two files: id_dsa (private key) and id_dsa.pub (public key).

I used dsa keys in this post and you can use rsa keys if you pass -t rsa to ssh-keygen.

In the same folder on the server create a file called authorized_keys which has the public key (it can just be a copy of id_dsa.pub but has the potential to contain many keys – possibly one per user that can connect to said account or one per roaming device).

When creating the key pair you will be prompted for a pass phrase. This is where you choose whether or not you will need a pass phrase (which acts as a password) in order to access this account. If you leave the pass phrase empty you’re allowing key only access with no password which is dangerous since if anyone gets a hold of your roaming device he/she can access your account with no extra data.

Distributing the keys
Copy the private key ~/.ssh/id_dsa to the roaming devices you want to access the server from (laptop, phone, whatever). If the roaming device is a Linux box then put the private key in the same location (~/.ssh/id_dsa) in the home folder of the user that wishes to access the server. If you are using some other ssl tool besides command line ssh on a Linux box to access the server then it should have a place where you plug the private key into. If it doesn’t have such a place then dump it. Putty (a widely used ssh client on windows) has an option to use a private key for connection.

Note:
While trying this out a lot of people seem to fail because they do all the experimentation on a desktop. In a desktop there is a system called ssh-agent which does the authentication for you in order to save you typing the same password multiple times. This agent is a problem when doing experimentation since it needs to be notified that you switched keys. So, every time you switch keys (regenerate the ~/.ssh/{id_dsa,id_dsa.pub} files) you need to run ssh-add to let the agent know this. Another option is not do all of the experimentation from a desktop but rather from a login shell (Ctrl+Alt+1 or whatever) so that the agent does not come into the game (which is complicated enough without it). Only after everything is setup re login to the graphical desktop and try everything out.

Posted on

Real time programming tips: running critical tests at application startup

There is much accumulated wisdom in the embedded systems programming field as to how to correctly write a real time application. Examples of this wisdom could be found in the methodology of breaking up the application to a startup phase and a run phase, avoiding exiting the application, avoiding dynamic memory allocation and deallocation at runtime and more. There is also much accumulated wisdom in the programming field in general where a very important principle is ones control of ones software, as opposed to the other way around, and the notion of finding bugs and problems early whether that be in code writing, QA, deployment or beginning of execution.

The combination of the two aforementioned elements forms the principle of critical condition testing at application startup. According to this principle you should put all environmental concerns as tests to be executed at the startup phase of your embedded application. Environmental conditions to be checked may include, among others, the following:

  • Operating system or C library versions as the software may be adjusted for specific versions of these.
  • Real time patch availability and version as the software may require real time capabilities.
  • System real time clock accuracy as the software may require the availability of an accurate system clock.
  • User under which the software is running as the software may require special permission or user at some point in it’s execution.
  • Free disk space availability as the software may require some disk space.
  • Free memory availability as the software may accidentally be run on a system with less than the required amount.
  • A previously running instance of the same or other software that may hinder the softwares operation.
  • The availability of certain API’s of the kernel or certain kernel modules which are required.
  • The availability of certain devices (/dev files) with permission to access these.

All of these checks should be run in the first second or so of the software’s execution and, contrary to normal wisdom, should cause the software to halt and not proceed with normal execution. The reasons for this scary tactic is that:

  • You may miss error printouts from your application and so run around trying to find errors in all the wrong places.
  • You want the errors to show up early and anything that can be made to show up early should be made so.
  • I have seen programmers confidence in their hardware/OS/environment break too many times and lead to endless hours of wasted effort which could have been prevented by using this strategy.
  • Some requirements are of the make or break type and you really should not go on running without them.
  • Some of the requirements of real time and embedded systems are so subtle that you would not even notice these break as error in runtime but rather get weird behavior from your system. These are very hard to pin point and should be avoided.

These checks should also be written in a way which enables them to be easily removed when the system has stabilized, when it’s environment has stabilized (like when the system moves to production) or in order to reduce boot time.

This principle is especially important to real time and embedded systems programmers because of a few factors:

  • real time and embedded systems are harder to debug and monitor.
  • real time and embedded systems have less tools on them that enable one to find bugs.
  • real time and embedded applications are much more sensitive than other types of applications to various changes in the environment.
  • embedded systems programs usually interact with other systems which are in the debug phase as well and so may throw the developers on endless bug hunts which waste valuable time and cause the developers to mistrust their entire design or the system and tools that they are using.
  • embedded software systems usually run 24/7 and have only an end user interface. if at all. Due to this many embedded applications only output a long log and as such either encourage the user to disregard the log completely or make the task of discerning which log lines pertain to critical errors a daunting task.
Posted on

Command line id3 2.4 tagging in Linux

In the past couple of weeks I have been dabbling with command line to tag and inspect tags of mp3 files. Linux has an abundance of tools to fill this niche and it’s confusing to know what to use. What follows are my conclusions.

id3: only handles 1.1 tags.
id3v2: does not handle 2.4 tags.
id3tool: does not handle 2.4 tags. Not maintained.
eyeD3: handles everything (especially 2.4 tags), has lots of options and seems to be supported.

So it’s fairly obvious what to choose. Please let me know if you have any more candidates to add to this list or you disagree with any of my conclusions.

Posted on

Finding broken symbolic links

I sometimes need to find all broken symbolic links in a folder, recursively or not. find(1) is the all UNIX right tool for the job as far as finding files is concerned but it does not have an explicit -and -type brokenlink option…

Some solutions involve sending the output of find(1) to some other tool. These solutions are sub-optimal in that once you leave the comfort of find(1) you give up the ability to use many of it’s fine features and run into other problems (file names with white space characters just to name one such problem).

Other solutions involve doing find -L . -type l which forces find(1) to follow all symbolic links and ultimately only print those which it cannot follow. This solution has other drawbacks. One is that you do not necessarily want find(1) to follow every symbolic link since this may cause it to wander to huge areas of your hard drive that you do not wish to scan. Another deficiency is the fact that there is a difference between a symbolic link that cannot be followed and one which cannot be read.

My solution is this:

find . -type l -and -not -exec test -e {} \; -print
Posted on

Switching Java versions on a Debian/Ubuntu system

I recently found some issues with the openjdk Ubuntu/Debian default Java implementation. Specifically I had issues with their web start support (javaws). I found that the Sun implementation of Java did not have such a deficiency and the Sun implementation is available through the regular Ubuntu/Debian package sources. I installed the Sun implementation and wanted to switch the default Java to that version.

So what have I found out ?

When you want to switch to the Sun implementation:

sudo update-java-alternatives --set java-6-sun

When you want to go back to the openjdk implementation:

sudo update-java-alternatives --set java-6-openjdk

Notice that once you do any of the above you leave “auto” mode which means that new installation of Java implementation will not switch your default one. If that is what you want then ok. If not you can return to “auto” mode with:

sudo update-java-alternatives --auto

Posted on

Using “noatime” on a running Linux system

When upgrading my Ubuntu system my /etc/fstab got overwritten by the upgrade process. It seems that the new /etc/fstab file did not keep my old preferences for file systems. I didn’t notice this for some time but what I did notice was that my system was sluggish. After some time I recalled that I had previously used noatime as a mount option for all of my hard drives which gave me some more speed and treated my hard drives with a softer touch.

First lets explain what noatime means. atime or Access Time is an attribute stored by all well behaved UNIX file systems for each and every file. It is one of 3 dates stored: meta info modification time – ctime, last modification time – mtime and last acces time – atime. Out of the 3 atime is the most controversial since it means that for every read from the disk there is a write operation. This is one of the worst defaults in your UNIX system.

The solution is just to disable atime altogether. Warning – this may cause some weird applications that rely on atime to break. If you want your system to be as “default” as possible don’t do what I suggest. If you want better performance and hard disk lifetime and on the other hand don’t mind parting ways with one or two misfit applications then this trick is for you.

How do you do it? Just edit /etc/fstab and add noatime at the 4′th column where file system mount options are for any file system you want to avoid access time updating. Reboot your system. Run mount(1) to see that all your file systems are mounted correctly. Enjoy.

What applications break? Actually – I have yet to see an application break because of this change. I have been running with “noatime” for 2 years now and all the applications seem to behave well. If you know of an application that breaks please let me know…