Original from German blog:
Linux From Scratch blogspot German Lanuage post
I build a Linux as I like it - 1
For many years I have been an enthusiastic user of a very special Linux variant called " Linux from scratch ".
As the name suggests, a Linux system is built from scratch from scratch. The source texts are used to translate all the packages yourself and to equip a PC (or another computer system) with a Linux that can be started from the hard drive.
Of course, you have a chicken and egg problem: you need a Linux to build the new Linux. There are several possible solutions to this problem: On the one hand, the LFS can be assembled in a virtual machine. Alternatively, you can first set up a small partition with a different Linux on the target computer, start from there and then throw the LFS onto the free space on the hard drive. The Linux for building can then be kept as an emergency system afterwards, or the partition can be made into a swap partition or used for other purposes. For the very first ("bootstrap") Linux, a mini-partition of 8-16 GB is sufficient, and you can then install a Linux Mint or Fedora there, for example.
"Linux from Scratch" is not only a tutorial for a Linux to build yourself, but is mainly intended to gain experience in building Linux programs from the source code.
You learn a lot, namely the relationships between the various programs, configuration files, but also a lot about networks, and above all, of course, about the technical requirements, including the use of makefiles, and you learn to read log files to resolve problems to eliminate.
At "LFS" there are various ways to improve how you can make life difficult and exciting for yourself. You can build the basic system , then you "only" have a Linux command line.
Beyond Linux from scratch "(BLFS) it is possible to build a server with which you can, for example, build a firewall, a NAS, a web server, or much more. Or you build a desktop system with a graphical user interface, and at the end Then the self-compiled Firefox is displayed. Doesn't that sound cool?
One level of difficulty above is the automation of the construction process - and this is mainly what I want to write here. The instructions from "LFS" for a base system and for the "BLFS" are pretty good. If you stick to it, you have a good chance of manually creating a working system step by step. You go through all the chapters one after the other on the browser and carry out the individual steps according to the instructions. To be honest, that's a hassle the first time - I speak from experience ;-). On the other hand, it is exciting to see how the Linux system grows with every step and actually looks more and more like what you know as a Linux user.
The automation " Automated Linux from scratch " (ALFS), however, is in a README- The file is only sparsely described. Once you have internalized the system, you have a wonderful and very esoteric possibility to install updates or to completely regenerate the entire system at the push of a button.
In this way I built a Linux overnight - or better said: had it built that would be protected against Specter and Meltdown with the latest kernel 4.15 and C compiler gcc 7.3 . - If, yes, if my craft system would have been a 64-bit system (x86_64) and not 32-bit (i686). And if I had the feeling that I was really threatened by these vulnerabilities. But I think I have to write my own article on the subject of risk analysis.
And one step above you do not use the "stable" version of the books, but the LFS and BLFS books that are currently in development. The authors of the chapters are constantly busy integrating new versions of the packages and adapting the articles. If, for example, the procedure for a package changes how the package is compiled in an executable manner, this will be adopted extremely quickly in the "development" version of LFS or BLFS. This is exactly what happened to me while writing this article: I started the Makefile , and in the middle of it, the book switched to ncurses 6.1. As soon as I was finished, the developers released the new glibc version 2.27. This is how it works ...
To come back to "Meltdown" and "Specter" again: the LFS book recently started to contain brand new kernel and gcc versions that are supposed to fix these security holes. This is of course always "work in progress". The current status is that Specter V2 and Meltdown are fixed, and there is still some development work to be done for Specter V1. The old 32-bit processors are unfortunately lagging behind, for 64-bit (i.e. everything that is roughly less than 10 years old) all fixes should definitely be in progress.
So now, as promised, a more detailed description of the automation for LFS (ALFS). To do this, first download the current package . For historical reasons, the package name is " jhalfs "and honors the first developer of the process with his initials" jh ".
As the name suggests, a Linux system is built from scratch from scratch. The source texts are used to translate all the packages yourself and to equip a PC (or another computer system) with a Linux that can be started from the hard drive.
Of course, you have a chicken and egg problem: you need a Linux to build the new Linux. There are several possible solutions to this problem: On the one hand, the LFS can be assembled in a virtual machine. Alternatively, you can first set up a small partition with a different Linux on the target computer, start from there and then throw the LFS onto the free space on the hard drive. The Linux for building can then be kept as an emergency system afterwards, or the partition can be made into a swap partition or used for other purposes. For the very first ("bootstrap") Linux, a mini-partition of 8-16 GB is sufficient, and you can then install a Linux Mint or Fedora there, for example.
"Linux from Scratch" is not only a tutorial for a Linux to build yourself, but is mainly intended to gain experience in building Linux programs from the source code.
You learn a lot, namely the relationships between the various programs, configuration files, but also a lot about networks, and above all, of course, about the technical requirements, including the use of makefiles, and you learn to read log files to resolve problems to eliminate.
At "LFS" there are various ways to improve how you can make life difficult and exciting for yourself. You can build the basic system , then you "only" have a Linux command line.
Beyond Linux from scratch "(BLFS) it is possible to build a server with which you can, for example, build a firewall, a NAS, a web server, or much more. Or you build a desktop system with a graphical user interface, and at the end Then the self-compiled Firefox is displayed. Doesn't that sound cool?
One level of difficulty above is the automation of the construction process - and this is mainly what I want to write here. The instructions from "LFS" for a base system and for the "BLFS" are pretty good. If you stick to it, you have a good chance of manually creating a working system step by step. You go through all the chapters one after the other on the browser and carry out the individual steps according to the instructions. To be honest, that's a hassle the first time - I speak from experience ;-). On the other hand, it is exciting to see how the Linux system grows with every step and actually looks more and more like what you know as a Linux user.
The automation " Automated Linux from scratch " (ALFS), however, is in a README- The file is only sparsely described. Once you have internalized the system, you have a wonderful and very esoteric possibility to install updates or to completely regenerate the entire system at the push of a button.
In this way I built a Linux overnight - or better said: had it built that would be protected against Specter and Meltdown with the latest kernel 4.15 and C compiler gcc 7.3 . - If, yes, if my craft system would have been a 64-bit system (x86_64) and not 32-bit (i686). And if I had the feeling that I was really threatened by these vulnerabilities. But I think I have to write my own article on the subject of risk analysis.
And one step above you do not use the "stable" version of the books, but the LFS and BLFS books that are currently in development. The authors of the chapters are constantly busy integrating new versions of the packages and adapting the articles. If, for example, the procedure for a package changes how the package is compiled in an executable manner, this will be adopted extremely quickly in the "development" version of LFS or BLFS. This is exactly what happened to me while writing this article: I started the Makefile , and in the middle of it, the book switched to ncurses 6.1. As soon as I was finished, the developers released the new glibc version 2.27. This is how it works ...
To come back to "Meltdown" and "Specter" again: the LFS book recently started to contain brand new kernel and gcc versions that are supposed to fix these security holes. This is of course always "work in progress". The current status is that Specter V2 and Meltdown are fixed, and there is still some development work to be done for Specter V1. The old 32-bit processors are unfortunately lagging behind, for 64-bit (i.e. everything that is roughly less than 10 years old) all fixes should definitely be in progress.
So now, as promised, a more detailed description of the automation for LFS (ALFS). To do this, first download the current package . For historical reasons, the package name is " jhalfs "and honors the first developer of the process with his initials" jh ".
In order for jhalfs to work, a few additional Linux packages must be installed on the guest system, because they are normally not required for end users the C compiler gcc and some auxiliary programs and libraries like bison , awk , ncurses etc. The nice thing is: when you start jhalfs, it checks what is still missing and complains, "Install Subversion to download the latest files from the version control system."
The automation actually starts with the very first steps that one would carry out according to the LFS book: creating a new Unix user "lfs" for compiling the packages, creating directories, etc. The only step that you have to carry out yourself is creating a hard drive partition and then mounting it so that the scripts can find it. Traditionally this is / mnt / lfs , but you can change it if you know what you are doing.
First a few words about the design of jhalfs: the authors create LFS and the other books in a special XML format called " DocBook "; turn it into tricky XSLT-Transformations then readable books into HTML, PDF or even TeX. jhalfs also uses XSLT transformations to filter out the command line instructions from the book and uses them to create a Makefile and installation scripts for each chapter, ie each package to be installed. Automation with ALFS is a two-step process: first, the commands for building are filtered out of the book and converted into a makefile with auxiliary scripts, and in a second step this list of commands is executed.
The authors also provide a complete list of the software packages with version numbers that you can either download yourself or that can be retrieved automatically if you have a reasonably fast internet connection. Roughly estimated, you have to download around 400 MB of packages for LFS and up to 2 GB for BLFS, depending on the size.
The packages should be stored in a directory / mnt / lfs / sources below the future LFS partition so that all packages can be found during the process. To do this, a Unix technique called "chroot" (change root) is used to simulate a running program with a different hard drive structure - effectively, this is used to pretend that it is already running in the "real" LFS environment.
The Makefile for LFS is created in a new subdirectory, into which you have to change after a few basic assumptions have been made. Just like / sources , this directory should be on the future LFS partition so that all scripts can also be accessed in the "chroot" environment. If you now start " make " there , after a few hours (depending on the speed of the computer *) you should have an almost bootable Linux on the new partition. Steps that can be automated, but not mandatory, are compiling a kernel and specifying the partitions to be mounted on startup.
*)As a rough guide, two comparative figures that I measured with the current LFS version. The package that takes by far the longest is the final step in compiling the C compiler. This takes 540 minutes on an old Pentium 4, and just under 230 minutes on a more recent i5-3350. An SSD instead of a hard drive is a great help in saving time.
The first step after unpacking the jhalfs package is to change to the directory and enter "make" there. If you have ever compiled a kernel yourself, you will immediately experience an aha effect: the configuration uses the same text interface for menu selection as the kernel for "make menuconfig". Here you can choose LFS / BLFS, "stable" or "development", whether the kernel should also be compiled automatically and a lot more. The basic settings are not that bad, and you should only change what you understand ;-)
After "exit" from this ALFS menu, the question arises whether you are satisfied with the configuration. If "yes", the already mentioned checks are carried out to determine whether all developer packages are available on the guest Linux and meet certain minimum requirements, such as gcc at least in version 4.6 (because earlier versions contain errors, so that the LFS would not work or packages are not compiled correctly, etc.). If all of this has been checked successfully, change to the new directory (as a user, not as root) and start " make " again there .
Cool socks take a look at the files below chapter04 , 05 , in the subdirectory lfs-commands. First start with the editor before you start make . With the files for chapter 7, in particular, you could directly get the idea of editing the passages marked with ** EDITME ** ;-)
At the same time, it is advisable to open the same version of the book in a browser and to read the chapters that the automatism is currently trying to build. In the event of problems, you can read what is happening in the respective chapter.
It will probably not work completely successfully the first time; But that's not bad: if an error is reported, you can look at the log file for this step from the logs subdirectory , correct the error and then simply go back to it "make ". The make program is so clever that it continues at the same point. Some packages are built multiple times, for example the C compiler and some auxiliary programs. The purpose of this is to avoid accidentally building dependencies on the guest system If an important program were linked to the guest system's C library, it would no longer work after booting, so an "intermediate system" is built in a different path ( / tools ), which can then compile the final LFS.
A few general tips if the compilation of a package fails: gcc 7.3 is "bleeding edge", so brand new that you can cut your fingers when you use it ;-). I mean to say that there are so many changes in it, for example adjustments to the current C and C ++ standards, which have not yet reached most packages and the brains of developers. But: you want to use gcc 7.3 and current binutils, because this is where the repair work for Meltdown and Specter takes place. Nevertheless the recommendation again: first build a "stable" LFS and then switch to "development" as an increase.
Typical compilation problem:
The only important point is that the file system format of the boot drive should be compiled in the kernel. It is of no use if the root file system is formatted with ext4 and the kernel should first learn the ext4 capability with a module that is on the ext4 hard drive - this is where the cat will bite its tail. Oh yes:make modules_install "and check it in / lib / modules !
For later educational and research purposes you can of course also provide an" initial ramdisk "(initrd) and store the required modules and other files there, but if you have a tailor-made kernel for this special PC generated, this is not necessary at first. You can do it later to activate microcode updates via "early load", but it is not essential for life. For intel processors there are the microcode files .
Before Reboot you should then look in the new / etc directory (which is currently still called / mnt / lfs / etc ) for files that contain "** EDITME **". There you can then incorporate the host name, IP address, name server, etc. that match your own environment (if you do not operate your own DNS server, the IP address of the DSL router that serves as a DNS forwarder for the requests The files can be found with a courageous " grep -l EDITME / mnt / lfs / etc / * ", and if you want to edit all of them in one go, you just throw the list of results to an editor: " vi $ (grep -l EDITME / mnt / lfs / etc / *) " .
The most important file here is / etc / fstab which describes which partition of the hard disk has which purpose and where it belongs in the file system. A swappable partition should definitely be provided, even if the PC feels that it has enough RAM installed. A reason for this can be found here (the author of the article works in a data center on Facebook). The re used to be a rule of "twice as much swap as RAM", but in principle I would allow between 2 and 8 GB of swap for normal desktop systems.
One thing you should definitely not forget: give the root user of the new LFS system a password before rebooting. Otherwise you can boot wonderfully, but not log in ;-)
In all honesty: the feeling is indescribable when the self-compiled Linux actually boots for the first time and you see the login prompt ;-)
If the new system does not want to boot, you have to investigate the cause. A "kernel panic" occurs if the root file system was not found. The kernel module for the file system in question is then probably missing. Or the information in /boot/grub/grub.cfg does not match which partition it is. Unfortunately, the spelling of grub and Linux differ a little here. "sda" for Linux is "hd0" in grub.
A mistake I made once: all USB drivers were declared as modules and then not defined in / etc / modules . In this situation it becomes difficult for a USB keyboard to deliver its inputs :-).
In such situations, you boot the guest system again and mount the LFS partition again under / mnt / lfs and begins to repair.
When it boots up and you can log in, repairs are easier: the error messages, for example from the init scripts, are still on the screen, and you can correct everything you need to do step by step and set up the new, self-made Linux system. The LFS already has everything on board to translate and install packages independently.
After the successful LFS, the BLFS continues.
A few suggestions on how to make the LFS a little more convenient to use:
The automation actually starts with the very first steps that one would carry out according to the LFS book: creating a new Unix user "lfs" for compiling the packages, creating directories, etc. The only step that you have to carry out yourself is creating a hard drive partition and then mounting it so that the scripts can find it. Traditionally this is / mnt / lfs , but you can change it if you know what you are doing.
First a few words about the design of jhalfs: the authors create LFS and the other books in a special XML format called " DocBook "; turn it into tricky XSLT-Transformations then readable books into HTML, PDF or even TeX. jhalfs also uses XSLT transformations to filter out the command line instructions from the book and uses them to create a Makefile and installation scripts for each chapter, ie each package to be installed. Automation with ALFS is a two-step process: first, the commands for building are filtered out of the book and converted into a makefile with auxiliary scripts, and in a second step this list of commands is executed.
The authors also provide a complete list of the software packages with version numbers that you can either download yourself or that can be retrieved automatically if you have a reasonably fast internet connection. Roughly estimated, you have to download around 400 MB of packages for LFS and up to 2 GB for BLFS, depending on the size.
The packages should be stored in a directory / mnt / lfs / sources below the future LFS partition so that all packages can be found during the process. To do this, a Unix technique called "chroot" (change root) is used to simulate a running program with a different hard drive structure - effectively, this is used to pretend that it is already running in the "real" LFS environment.
The Makefile for LFS is created in a new subdirectory, into which you have to change after a few basic assumptions have been made. Just like / sources , this directory should be on the future LFS partition so that all scripts can also be accessed in the "chroot" environment. If you now start " make " there , after a few hours (depending on the speed of the computer *) you should have an almost bootable Linux on the new partition. Steps that can be automated, but not mandatory, are compiling a kernel and specifying the partitions to be mounted on startup.
*)As a rough guide, two comparative figures that I measured with the current LFS version. The package that takes by far the longest is the final step in compiling the C compiler. This takes 540 minutes on an old Pentium 4, and just under 230 minutes on a more recent i5-3350. An SSD instead of a hard drive is a great help in saving time.
The first step after unpacking the jhalfs package is to change to the directory and enter "make" there. If you have ever compiled a kernel yourself, you will immediately experience an aha effect: the configuration uses the same text interface for menu selection as the kernel for "make menuconfig". Here you can choose LFS / BLFS, "stable" or "development", whether the kernel should also be compiled automatically and a lot more. The basic settings are not that bad, and you should only change what you understand ;-)
After "exit" from this ALFS menu, the question arises whether you are satisfied with the configuration. If "yes", the already mentioned checks are carried out to determine whether all developer packages are available on the guest Linux and meet certain minimum requirements, such as gcc at least in version 4.6 (because earlier versions contain errors, so that the LFS would not work or packages are not compiled correctly, etc.). If all of this has been checked successfully, change to the new directory (as a user, not as root) and start " make " again there .
Cool socks take a look at the files below chapter04 , 05 , in the subdirectory lfs-commands. First start with the editor before you start make . With the files for chapter 7, in particular, you could directly get the idea of editing the passages marked with ** EDITME ** ;-)
At the same time, it is advisable to open the same version of the book in a browser and to read the chapters that the automatism is currently trying to build. In the event of problems, you can read what is happening in the respective chapter.
It will probably not work completely successfully the first time; But that's not bad: if an error is reported, you can look at the log file for this step from the logs subdirectory , correct the error and then simply go back to it "make ". The make program is so clever that it continues at the same point. Some packages are built multiple times, for example the C compiler and some auxiliary programs. The purpose of this is to avoid accidentally building dependencies on the guest system If an important program were linked to the guest system's C library, it would no longer work after booting, so an "intermediate system" is built in a different path ( / tools ), which can then compile the final LFS.
A few general tips if the compilation of a package fails: gcc 7.3 is "bleeding edge", so brand new that you can cut your fingers when you use it ;-). I mean to say that there are so many changes in it, for example adjustments to the current C and C ++ standards, which have not yet reached most packages and the brains of developers. But: you want to use gcc 7.3 and current binutils, because this is where the repair work for Meltdown and Specter takes place. Nevertheless the recommendation again: first build a "stable" LFS and then switch to "development" as an increase.
Typical compilation problem:
- a C program uses data types that are now declared in a newly introduced include file <stdint.h> and no longer in <stdlib.h> . For the time being, you have to put the additional include command somewhere before the program can be compiled successfully. Make kindly tells you exactly where a problem has occurred.
- when correcting a problem, files or directories were edited or edited as root. This leads to a subsequent error, because LFS wants to do everything with the user " lfs " and then it can be aborted due to authorization problems (I can do it all the time ).
chown lfs: lfs is your friend.
- the internal gcc macro __sigemptyset no longer exists. You can use sigemptyset (without the underscores) instead.
- the macros major and minor for Devices are now in <sys / sysmacros.h> to find .
- grub cannot be compiled with binutils 2.30 on 32-bit systems. In addition to the configure switches in the LFS book , you have to specify the switch " --enable-64-bit-bfd ". Alternatively, use binutils 2.29.1 instead of 2.30 .
The only important point is that the file system format of the boot drive should be compiled in the kernel. It is of no use if the root file system is formatted with ext4 and the kernel should first learn the ext4 capability with a module that is on the ext4 hard drive - this is where the cat will bite its tail. Oh yes:make modules_install "and check it in / lib / modules !
For later educational and research purposes you can of course also provide an" initial ramdisk "(initrd) and store the required modules and other files there, but if you have a tailor-made kernel for this special PC generated, this is not necessary at first. You can do it later to activate microcode updates via "early load", but it is not essential for life. For intel processors there are the microcode files .
Before Reboot you should then look in the new / etc directory (which is currently still called / mnt / lfs / etc ) for files that contain "** EDITME **". There you can then incorporate the host name, IP address, name server, etc. that match your own environment (if you do not operate your own DNS server, the IP address of the DSL router that serves as a DNS forwarder for the requests The files can be found with a courageous " grep -l EDITME / mnt / lfs / etc / * ", and if you want to edit all of them in one go, you just throw the list of results to an editor: " vi $ (grep -l EDITME / mnt / lfs / etc / *) " .
The most important file here is / etc / fstab which describes which partition of the hard disk has which purpose and where it belongs in the file system. A swappable partition should definitely be provided, even if the PC feels that it has enough RAM installed. A reason for this can be found here (the author of the article works in a data center on Facebook). The re used to be a rule of "twice as much swap as RAM", but in principle I would allow between 2 and 8 GB of swap for normal desktop systems.
One thing you should definitely not forget: give the root user of the new LFS system a password before rebooting. Otherwise you can boot wonderfully, but not log in ;-)
In all honesty: the feeling is indescribable when the self-compiled Linux actually boots for the first time and you see the login prompt ;-)
If the new system does not want to boot, you have to investigate the cause. A "kernel panic" occurs if the root file system was not found. The kernel module for the file system in question is then probably missing. Or the information in /boot/grub/grub.cfg does not match which partition it is. Unfortunately, the spelling of grub and Linux differ a little here. "sda" for Linux is "hd0" in grub.
A mistake I made once: all USB drivers were declared as modules and then not defined in / etc / modules . In this situation it becomes difficult for a USB keyboard to deliver its inputs :-).
In such situations, you boot the guest system again and mount the LFS partition again under / mnt / lfs and begins to repair.
When it boots up and you can log in, repairs are easier: the error messages, for example from the init scripts, are still on the screen, and you can correct everything you need to do step by step and set up the new, self-made Linux system. The LFS already has everything on board to translate and install packages independently.
After the successful LFS, the BLFS continues.
A few suggestions on how to make the LFS a little more convenient to use:
- gpm (copy + paste with mouse in the text surface)
- cpio (for building initrd)
- openssl (ssl library)
- openssh (ssh server and client)
- dhcpcd (have the IP address set via DHCP)
- nfs-utils (if you have a NAS, e.g. a Fritzbox, or want to build it yourself)
- samba (if you want to build a NAS yourself or a Windows domain controller)
- lm_sensors (hardware monitoring of CPU and fan)
- lsusb (list USB devices)
- lspci (list PCI devices)
- ghostscript (for PDF)
- cups (for printing)
- Firefox (for cat videos on the internet)
