2017. szeptember 19., kedd

Alpine Linux install

If you start working on Docker these days it is inevitable to run into Alpine Linux at certain point. It offers 1/10 of the size of a Debian based container (5 mb vs 50 mb), and many application stacks already offer Alpine based image to derive your micro services from. Debian can be slimmed down pretty well and there is a loyal following using it for containers and servers as well.
To try it out I decided to build a VM based on Alpine so I can evaluate it as a container image base.
Adventures in the Alpines so far:
  1. There is a pretty good starting guide, with some caveats:
    1. use bridged connection instead of nat if you want to see your vm from your host/other machines in the network, and performs better too
    2. use the latest-stable url to enable community repository, otherwise you could get some ugly kernel incompatibility errors
  2. Minimalist samba install (loosely based on this wiki page)
    1. apk add samba
    2. mv /etc/samba/smb.conf /etc/samba/smb.conf.bak
    3. nano /etc/samba/smb.conf
    4. Enter the above to the text editor, save and exit
[global]
workgroup = WORKGROUP
netbios name = server1
security = user
map to guest = Bad User
dns proxy = no
  1. Start/enable service
    1. rc-update add samba
    2. rc-service samba start
  2. Enable ssh access
    1. nano /etc/ssh/sshd_config
    2. #PermitRootLogin prohibit-password => PermitRootLogin yes
    3. rc-service sshd restart
  3. Docker install is fairly simple
    1. apk add docker
    2. rc-update add docker boot
    3. rc-service docker start
  4. Some other utilities to install:
    1. apk add git
    2. apk add nodejs

2017. szeptember 16., szombat

CentOS howto V - Custom systemd service

  1. nano /etc/systemd/system/cloning-vat.service
[Service]
WorkingDirectory=/home/shared/cloning-vat/
ExecStart=/usr/bin/npm start
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=cloning-vat
User=nodejs
Group=nodejs
Environment=NODE_ENV=development

[Install]
WantedBy=multi-user.target
  1. systemctl daemon-reload
  2. systemctl enable cloning-vat
  3. systemctl start cloning-vat

CentOS howto IV - Poking holes in security (in a meaningful way)


  1. Add an existing service
    1. firewall-cmd --permanent --zone=public --add-service=https
    2. firewall-cmd --reload
  2. Create a new service (to be added as above). In this example we will be using livereload default port 35729
    1. firewall-cmd --permanent --new-service=live-reload
    2. firewall-cmd --permanent --service=live-reload --set-description="live reload"
    3. firewall-cmd --permanent --service=live-reload --set-short="live reload"
    4. firewall-cmd --permanent --service=live-reload --add-port=35729/tcp
  3. Enabling an application (node.js in this example) to bind ports <1024
    1. setcap 'cap_net_bind_service=+ep'/ usr/bin/node

CentOS howto III - Advanced samba

  1. Create shared folder and grant permissions
    1. mkdir /home/shared/
    2. chmod -R 0777 /home/shared/
    3. chown -R nobody:nobody /home/shared/
    4. chcon -t samba_share_t /home/shared/
  2. add the text below to the /etc/samba/smb.conf
[shared]
path = /home/shared
browsable = yes
guest ok = yes
read only = no
  1. Restart samba
    1. systemctl restart smb.service
    2. systemctl restart nmb.service
  2. Test samba
    1. testparm /etc/samba/smb.conf

CentOS howto II - Shared folders

  1. Install Guest additions on the guest
    1. Prerequisites on the guest OS side:
      1. yum install dkms kernel-devel
      2. yum groupinstall "Development Tools"
    2. and now the additions themselves:
      1. Devices / Insert Guest Additions CD image...
      2. mount /dev/sr0 /mnt
      3. cd /mnt
      4. ./VBoxLinuxAdditions.run
    3. Create a shared folder:
      1. Devices / Shared Folders / Shared folder settings
      2. Add new..
      3. Select "Auto mount" and "Permanent" options
      4. you can find the shared folder in /media/sf_{SHARE_NAME}
  2. Install guest additions on the host
    1. {VBoxManage} setextradata {VM_NAME} VBoxInternal2/SharedFoldersEnableSymlinksCreate/{SHARE_NAME} 1
      1. {VBoxManage} executable (ususally in C:\Program Files\Oracle\VirtualBox)
      2. {VM_NAME} is the name of the VM
      3. {SHARE_NAME} name of the share as in shared folders settings on in /media/sf_{SHARE_NAME}
    2. Run the VM as administrator. Simplest way to get that going is:
        1. In VirtualBox UI, right click on the VM and "Create Desktop shortcut"
        2. Right click on the desktop icon, properties, advanced, check "Run as administrator"
    3. Start the VM
    4. Watcher configuration: The Guest OS have 0 idea about any changes outside of its jurisdiction. In this setup this pretty much makes watching files from the Linux guest side nearly impossible - except if you use polling. This is an example take from my project's browsersync configuration, something similar must work for any other chokidar based frameworks as well.
    var sync = require("browser-sync").create();
    sync.init({
      server: 'dist',
      port: 80,
      watchOptions: {
        usePolling: true
      }
    });

    CentOS howto I - basic VM

    1. Install Oracle VirtualBox
    2. Download CentOS DVD ISO: https://www.centos.org/download/
    3. Create a new VM
      1. mount DVD image as optical drive
      2. swap the network adapter type to Bridged network
    4. Start the VM, Install CentOS with "Minimal install"
      1. Start with partitioning, defaults should be good
      2. Go to network
        1. Make sure adapter configured to start automatically
        2. Rename the host
      3. Set time zone and ntp
    5. Login (as root) and make sure the adapter is set to autostart
      1. nmtui
      2. Edit a connection / enp0s8 (check the name in the VM settings)
      3. check Automatically connect
      4. Exit the ui
      5. systemctl restart network
    6. Disable ipv6 (it is sooo slow)
      1. Append below lines in /etc/sysctl.conf:
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1

      1. sysctl -p
      2. nano /etc/ssh/sshd_config
      3. Append this to the config file:

    AddressFamily inet

      1. systemctl restart sshd
    1. Do a minimal samba installation

      1. Configure firewall
        1. firewall-cmd --permanent --zone=public --add-service=samba
        2. firewall-cmd --reload
      2. Install samba
        1. yum install samba 
        2. mv /etc/samba/smb.conf /etc/samba/smb.conf.bak
        3. nano /etc/samba/smb.conf
        4. Enter the above to the text editor, save and exit
    [global]
    workgroup = WORKGROUP
    netbios name = server1
    security = user
    map to guest = Bad User
    dns proxy = no
    1. Enable and start samba
      1. systemctl enable smb.service
      2. systemctl enable nmb.service
      3. systemctl restart smb.service
      4. systemctl restart nmb.service
    2. Test samba
      1. testparm /etc/samba/smb.conf
      2. At this point you should be able to login with Putty as well
    3. update your new server
      1. yum makecache fast
      2. yum install epel-release
      3. yum update
      4. yum install nano (unless you prefer vi)

    2017. július 27., csütörtök

    The case against automatic memory management (a.k.a. Garbage Collection)

    The state of the memory...

    Microservices are good for scalability, because they are small and if one of your servers become hot, you get the option to move it around. They solve the smallest meaningful problem and communicate nicely with each other. CPU, memory and other hw related issues are things from the past since we have our machine learning based automated service deployment utility. Life is good... for most of us. Some are cursed with problems less easily "micro-serviced".
    Consider the world's arguably easiest-to-parallelize problem: ray-tracing. In ray tracing you can potentially calculate every pixel with a separate service and combine it to the whole picture at the end of the processing. The problem is, ideally you load all the different components of the scene (objects, textures, etc.) to each service's memory. If you are hard pressed, you can start with the wireframe models and load textures on demand. Optimizations exists (i.e. bounding boxes instead of complete objects), but at the end of the day you need to know what the current ray hits with some precision. A complicated scheme might cause out of memory problems on all of your rendering microservices at the same time.

    ... and the way we manage it

    Considering the top 10 programming languages from Tiobe Index we can see 75% of the most popular languages utilizes some kind of memory management works independently from what we specify in the code. The notable exception is C and C++, where garbage collection is available, but optional part of life: the C++11 specification allows for optional GC mechanism.
    This means 75% of the cases we let somebody else's code take care of the memory, while we are becoming increasingly particular about what the CPU supposed to be doing.
    Considering how easily a memory problem turns into a CPU problem due to increased GC activity it looks like we tend to play favorite with the CPU and trying to ignore memory as much as we can.

    Any better way to do this?

    I've been playing with Rust for quite some time now - nothing valuable just some hello world and dice rolling. Rust is not an easy language to begin with, lots of things done in a completely different ways than i.e. Java. Yes, memory management is one of them.
    At its heart all memory manager (or garbage collector) is a reference counter. We say: "I don't care about who owns this stuff, figure out when it is not being used and get rid of it" and the GC tries to do it as unobtrusively as possible.
    In Rust, you do have to figure out who is the owner of your data (variables, references). When you refer to the same data with a new variable, you have to explicitly tell if you want to transfer ownership (R/W access) to the variable. If yes, the old one is not going to be available!
    This has significant advantages not only in sense of memory management (when owner variable goes out of scope, data will be deleted). It helps a lot in multi-threading with data sharing as well. Combined with the deterministic GC behavior, in Rust you know what's going on with memory. It is less comfortable than outsourcing it to some other thread, but definitely more dependable.




    2017. július 9., vasárnap

    How to contribute to github projects?

    Just another "note to self" type entry about open source contribution.
    1. Find a good, welcoming project i.e. here: http://up-for-grabs.net/#/
    2. Fork the GitHub repo (let's call it upstream repo from now on) to your local repositories: on GitHub, press the fork button. Now you have a copy of the source code
    3. Get the code down to your developer machine
      1. To obtain clone url, go to your repo's page on GitHub and click "clone or download"
      2. git clone https://github.com/vizmi/sequelize.git
    4. Now you have you nice, isolated copy of the original repo. Time to sync it up with the upstream repo
      1. Navigate to the original repo (there's a link to it under the project name)
      2. Get the clone url (click "clone or download")
      3. Set-Location .\sequelize\
      4. git remote -v show the currently configured remotes. Ideally you have 2, pointing to your own repo. We are about to add 2 more, with one single command
      5. remote add upstream https://github.com/sequelize/sequelize.git
    5. Time to sync the local repo to the upstream repo (to avoid merge conflicts later)
      1. git fetch upstream
      2. git checkout master
      3. git merge upstream/master


    2017. január 22., vasárnap

    What is wrong with scaffolding?

    This is a strongly opinionated post, reflect my personal standpoint only.
    The point of scaffolding is to get you up and running real quick, which is a good thing, right? Well, not always.
    Let's say you are trying to get something done in a fairly new way (this is what we love about javascript development: every month there's a new way to do the same thing). You found the perfect yeoman generator for it, downloaded it down and have half dozen of config files generated for you. Now you are ready to develop the next killer app, except you have no idea which files are needed in the git repo, what changes needed for a production ready config and many more similar questions. Now you have to backtrack whatever template you have, understand why it is there and what does it actually do, and possibly spend more time on it than doing it step by step.
    The point is: if you are doing something the first time, it probably worth doing it without scaffolding.
    For the nth similar javascript project scaffolding is just great!