Disabling Indexing On the Synology DS216+

After 6 years, I finally upgraded my Synology DS209 to a DS216j, and still find indexing to be a problem service that I want to turn off. A few things I found out about how the indexing service is setup on the newer DS216j:

1) Finding the running index processes:
ps ax|grep index
6291 ? SNs 0:00 /usr/syno/sbin/synoindexd
6788 ? SN 0:00 /usr/syno/sbin/synoindexscand
6789 ? SN 0:00 /usr/syno/sbin/synoindexworkerd
6790 ? SN 0:00 /usr/syno/sbin/synoindexplugind
22715 pts/3 S+ 0:00 grep –color=auto index

2) Indexing is controlled through an Upstart job located at /usr/syno/sbin/synoindexd
You can review all Upstart jobs by issuing the command “initctl list”

3) If you want to look at the index job specifically, issue the “initctl show-config synoindexd” command to reveal the specifics of the job:

synoindexd
start on started pgsql-adapter
stop on stopping pgsql-adapter

4) If you want to review the script itself, open the /etc/init/synoindexd.conf file:

description “configure network device”
author “Web Application Team”

start on started pgsql-adapter
stop on stopping pgsql-adapter

expect fork

respawn
respawn limit 5 10

pre-start script

# make sure pgsql is running
if /usr/syno/sbin/synoservice –is-enabled pgsql > /dev/null 2>&1; then
echo “PGSQL service is disabled. Skip…”
stop
exit 1
fi

end script

exec /usr/syno/sbin/synoindexd

post-stop script
echo “Stopping Synology Index Daemon…”
killall synoindexscand > /dev/null 2>&1 || true
killall synoindexworkerd > /dev/null 2>&1 || true
killall synoindexplugind > /dev/null 2>&1 || true
killall synomediaparserd > /dev/null 2>&1 || true
end script

 

Notice that the index script is tied to the Postgresql database start. If you try killing the index service with a “kill -9” command it will kill the process, and then restart it. I have not yet tried, but perhaps the best way to stop the index service is to edit the /etc/init/synoindexd.conf file and change the “start on started pgsql-adapter” line to “stop on started pgsql-adapter.”

I will update this post as I try this solution.

 

Installing WS02 API Manager On Ubuntu Linux 14.04

There are official instructions to installing WSO2 API Manager that can be found here, but I found them lacking in critical details. The walkthrough below lays out the detailed steps for doing this.  I spun up the cheapest $5/month VM at Digital Ocean to keep costs down as I experimented. These VMs have 20GB of hard disk and 512MB of RAM, which are FAR below the recommended sizing for API Manager. I  recommend adhering to those guidelines if you are deploying to production.

  1. Login to Digital Ocean and spin up a preconfigured Ubuntu 14.4 x64 Droplet.
  2. Once provisioned, update the Droplet to have the latest packages (sudo apt-get update, sudo-apt-get upgrade,sudo apt-get dist upgrade) and reboot into the latest Linux kernel.
  3. I like to install HTOP for a more visual understanding of how a server is consuming resources: sudo apt-get install htop
  4. Install Oracle Java 7 (NOT a newer version – the official documents explicitly state this)
    (UPDATE: the latest API Manager 1.10 fully supports JDK 1.8)

    1. sudo apt-get install python-software-properties
    2. sudo add-apt-repository ppa:webupd8team/java
    3. sudo apt-get install oracle-java7-installer
    4. Check that Java was installed properly with the “java -version” command:
      java version “1.7.0_80”
      Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
      Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
    5.  sudo apt-get clean all
  5. Create a 2GB swapfile to ensure you can start WSO2 in a memory-constrained environment:
    1. Create a swapfile:
      1. sudo swapoff /swapfile
        sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024
        sudo dd if=/dev/zero of=/swapfile bs=2048 count=2048
        sudo mkswap /swapfile
        sudo swapon /swapfile
        sudo chown root:root /swapfile
        sudo chmod 0600 /swapfile
        sudo nano /etc/fstab
        /swapfile none swap sw 0 0
  6. Edit the /etc/environment file to point two variables to their proper places:
    1. Add the following:
      1. JAVA_HOME=”/usr/lib/jvm/java-7-oracle/”
        CARBON_HOME=”/usr/local/opt/wso2am-1.10.0″
      2. source /etc/environment
      3. Test the variables:  echo #JAVA_HOME
  7. WSO2 needs a backing MYSQL server to run. Here is how you install it:
    1. sudo apt-get install mysql-server
    2. Set a password: mysqladmin -u root password (password)
    3. Change the max_connections variable from 100 to 2 in /etc/mysql/my.conf
      max_connections = 2
    4. Import the WSO2 database schema in MySQL:
      1. mysql -u root -p WSO2AM_STATS_DB < /usr/local/opt/wso2am-1.10.0/dbscripts/stat/sql/mysql.sql
  8. Download the WS02 zipped binary from here, and copy it to the Droplet. Its 381Mb in size, so it may take awhile to download.
  9. Once the file is on the server, unzip it to the /usr/local/opt directory:
    1. mkdir -p /usr/local/opt
    2. cp wso2am-1.10.0.zip /usr/local/opt
    3. unzip wso2am-1.10.0.zip
  10. Note that logs for the server are kept in the /usr/local/opt/wso2am-1.10.0/repository/logs directory if you need to review them. The wso2carbon.log contains a great deal of useful information.
  11. Edit the server startup script (wso2server.sh), adding the same variables you added to /etc/environment to the top of the file:
    1. sudo nano /usr/local/opt/wso2am-1.10.0/bin/wso2server.sh
    2. JAVA_HOME=”/usr/lib/jvm/java-7-oracle/”
      CARBON_HOME=”/usr/local/opt/wso2am-1.10.0″
  12. Edit the server startup script (wso2server.sh) to alter the JVM size parameters:
    1. sudo nano /usr/local/opt/wso2das-3.0.0/bin/wso2server.sh
    2. Scroll down to approximately line 294
    3. Change the parameters to read: -Xms256m -Xmx1024m -XX:MaxPermSize=256m \
  13. Start the server: /usr/local/opt/wso2am-1.10.0/bin/wso2server.sh start
    1. Note that it takes AWHILE to startup the server, as a number of applications have to be deployed. Be patient and expect high CPU usage.
    2. You can tail the wso2carbon.log file to watch startup happen:   tail -f /usr/local/opt/wso2am-1.10.0/repository/logs/wso2carbon.log
    3. Stopping the server is done via the same command: /usr/local/opt/wso2am-1.10.0/bin/wso2server.sh stop
  14. If everything started successfully, you will see the assertions below in the wso2carbon.log

TID: [-1234] [] [2016-01-10 20:17:11,502] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} – Server : WSO2 API Manager-1.10.0 {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}
TID: [-1234] [] [2016-01-10 20:17:11,547] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} – WSO2 Carbon started in 106 sec {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}
TID: [-1234] [] [2016-01-10 20:17:13,710] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} – Mgt Console URL : https://10.17.0.5:9443/carbon/ {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
TID: [-1234] [] [2016-01-10 20:17:13,715] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} – API Publisher Default Context : http://10.17.0.5:9763/publisher {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
TID: [-1234] [] [2016-01-10 20:17:13,716] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} – API Store Default Context : http://10.17.0.5:9763/store {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}

15. You can also run the “netstat -pantl” command to verify all the ports are listening:

netstat -pantl
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:8243 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 884/sshd
tcp 0 0 10.17.0.5:10711 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:8280 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:48123 0.0.0.0:* LISTEN 2373/java
tcp 0 0 10.17.0.5:10397 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:41919 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:9443 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:9763 0.0.0.0:* LISTEN 2373/java
tcp 0 0 0.0.0.0:11111 0.0.0.0:* LISTEN 2373/java

15. OPTIONAL: change the listener.

By default the server is set to only be accessible by localhost. This may be suboptimal if you want to make the server available on the Internet. Make this change in the /usr/local/opt/wso2am-1.10.0/repository/conf/carbon.xml file:

sudo nano -c carbon.xml
Change the <HostName> and (MgtHostName> elements to reflect either your Droplet’s external IP or DNS name:

<HostName>apgmgr.yourdomain.com</HostName>
<MgtHostName>apgmgr.yourdomain.com</MgtHostName>

OR

<HostName>104.131.83.69</HostName>
<MgtHostName>104.131.83.69</MgtHostName>

16. Login to API Manager

Note that the product uses a self-signed SSL cert, and your browser will balk at it. Provision a legitimate cert or tell your browser to ignore the error. The former is recommended. Here is what that process looks like, which gets you to the main console:

2016-01-17_11-42-07  2016-01-17_11-43-03

17. The main login screen will look like the screenshot below. Note that I have deployed a sample API app.

2016-01-17_11-43-27

This should get you up and running, deploying APIs with the product. Note that the API statistics portion of the API Manager product will not be available unless you install the DAS (Data Analytics Server) module on a separate server, as DAS provides that functionality. If you do not install DAS, you will not be able to view any statistics about your APIs, and will see the following soft error when you try to access the Statistics section:

2016-01-17_11-49-07

Installing Mitmproxy on OSX Yosemite

There are only a few steps here, but they are important ones:

1) $ sudo easy_install pip
2) $ sudo pip install mitmproxy
3) $ mitmproxy –version
You are using an outdated version of pyOpenSSL: mitmproxy requires pyOpenSSL 0.14 or greater.
Your pyOpenSSL 0.13.1 installation is located at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/OpenSSL

4) $ sudo easy_install –upgrade PyOpenSSL
5) $mitmproxy –version
mitmproxy 0.11.3

Upgrading Ruby To A New Patch Version

Ruby exploits are becoming more common. When one is found, it is often necessary to upgrade the Ruby in your Ruby-on-Rails  stack to the latest patch version to mitigate the vulnerability du jour. If you are using RVM, here’s how you do it:

1) Verify the latest, stable version of Ruby currently available for download at ruby-lang.org
Note the version number.

2) Check your current version of ruby.
$ rvm list
=* ruby-2.0.0-p247 [ x86_64 ]

3) Download and upgrade to the latest, stable version of RVM
$ rvm get stable

4) List the versions of Ruby that RVM knows about.
$ rvm list known
You are looking for the “MRI Rubies” section-scroll up to find it. Then ensure the Ruby version you found in step one is in the list:
[ruby-]2.0.0[-p353]

5) Upgrade Ruby by passing in the name of your current version and the top-level version number.
rvm <your_current_version> 2.0.0
$ rvm upgrade 2.0.0-p247 2.0.0
You should see a confirmation message appear:
Are you sure you wish to upgrade from ruby-2.0.0-p247 to ruby-2.0.0-p353? (Y/n):

6) Select Yes to comments the upgrade. It may take some time to complete, as Ruby must be downloaded and compile, but finish without incident. Note that the installer will also offer to move your gems to the new version, while deleting them from the old version. It will also offer to move your aliases and wrappers. For convenience sake, its advisable to accept these choices.
Are you sure you wish to MOVE gems from ruby-2.0.0-p247 to ruby-2.0.0-p353?
This will overwrite existing gems in ruby-2.0.0-p353 and remove them from ruby-2.0.0-p247 (Y/n):
Do you wish to move over aliases? (Y/n): y
Do you wish to move over wrappers? (Y/n): y

7) Removing old, vulnerable version of Ruby.
It is also advisable to allow the installer to remove the older version of Ruby:
Do you also wish to completely remove ruby-2.0.0-p247 (inc. archive)? (Y/n):
Removing ruby-2.0.0-p247……..

8) If all has gone as expected, you will see a confirmation message.
Successfully migrated ruby-2.0.0-p247 to ruby-2.0.0-p353
Upgrade complete!

9) Set the upgraded version of Ruby as your RVM default.
$ rvm –default use 2.0.0

10) Verify you are running the new, patched version of Ruby:
$ rvm list
=* ruby-2.0.0-p353 [ x86_64 ]

Starting Chrome on a blank page.

As a new Chrome user, I wanted a plain, white blank page to start my web browsing. What I got by default was the Google tab view page. Here is how you set a blank startup page for Chrome:

1) Settings -> On Startup -> Open a specific page or set of pages.
2) Click Set pages to open Startup Pages window
3) Click the X to delete what you find there.
4) In the field for Add a New Page, enter: about:blank
5) Click OK to save the change.

Now completely close out of Chrome, and restart it. You should be brought to a blank page.

10 Steps to Ruby 2.x and Rails 4.x on Mountain Lion

1) Download and install Xcode Command Line Tools
Mountain Lion file name: xcode462_cltools_10_86938259a.dmg

2) Install GitHub for Mac
This also has an option to install the Git command-line tools. Choose that option.
http://mac.github.com/

3) Install RVM
$ curl -L https://get.rvm.io | bash -s stable –ruby

4) Source rvm to open in the existing shell
$ source ~/.rvm/scripts/rvm

5) Set default Ruby version
$ rvm –default use 2.1.0

6) Update Gem manager
$ gem update –system

7) Use the global Gemset by default
$ rvm gemset use global

8) Update all Gems
$ gem update

9) Don’t download Gem documentation at install
$ echo “gem: –no-document” >> ~/.gemrc

10) Install Rails
$ gem install rails

Latest Ruby version not showing up in RVM?

I just realized I’m running an out-of-date version of Ruby (1.9.3-p327), and browsed to the Ruby website to find out what the latest, stable version is. It turns out 1.9.3-p362 is the latest version:

2013-01-03_23-13-57

When I tried to install this version via RVM, RVM complained it could not find the binaries for it, and threw the following error:
2013-01-03_23-14-54

Well, that doesn’t seem right. Lets list the versions of Ruby that RVM knows about:

$ rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-p371]
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p320]
[ruby-]1.9.3-p125
[ruby-]1.9.3-p194
[ruby-]1.9.3-p286
[ruby-]1.9.3-[p327]
[ruby-]1.9.3-head
[ruby-]2.0.0-preview2
ruby-head

No version 1.9.3-p362 in that that list, huh? Lets update RVM to obtain the latest stable release with the “rvm get stable” command:
2013-01-03_23-16-05

Now that’s more like it. Lets see what “rvm list known” shows us now:

$ rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-p371]
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p320]
[ruby-]1.9.3-p125
[ruby-]1.9.3-p194
[ruby-]1.9.3-p286
[ruby-]1.9.3-p327
[ruby-]1.9.3[-p362]
[ruby-]1.9.3-head
[ruby-]2.0.0-preview2
ruby-head

Annnd there’s the 1.9.3-p362 version, ready for installation with the usual “rvm install ruby-1.9.3-p362” command.

Tuning your Synology NAS for speed

If you own a Synology NAS device at home, you know its an incredible machine for the money. However, it likely has a small amount of RAM (mine has 256MB) and that poses a problem. I have found that if you are not using many of the programs (Disk Station, iTunes server, File Station, etc…), you can free up a great deal of RAM by disabling unused services. This gives precious RAM back to processes that can actually use it, and frees up CPU power.

The first step is to disable services you do not need. For me, this includes all the file indexing and thumbnail services, CUPS printing, USB hotplug, and a few other miscellaneous services. Ensure that SSH access is on (Control Panel – Terminal – Enable SSH Service) and login to the device via SSH.  Then issue the following commands:

/usr/syno/etc/rc.d/S56synoindexd.sh stop
/usr/syno/etc/rc.d/S88synomkflvd.sh stop
/usr/syno/etc/rc.d/S77synomkthumbd.sh stop
/usr/syno/etc/rc.d/S55cupsd.sh stop
/usr/syno/etc/rc.d/S20pgsql.sh stop
/usr/syno/etc/rc.d/S66fileindexd.sh stop
/usr/syno/etc/rc.d/S03hotplugd.sh stop
/usr/syno/etc/rc.d/S03inetd.sh stop
/usr/syno/etc/rc.d/S98findhostd.sh stop

These commands will stop the services from running, but they will restart at the next reboot. To prevent this, simply change the executable status of each service’s start script to be un-executable. This will ensure that disabling these services survives the next reboot. Issue the following commands:

chmod -x /usr/syno/etc/rc.d/S66synoindexd.sh
chmod -x /usr/syno/etc/rc.d/S88synomkflvd.sh
chmod -x /usr/syno/etc/rc.d/S77synomkthumbd.sh
chmod -x /usr/syno/etc/rc.d/S55cupsd.sh
chmod -x /usr/syno/etc/rc.d/S20pgsql.sh
chmod -x /usr/syno/etc/rc.d/S66fileindexd.sh
chmod -x /usr/syno/etc/rc.d/S03hotplugd.sh
chmod -x /usr/syno/etc/rc.d/S03inetd.sh
chmod -x /usr/syno/etc/rc.d/S98findhostd.sh

Note that some of these services might be needed by services you want to run on your DiskStation. You may cherry-pick the services you wish to disable. I only use my DiskStation as a local backup target and have it push to Crashplan, so my requirements are very low. You can reenable a service by using the reverse “chmod +x” command and restarting it.

Next, open the /etc/sysctl.conf file, and add the following network-related settings:

kernel.panic=3
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1

This will provide a number of TCP tunings for the Gigabit ethernet card that the DiskStation uses. After making all of these changes, be certain to reboot the box to clear system RAM and caches. You should find that your DiskStation is now more responsive, and tasks should execute much faster.